Java Hibernate many-to-many bidirectional inserting results in strange behaviour

Question

i have next very simple entities (Author and Book) with many-to-many relationship:

@Entity
@Data
@AllArgsConstructor
@NoArgsConstructor
@Table(name = "authors")
@SequenceGenerator(name="author_id_seq_generator", sequenceName="author_id_seq", allocationSize = 1)
public class AuthorEntity {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @Column(name = "id")
    Long id;
    @Column(name = "surname")
    String surname;
    @Column(name = "name")
    String name;

    @ManyToMany(mappedBy = "authors")
    Set<BookEntity> books;
}

@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity
@Table(name = "books")
@SequenceGenerator(name="book_id_seq_generator", sequenceName="book_id_seq", allocationSize = 1)
public class BookEntity {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @Column(name = "id")
    private Long id;
    @Column(name = "title")
    private String title;
    @Column(name = "price")
    private Double price;

    @ManyToMany
            @JoinTable(
                    name = "Books_To_Authors",
                    joinColumns = @JoinColumn(name = "book_id"),
                    inverseJoinColumns = @JoinColumn(name = "author_id")
            )
    Set<AuthorEntity> authors;
}

Then in my repository i receive a request type of

@Value
public class AuthorWithKnownBooksRequest {
    String surname;
    String name;
    List<Long> bookIds;
}

and here is the method:

@Override
    @Transactional
    public void addBookWithKnownAuthors(BookWithKnownAuthors bookWithKnownAuthors) {
        // get authors
        MultiIdentifierLoadAccess<AuthorEntity> multiLoadAccess = entityManager.unwrap(Session.class).byMultipleIds(AuthorEntity.class);
        List<AuthorEntity> authors = multiLoadAccess.multiLoad(bookWithKnownAuthors.getAuthorIds());
        // to avoid infinite recursion
        authors.stream().forEach(author -> author.setBooks(null));
        // create book entity
        BookEntity be = new BookEntity();
        be.setPrice(bookWithKnownAuthors.getPrice());
        be.setTitle(bookWithKnownAuthors.getTitle());
        be.setAuthors(authors.stream().collect(Collectors.toSet()));
        // save to db
        entityManager.persist(be);
    }

and it works fine,

Hibernate: select authorenti0_.id as id1_0_0_, authorenti0_.name as name2_0_0_, authorenti0_.surname as surname3_0_0_ from authors authorenti0_ where authorenti0_.id in (?, ?, ?)
Hibernate: select nextval ('hibernate_sequence')
Hibernate: insert into books (price, title, id) values (?, ?, ?)
Hibernate: insert into books_to_authors (book_id, author_id) values (?, ?)
Hibernate: insert into books_to_authors (book_id, author_id) values (?, ?)
Hibernate: insert into books_to_authors (book_id, author_id) values (?, ?)

however if I'm trying to do the same thing from author's side, it behaves strangely

@Override
    @Transactional
    public void addAuthorWithKnownBooks(AuthorWithKnownBooks authorWithKnownBooks) {
        MultiIdentifierLoadAccess<BookEntity> multiLoadAccess = entityManager.unwrap(Session.class).byMultipleIds(BookEntity.class);
        List<BookEntity> books = multiLoadAccess.multiLoad(authorWithKnownBooks.getBookIds());
        // to avoid infinite recursion
        books.stream().forEach(book -> book.setAuthors(null));
        // create author entity
        AuthorEntity ae = new AuthorEntity();
        ae.setName(authorWithKnownBooks.getName());
        ae.setSurname(authorWithKnownBooks.getSurname());
        ae.setBooks(books.stream().collect(Collectors.toSet()));
        // save to db
        entityManager.persist(ae);
    }

The result is instead of inserting I get deleting of rows from connection table.

Hibernate: select bookentity0_.id as id1_1_0_, bookentity0_.price as price2_1_0_, bookentity0_.title as title3_1_0_ from books bookentity0_ where bookentity0_.id in (?, ?, ?)
Hibernate: select nextval ('hibernate_sequence')
Hibernate: insert into authors (name, surname, id) values (?, ?, ?)
Hibernate: delete from books_to_authors where book_id=?
Hibernate: delete from books_to_authors where book_id=?
Hibernate: delete from books_to_authors where book_id=?

It looks to me as if the problem is with the owner side, because if I change @ManyToMany in BookEntity and AuthorEntity the other way around it works for Authors but doesn't work for Books

Can anyone explain to me how I can fix it? or the way it should be done? I feel like i'm missing some important concept.