Execute huge Cypher script with CREATE (node and relationships)

Question

I have a question about settings (Neo4j configuration). I have a 15GB Cypher script with CREATE keywords only (no CSV loading) , which has a repeating pattern inside: create of a few (usually 10-30) nodes and create of a few relationships that use previous nodes in the sentence of IDs (a sequence of relationships in one CREATE keyword).

I do not need any optimization (I can wait a long time for loading), I do not need transactions, relationships and nodes are not scattered in the file.

Of course, Neo4j Browser is not suitable for this, so I chose Cypher-shell. I try to load that data by cypher-shell but keep getting messages that loading failed because of the heap. The same message on Bash and inside cypher-shell.

I think cypher-shell is trying to parse the entire file (maybe to find identifiers scattered around in the file). It doesn't make sense, of course, because they aren't there.

I use different versions of Neo4j. (e.g. 3.3.9), Linux Mint 20.2. I have 16GB RAM.

Can you advise me what lines should I change in the settings so that the loading process is successful? I play a bit with dbms.memory.heap.initial_size, dbms.memory.heap.max_size, dbms.memory.pagecache.size, and dbms.tx_state.memory_allocation but without success.

Answer 1

Wrapping the existing Cypher query with an apoc.periodic.commit and specifying a smaller batch size might do the trick.
From the docs:

apoc.periodic.commit(statement,params) - runs the given statement in separate transactions until it returns 0