graphify export falkordb: MERGE without an index causes O(n²) push, 100x slowdown on large graphs

[abderrahim-lectures]

What happened

Pushing a 167,428-node / 209,215-edge graph (graphify export falkordb --push falkordb://localhost:6379) started fast (~10,000 node upserts/60s) and then collapsed to ~100 upserts/300s partway through, with the FalkorDB container pegged at ~97% CPU. Total wall time for the full push was multiple hours.

Root cause

push_to_falkordb in graphify/exporters/graphdb.py issues, per node:

graph.query(
    f"MERGE (n:{ftype} {{id: $id}}) SET n += $props",
    {"id": node_id, "props": props},
)

No index is created on (label, id) before this loop runs. Without one, MERGE on the match pattern does a full label scan to check for an existing node, so each upsert costs O(current nodes with that label) — the whole push is O(n²) in the number of nodes per label.

Workaround (confirmed fix)

Creating one index per label before the push restores linear-time throughput:

for label in <distinct capitalized file_type labels in the graph>; do
  redis-cli -p 6379 GRAPH.QUERY <graph_name> "CREATE INDEX FOR (n:$label) ON (n.id)"
done

Verified on the graph above: after adding the indexes, throughput recovered to the original ~10,000 upserts/60s rate for the remainder of the push.

Suggested fix

Have push_to_falkordb (and the equivalent Neo4j path, if it has the same gap) create CREATE INDEX FOR (n:<label>) ON (n.id) for each distinct node label present in the graph before the node-upsert loop runs, the same way a CREATE CONSTRAINT/index-first pattern is standard for bulk Cypher loads. This is idempotent and safe to run on every push, including re-runs against an already-populated graph.

Environment

• graphifyy 0.9.67
• FalkorDB (falkordb/falkordb:latest, Docker)
• Graph: 167,428 nodes across 6 labels (Code, Concept, Document, Image, Paper, Rationale), 209,215 edges