Add optional Max Input Size guard to the write processors

[snowch]

Summary

PutVastDB / UpdateVastDB / DeleteVastDB read the whole FlowFile into memory (getContentsAsBytes) and decode it into an in-memory Arrow table before writing, so a large enough FlowFile exhausts the NiFi Python process and gets it OOM-killed — an unrecoverable crash, not a routed failure. (The write itself is safe: the vastdb SDK already slices inserts into ~4.5 MB RPCs, so the ceiling is client-side memory, not a server limit.)

This adds an opt-in Max Input Size property so an over-large FlowFile fails gracefully instead of crashing the interpreter.

Behaviour

• Property Max Input Size (required=False, DATA_SIZE_VALIDATOR, e.g. 500 MB / 2 GB; empty = disabled).
• At the very start of transform(), before any content is read, if flowfile.getSize() > limit the FlowFile is routed to failure with a vastdb.error attribute. The check uses only the FlowFile's size metadata — it never calls getContentsAsBytes(), so the guard itself allocates nothing.
• Shared logic in input_guard.py, used by all three write processors. ImportVastDB is intentionally excluded — its incoming FlowFile is just the small {key,bucket} list; the Parquet is read server-side.

Design notes

The ceiling is operator-set, not inferred. The safe size depends on the node's Python memory budget, its concurrency, and the format's decode expansion (Parquet decompresses to many times its on-disk size), none of which the processor can reliably know — and reading cgroup limits or probing by allocation is unreliable (the OOM killer sends an uncatchable SIGKILL; Linux overcommit makes MemoryError unreliable). So this is a safety tripwire, not a memory oracle. The docs steer bulk loads to SplitRecord (JVM-side streaming split) or ImportVastDB (server-side read), where memory never depends on input size.

Docs & tests

• PutVastDB.md / UpdateVastDB.md / DeleteVastDB.md document the property and a "Guarding against large inputs" rationale section.
• 20 new unit tests (data-size parsing incl. rejects, the size comparison, and a PutVastDB wiring test that an oversized FlowFile routes to failure). 48 pass overall; hatch fmt --check clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G7cjFLk6huEsGcV6nG21n5