Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ version's driver exists.
| `SMOLQUERY_CLAIM_VALVE_FACTOR` | The multiplier from the two seal triggers to the two claim valves (`16`, T-335). One claim freezes at most `SMOLQUERY_SEAL_MAX_BYTES ×` this value in bytes, and `SMOLQUERY_SEAL_MAX_FILES ×` this value in micro-segments — `1 GiB` and `1024` on the defaults. The claim is what the storage tier must merge in one go. Every limit it has to fit inside is a separate setting: the merge engine's memory limit, its spill limit, and its three call budgets. Nothing checks the relationship. A 488-segment claim on a 63-column table exhausted all five in turn, and the partition wedged: a claim's inputs are frozen, so every retry merges the same set. Lower the factor to seal a wide table in smaller pieces. Seal cost measured as roughly `segments^1.21`, so a claim four times smaller costs more than four times less, and the ref's one live claim blocks it for a shorter window. Read the resolved valves off the `buffer shape:` line at boot |
| `SMOLQUERY_ENCODE_CONCURRENCY` | How many of a table's Parquet encodes run at once (default: the node's scheduler count). A container held to one core thus encodes serially; that is what one core means. The manifest append, the replication round, and the replies stay serialized in the Committer regardless. The scheduler count follows cpusets, not CFS (Completely Fair Scheduler) quotas. Where quotas are the fence, set this value and the pool size explicitly |
| `SMOLQUERY_FLUSH_WRITER` | Selects the writer that turns a flush into Parquet: `duckdb` (default) or `polars`. Any other value fails the boot. `duckdb` also stops the parsing at the ingest edge. The NDJSON body goes to the owning buffer as bytes. One `COPY ... read_json` parses, sorts, and writes it at flush. The default path defers schema validation to flush. It then salvages a failed batch row by row. This preserves per-row `insertErrors`. A bad row does not always fail the whole commit; the writer still writes the successful rows. `/insert` accepts NDJSON only; the JSON-array envelope was removed |
| `SMOLQUERY_MAX_LIVE_CLAIMS` | How many seal claims one table ref may hold open at once (`1`, T-339). At `1` a ref seals serially: the next claim forms only after the previous one retires, so one slow merge is the whole ref's seal throughput. Above `1`, the buffer keeps freezing valve-sized claims from the unclaimed tail while earlier claims merge. One ref's backlog can then occupy that many storage seal slots in parallel, and the unsealed tail drains toward zero instead of growing behind one merge. Sealed segments may land out of input order within the ref; every manifest consumer is per-entry, so nothing reads that order. Size it with `SMOLQUERY_MAX_CONCURRENT_SEALS` and the merge engine's memory limit, which concurrent merges share |
| `SMOLQUERY_MAX_LIVE_CLAIMS` | How many seal claims one table ref may hold open at once (`1`, T-339). At `1` a ref seals serially: the next claim forms only after the previous one retires, so one slow merge is the whole ref's seal throughput. Above `1`, the buffer keeps freezing valve-sized claims from the unclaimed tail while earlier claims merge. One ref's backlog can then occupy that many storage seal slots in parallel, and the unsealed tail drains toward zero instead of growing behind one merge. Sealed segments may land out of input order within the ref; every manifest consumer is per-entry, so nothing reads that order. Size it with `SMOLQUERY_MAX_CONCURRENT_SEALS` and the merge engine's memory limit, which concurrent merges share. Read the resolved value off the `buffer shape:` line at boot |
| `SMOLQUERY_WRITE_POOL_SIZE` | How many DuckDB instances a `duckdb` flush writer runs (default: the node's scheduler count, capped at `32`). Valid values are `1..32`; the boot refuses anything else. The pool selects a member per segment, not per table. A table has one committer, so a hash on the table would send every flush to one connection. Each member gets the thread count of `Smolquery.Engine` divided by the pool size (floor one). Each member inherits the engine's memory limit whole. The two variables below size a member explicitly. With no explicit engine thread count, both the engine and the pool resolve `System.schedulers_online()` at boot on the deployment host, not on the release builder. The host-derived count means the *declared* DuckDB write memory scales with it — see `SMOLQUERY_WRITE_ENGINE_MEMORY_LIMIT`. Read the resolved numbers off the `buffer shape:` line at boot |
| `SMOLQUERY_WRITE_ENGINE_THREADS` | The DuckDB threads for one write-pool member. It replaces the division. The division describes a budget only while the pool is smaller than the thread count. Past that point, the result sits at its floor of one. An operator who wants a different shape states the number |
| `SMOLQUERY_WRITE_ENGINE_MEMORY_LIMIT` | The DuckDB memory limit for one write-pool member, for example `512MB`. Unset, every member inherits `SMOLQUERY_MEMORY_LIMIT` whole. The node's declared DuckDB write budget is then `write_pool_size ×` that value. A size string has its own grammar; nothing divides it for you |
Expand Down
1 change: 1 addition & 0 deletions lib/smolquery/deployed_shape.ex
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ defmodule Smolquery.DeployedShape do
claim_valve_factor: runtime.claim_valve_factor,
claim_max_bytes: runtime.seal_max_bytes * runtime.claim_valve_factor,
claim_max_files: runtime.seal_max_files * runtime.claim_valve_factor,
max_live_claims: runtime.max_live_claims,
transport_tls: transport_tls?()
]

Expand Down
7 changes: 7 additions & 0 deletions test/smolquery/deployed_shape_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,13 @@ defmodule Smolquery.DeployedShapeTest do
assert log =~ "claim_valve_factor=2"
assert log =~ "claim_max_files=128"
assert log =~ "claim_max_bytes=2000"
assert log =~ "max_live_claims=1"
end

test "states how many claims a ref may hold open (T-339)" do
log = capture_log(fn -> DeployedShape.announce(buffer_runtime(max_live_claims: 3)) end)

assert log =~ "max_live_claims=3"
end

test "states an explicit member budget over the derived one" do
Expand Down
Loading