[pull] master from kevoreilly:master#503
Merged
Merged
Conversation
…or-neutral) Adds an opt-in [nexthop]/[gwX] routing mode that binds each analysis task's guest source IP to a chosen egress gateway via a dedicated policy-routing table, MASQUERADing onto that gateway's interface. Disabled by default: with [nexthop] enabled=off the existing routing paths are byte-for-byte unchanged. - conf/default/routing.conf.default: [nexthop] + [gwX] profile sections; the "nexthop" sentinel route is the pool default (default_policy). - lib/cuckoo/core/rooter.py: gateways registry + _select_gateway (explicit id / roundrobin / random; liveness-gated; fail-closed on None). - lib/cuckoo/core/startup.py: load_nexthop_profiles + validate_default_route (section-header id; required-option validation; each [gwX] rt_table must be a dedicated unique table -- rejects reserved/fail/duplicate/VPN/dirty-line; accepts gateway id / roundrobin / random / nexthop as a VPN-less default route; intra-subnet exception installed whenever nexthop enabled, blackhole only when fail_closed). - lib/cuckoo/core/analysis_manager.py: route_network resolves tunX before nexthop and lets nexthop consume the route (bind or fail-closed drop); reserved routes never resolve to a gateway; unroute mirrors the persisted tuple + guest ingress. - utils/rooter.py: nexthop_init/enable/disable/intra_exception/fail_closed/teardown (reserved-table guard; per-task forward ACCEPT via CAPE_ACCEPTED_SEGMENTS, constrained to the guest ingress iface (anti-spoof) and idempotent delete-until-gone; intra-subnet exception just below the per-task band; teardown sweeps only vm_net-sourced band rules by full selector; SIGTERM no-op when disabled). - web/submission: the route selector lists the "nexthop" pool + each configured [gwX] gateway when nexthop is enabled (defensive; no-op when disabled). - tests: selection, route/unroute, rooter primitives, netns integration. Fail-closed throughout: any unresolved/typo'd/reserved route drops rather than leaking onto the host default route.
…ngines) Design for replacing the fork+long-lived-pool+nested-pool model that deadlocked cape-processor. Pluggable engine seam: current pebble path preserved as A/B control; single-threaded prefork supervisor (ephemeral per-task children, process-group timeout kill, os._exit) as the target. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…design spec Audit verdict: all heavy load-once structures (YARA, geoip, capa, pyattck, DECRYPTORS) are read-only after load; _CLAMAV_CACHE is per-worker and self-clearing. Nothing needs live mutable cross-worker sharing, so Approach A's warm-base COW fork preserves the "load once, share" model identically and reclaims C-lib leaks per task. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…d lazy-load cost Lifecycle decided: fork-per-task, no supervisor->child dispatch channel (child runs one task then os._exit). Avoids rebuilding the inter-process channel whose deadlock motivated the redesign; max leak containment. recycle-after-N left as a documented future knob. Adds the load taxonomy (Category 1 init-time/COW-shared incl. yara-forge; Category 2 lazy first-use) and the measured Category-2 cost: loldrivers ~0.35s/+41MB, security_tools ~0, mmbot disabled, sigma via subprocess. ~1.6% per-task => no pre-warm registry needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
11 TDD tasks across 3 phases: engine seam + PebbleEngine control (behavior- preserving), PreforkEngine supervisor (fork-per-task, os._exit, killpg timeout, single-threaded invariant), extractor sub-pool spike+fix, A/B runbook. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-only Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…to match production txn discipline
… failed_processing path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… setup to match production txn discipline
…ault pebble) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…store per-task proctitle/formatter cleanup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ling; strengthen no-orphan test
…rdering Add test_worker_init_called_in_child to tests/test_prefork_engine.py using the db_file fixture (file-backed SQLite visible across fork boundary). The test asserts that a filesystem flag written by worker_init is present when task_fn runs in the child, locking the _child_main contract from Task 6. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolves design-spec §4.2(d): measured pebble.ProcessPool latency and killpg orphan safety for both 'fork' and 'forkserver' contexts. Decision: use fork (115.7 ms mean vs 125.5 ms / 219 ms p95 for forkserver; both pass orphan check). Task 10 will implement _EXTRACTOR_POOL accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ss-group kill
Replace the process-wide _EXTRACTOR_POOL cache and _get_extractor_pool() lazy
init with a per-call pebble.ProcessPool using multiprocessing.get_context("fork")
(Task 9 SPIKE decision). The pool is created inside generic_file_extractors,
used for all extractor scheduling and result collection, then closed and joined
in a try/finally block. This ensures worker processes are deterministically torn
down before the ephemeral task child exits, and that killpg(child_pgid) from
the prefork supervisor sweeps any survivors if per-call teardown fails.
Deleted: _EXTRACTOR_POOL, _EXTRACTOR_POOL_LOCK, _get_extractor_pool(), threading import.
Added: test_extractor_pool_is_swept_by_killpg confirming fork+setsid+killpg leaves zero survivors.
…in Task 10 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mpletion log; runbook paths; decoded waitpid status; wire max_analysis_count to both engines
…es native C-extension threads like STPyV8 V8 workers)
…hreads at module load, breaking prefork single-threaded invariant) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nel-single-threaded imagehash (via PIL/Pillow C extensions) spawns +15 native kernel threads at import time, breaking PreforkEngine._assert_single_threaded which now checks both Python threads and /proc/self/task. Move the import inside _load_imagehash() with the same deferred-binding pattern used for peepdf/STPyV8. Also includes mtime-ordered reindex_screenshots and zxing-cpp QR support already in working tree. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The prefork child forks once per task, runs every extracted file, then os._exit()s. generic_file_extractors runs once per file, so a per-call pool re-forks worker processes N times per task (~1.2s/file on heavy tasks) — the exact cost upstream's shared _EXTRACTOR_POOL eliminated. Reclaim it safely: the prefork child opts into a single shared pool (enable_shared_extractor_pool) reused across all files and torn down once (shutdown_shared_extractor_pool) before exit. Safe here — unlike pebble worker recycling, which deadlocks joining a persistent nested pool in _exit_function — because the child never cleanly joins mid-life; it exits and any straggler is swept by the supervisor's process-group kill. Pebble keeps the per-call pool.
nic_available only proves the link exists (true for administratively DOWN interfaces), so _gw_live treated a down gateway NIC as live and round-robin/random selection could bind a task to a dead exit. Add a nic_up rooter verb that checks the IFF_UP flag and use it for gateway liveness.
…an't wedge the worker pool.join() was unbounded: a wedged extractor grandchild (D-state IO, a hung Java decompiler) could block teardown forever, hanging the worker with no external timeout to break it (the pebble task timeout only covers execution, not teardown). Both teardown paths (per-call pebble, shared prefork) now go through _teardown_extractor_pool: graceful join up to EXTRACTOR_POOL_JOIN_TIMEOUT, then stop()+join() to force-kill stragglers. Addresses the last unbounded-hang in the pebble A/B path without touching the -mc 7 recycling default.
…it it (COW) autoprocess now compiles the YARA ruleset once before the engine forks any workers. Every forked child inherits the compiled rules via copy-on-write fork (0s + shared read-only pages) and its init_worker File.init_yara() short-circuits on the idempotency guard. Prefork forks one child per task, so without this each task paid the full compile (~0.3-3s depending on ruleset); measured ~321ms -> ~0.05ms per child. Safe: compilation is single-threaded, preserving the prefork single-threaded-before-fork invariant.
…+ nexthop_init blackhole seed) - _resolve_nexthop clears self.nexthop_* on entry so a failed resolve (route->drop) cannot leave a stale binding that _dispatch_nexthop still installs on route_network re-entry/retry. - nexthop_init seeds a blackhole default before the real replace, so a failed route setup leaves the gateway table fail-closed instead of empty (falling through to main), even with fail_closed=no.
…ed + default_policy) - reserve the [routing] dirty-line rt_table only when the dirty line is enabled (internet != none), so a nexthop-only node reusing that id is not falsely rejected at startup. - validate [nexthop] default_policy is roundrobin/random or a configured gateway id (else every pool task silently drops).
…collision at startup - nexthop_enable deletes the VM's prior rule by (from,priority) regardless of table, so a rebind to a different gateway cannot leave a stale old-table rule that wins by priority. - load_nexthop_profiles rejects fail_closed when NEXTHOP_FAIL_TABLE collides with a VPN/dirty-line table.
…; killpg fallback; idle backoff - _reap now iterates per-pid and holds a timed-out child until its process group has been escalated to SIGKILL (kill_deadline cleared). Reaping it earlier popped it from _inflight so _escalate_kills never SIGKILLed the group, orphaning any surviving grandchildren. - _enforce_timeouts falls back to os.kill(pid) when killpg raises ProcessLookupError (group not yet created / already gone), and only skips setting kill_deadline when the process is truly gone, so escalation still runs. - run() backs off to idle_poll_interval (5s) when fully idle instead of a 0.2s hot loop hammering the DB ~5x/sec.
…orker starvation Querying only `limit` rows then filtering out in-flight ids could return fewer (or zero) tasks when the oldest tasks are all in-flight, idling free workers. Fetch `limit + len(exclude_ids)` and slice the filtered result back to limit.
- _teardown_extractor_pool now bounds the join AFTER stop() too: a grandchild wedged in uninterruptible (D-state) IO survives SIGKILL, so an unbounded reap join would still hang. Abandon after EXTRACTOR_POOL_STOP_JOIN_TIMEOUT. - _qr_decode_zxing opens the image via `with` so its fd is released promptly.
…in test The pool loop moved into the engine modules, leaving time/TimeoutError/ free_space_monitor/TASK_COMPLETED/TASK_FAILED_PROCESSING/Task unused in process.py (ruff F401). Keep the pebble dep guard with a noqa. Split a two-on-one-line import in test_prefork_engine.py (E401).
…sweep init_routing() is called by cuckoo.py (scheduler), web settings (Django startup) and vpncheck. The nexthop stale-sweep flushed the per-task rule band on EVERY call, so a web restart mid-analysis tore down running tasks' egress. Gate all nexthop rooter mutations behind an opt-in apply flag; only the scheduler passes init_routing(apply_nexthop_state=True).
…state) init_rooter() is also called by web/web/settings.py before init_routing; its cleanup_rooter/ forward_drop/state_* reset CAPE-rooter iptables incl. live per-task nexthop rules. A web/gunicorn restart on a nexthop node therefore tore down running analyses. Gate the reset behind apply_state (default False); only the scheduler passes init_rooter(apply_state=True). Reachability still checked.
… longer spawns 50 threads Module-level ThreadPool(50) spawned 53 threads in every process that imported cleaners_utils, including the processor. PreforkEngine.run() imports it (for free_space_monitor) before the first fork, so those threads tripped the single-threaded-before-fork invariant — prefork couldn't launch a single task on any upstream-based checkout. Defer pool creation to first .map() use; only the cleaner/deletion paths need it. Prefork prerequisite.
reindex_screenshots' chronological mtime sort is unrelated to the processing engine and broke the upstream test (which mocks listdir/rename, not getmtime). Restore upstream's sorted(os.listdir(...)) behavior.
…bprocess The pytest process is multi-threaded (other suites leak clients/pools), so it can't satisfy the prefork single-threaded-before-fork invariant nor safely fork. Run PreforkEngine in a fresh interpreter via a helper; expire the parent session before reads (subprocess writes over a separate SQLite connection). Also stub free_space_monitor in the pebble test (run() sys.exit()s on missing storage/analyses).
…pg misses Mirror the _enforce_timeouts fallback in _escalate_kills: if a child hung before os.setsid() its process group won't exist, so killpg(SIGKILL) raises ProcessLookupError and the child would leak. Signal the pid directly instead.
…tory (default OFF) Adds an opt-in "central mode" that lets a CAPE web/API node serve a fleet of workers from a shared data plane, WITHOUT changing single-node behavior. Everything is gated by a new [central_mode] stanza that defaults to enabled = no, so a stock install is byte-for-byte unchanged (verified by an OFF-path + MT-absent import smoke test). What it adds (all lazy / import-optional): - [central_mode] config (conf/default/cuckoo.conf.default + central_mode.py): toggle + vendor-neutral backend settings. boto3 stays an optional, lazily-imported dependency (never core) — the same way the repo's existing modules/machinery/aws.py already treats it. - FS -> object-store artifact seam (storage_backend.py + artifact_storage.py, read side; modules/reporting/centralstore.py, write side). Artifacts key by <prefix>/<job_id>/<rel> on any S3-compatible store (AWS S3 / MinIO / Ceph) OR a shared local/NFS mount. The read seam lazily stages the tree back to the local storage/analyses/<task_id>/ dir so the many report features that read the local filesystem work unchanged. A completion marker (.centralstore.done) gates the read-side staging cache and the worker's local cleanup. - Pluggable job -> worker directory (job_directory.py + central_guac.py) so the central node can route interactive Guacamole to the worker hosting a live task's VM. Default backend is the vendor-neutral broker HTTP status API; an optional lazy-boto3 DynamoDB backend ships too. Worker access paths (api-token file, port, ssh user/keyfile) are config-driven. - tenancy-optional facades (tenancy_optional.py x2, analysis/central_scope.py) + a settings.py _users_app_present() shim. Central mode is ORTHOGONAL to multi-tenancy: the facades delegate to the multi-tenant `users` app when it is deployed and degrade to single-tenant see-all (fail-CLOSED on runtime errors) via ImportError when it is not — so this lands independently of any multi-tenant layer. - DocumentDB compatibility knobs in dev_utils/mongodb.py (explicit tls= / retryWrites=, both defaulting to today's behavior, documented in conf/default/reporting.conf.default) and a $facet -> per-category $group hunt aggregation (hunt_query.py) that Amazon DocumentDB accepts; the single-node $facet path is unchanged. The web view/endpoint changes (analysis/views.py, apiv2/views.py, guac/*, submission/views.py) are all `if central_mode_config().enabled:` seam sites with function-local imports; the OFF path is the original code (submission status/remote_session gate the worker-VM fallback on central mode, so single-node is byte-for-byte upstream — no degenerate empty-guest_ip session). guac/views.py additionally derives the interactive VM label + guest IP authoritatively from the task rather than trusting the session_data path segment (prevents a takeover/SSRF via a forged label/IP). Tests: tests/test_central_mode.py (config + store + directory + marker + hunt), tenancy_optional + central_scope facade tests (MT-absent behavior runs for real; MT-present delegation/fail-closed cases importorskip when the users app is absent), test_mt_absent_smoke.py (proves every touched view imports and the facades degrade with the MT layer absent) and test_settings_mt_optional.py.
- ruff E701/E702: expand the one-line helper classes in test_tenancy_optional (was failing CI).
- artifact_storage._job_id_for_task: coalesce a null info doc ({"info": None}) before .get (was
AttributeError), and cache task_id->job_id (immutable) so a view's repeated artifact_exists/stream
calls don't each re-query mongo. Bounded, scope-keyed.
- ensure_local_analysis / ensure_local_memory: let a clean Http404 propagate (so a direct view
caller returns 404 instead of a broken page) while still swallowing transient errors best-effort.
- central_guac._job_id_for_task + central_views.central_job_id_for_task: also accept the bare-token
custom form, matching centralstore.resolve_job_id; coalesce null info doc in the mongo fallback.
- S3Store.materialize: move tempfile.mkstemp inside the try so an OSError returns (None, False)
instead of escaping and breaking the caller contract.
_get_vnc_port gained an optional dsn param (local hypervisor, or a worker's libvirt-over-SSH in central mode) and the consumer now passes it positionally; the app-factory stub was 1-arg and raised 'takes 1 positional argument but 2 were given', failing the 5 connect() tests.
…ing) - _JOB_ID_CACHE: only cache the UNSCOPED (see-all / MT-absent / break-glass) resolution, keyed by task_id alone. task_id->job_id is immutable so that is safe forever, but a tenant-SCOPED lookup is authorization-sensitive (a task's tenant/visibility can be reassigned) — a process-lifetime cache could serve a job_id the caller may no longer see. Scoped lookups now always re-query; the repeated-lookup win is kept for the common single-tenant central deployment. - _stage_tree docstring: drop the stale 'never raises' claim — it intentionally lets Http404 from _store_and_container propagate so callers can return a clean 404 (ensure_local_* catch the rest).
…ravity review) - ensure_local_analysis / ensure_local_memory: after re-raising a clean Http404, log.warning the swallowed exception instead of dropping it silently — S3 creds/permission/network staging failures are now diagnosable (added a module logger). - _JOB_ID_CACHE: OrderedDict LRU (move_to_end on hit, popitem(last=False) at threshold) instead of clear-all, so hot recently-viewed tasks stay cached and eviction is one-at-a-time (no miss-storm).
…a shared global suricata.py appended the passlist file into the imported module-global domain_passlist_re on every run(). In a long-lived worker process (e.g. the pebble engine with maxtasksperchild=0, which never recycles a worker) the shared list grew by the whole passlist every task, so the per-event re.search loop became O(tasks_processed x events) and eventually stalled Suricata processing for minutes -- presenting as a hung/"deadlocked" task. Engines that use a fresh process per task reset the global and hid it. Fix: build the passlist per run via a new _compile_passlist() helper (local task_passlist_re) starting from a base safelist compiled once at module import, matching with compiled.search(). network.py had the same append into the same shared global at import time, which also polluted Suricata's copy with duplicates -- switched to a private, pre-compiled dns_passlist_re. Invalid passlist entries are skipped instead of raising. Adds regression tests asserting the shared global is never mutated and that filtering behaviour is unchanged; the network test forces a clean reload so the module's top-level build is actually re-run and measured.
…match, sys.path.insert in tests - network.py: log a warning when a base/file passlist regex fails to compile (was silently suppressed), matching suricata.py. - network.py: break out of the Pcap2 hostname passlist loop on first match. - tests: sys.path.insert(0, ...) so the local checkout takes import priority.
central-mode: optional object-store artifact seam + job->worker directory (default OFF)
…lation fix: build Suricata/network DNS passlist per-run instead of mutating a shared global
Pluggable processing engine: prefork option to fix nested-pool deadlocks (pebble default unchanged)
rooter: generic next-hop egress routing primitive (opt-in, fail-closed)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )