Skip to content

Add fault injection: simulated errors, timeouts and latency - #50

Merged
devinpearson merged 3 commits into
mainfrom
feat/fault-injection
Nov 9, 2026
Merged

devinpearson merged 3 commits into
mainfrom
feat/fault-injection

Conversation

@devinpearson

Copy link
Copy Markdown
Owner

Summary

Make the emulated bank misbehave on purpose, to test an app's error handling, retries and timeouts, which the official sandbox can't do. Faults apply to /za/... and the token endpoint, never to the sandbox's own controls, so they can always be turned off.

Per request (deterministic)

Header Effect
X-Sandbox-Fault: 429 / 500 / 502 / 503 / 504 That error with { errorMessage: "Simulated fault: …" }; 429 adds Retry-After: 1
X-Sandbox-Fault: timeout / timeout:2000 No response; the connection drops after 30 s (or the given ms)
X-Sandbox-Fault: reset The connection drops immediately
X-Sandbox-Latency: 1500 Delays the response

Unknown values get a 400 that explains the allowed ones.

At random

  • GET / PUT / DELETE /faults set errorRate with errorStatus, timeoutRate, latencyMs, and an optional pathPrefix (e.g. only /za/v1/cards).
  • The settings are saved across restarts and broadcast live to dashboards.

Visibility

  • Faulted responses carry X-Sandbox-Fault-Injected, and every fault, including dropped connections, appears in the live server log.

Dashboard

  • Fault injection panel, plus a warning banner with Turn off while faults are on.
  • The dashboard tags its requests X-Sandbox-Client: dashboard, so random faults don't break it while your app sees them. Explicit fault headers still apply.

Docs

The /faults endpoints are in OpenAPI, and the headers are described in the API description, so they're visible in Swagger. The README has a new section.

Test plan

  • API: 203 Jest tests. The new test/faults.spec.ts covers each status, reset and timeout really dropping the connection, latency adding time, the token endpoint, header validation, 100% and ~50% rates, the path prefix, the dashboard exemption, persistence, and that /faults itself is never faulted.
  • Web: 41 unit tests (describeFaults), lint, typecheck, build; 17 Playwright tests. New: turn faults on in the panel, an outside request gets 503 while the dashboard keeps working, Turn off from the banner.
  • openapi:check (64 routes), knip

🤖 Generated with Claude Code

Make the emulated bank misbehave on purpose so apps can test error
handling, retries and timeouts. Applies to /za and the token endpoint,
never to the sandbox's own controls.

- Per request: X-Sandbox-Fault: 429|500|502|503|504 (JSON errorMessage,
  Retry-After on 429), timeout or timeout:<ms> (hold, then drop the
  connection), reset (drop now); X-Sandbox-Latency: <ms>. Unknown values
  get a 400.
- At random: GET/PUT/DELETE /faults with errorRate + errorStatus,
  timeoutRate, latencyMs and an optional pathPrefix; saved in the Setting
  table across restarts and broadcast to dashboards.
- Faulted responses carry X-Sandbox-Fault-Injected and show in the live
  server log (new logRequest helper, also used by formatErrorResponse).
- The dashboard tags its requests X-Sandbox-Client: dashboard and is
  exempt from random faults; it gets a Fault injection panel and a banner
  with Turn off while faults are on.
- OpenAPI: the /faults endpoints, and the headers in the API description.

Tests: 20 API (each status, reset, timeout, latency, token endpoint,
validation, rates, path prefix, dashboard exemption, persistence),
2 unit, 1 e2e. 203 API, 41 unit and 17 e2e passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/faults.ts Fixed
Comment thread src/faults.ts Fixed
devinpearson and others added 2 commits November 9, 2026 18:23
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
- The CodeQL Autofix commit (5c30ac6) renamed the parsed
  X-Sandbox-Latency value to requestedLatencyMs but dropped the
  assignment to latencyMs, so the header was validated and then ignored.
  That is what failed the latency test in CI. Restored the assignment
  (keeping the validated, bounded name).
- CodeQL (js/resource-exhaustion): the timeout fault's setTimeout is now
  bounded where the timer is created, not only when parsing the header,
  and at most 25 requests are held open at once; more timeout faults drop
  the connection straight away (logged), so a client can't tie up the
  server with held connections.
- Fault tests time with performance.now() (monotonic) instead of
  Date.now().
- New test: 26 concurrent timeout faults, one dropped at once, 25 held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devinpearson
devinpearson merged commit 4261371 into main Nov 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants