Part of #283. Follows #299 (device-loss latch), which deliberately left two policy questions open. Both are about how much the machine should stop on events that are not a device losing power, and both are one-place changes once decided. Neither affects the correctness of what shipped.
1. Should provider loss latch its devices?
Current behaviour (#299): no. A supervised provider restart — the runtime's transport to the provider process failing, whether by crash, hang, or broken pipe — marks the provider's devices unavailable without engaging the loss latch. Automation resumes against them when the provider comes back. Rationale in the design: the boards did not lose power because their provider did, and supervised restarts happen in normal operation, so latching there would force an operator re-arm for events that are not stops.
The case for yes: a provider that hung because its I2C bus wedged leaves every board on that bus running its last command, and the runtime has no way to know what those commands were once the provider restarts. The reassert-on-return in #299 means the cost of latching is now a re-arm plus a safe-state drive, not a silent restart. Under that view, provider loss is at least as unknown a state as device loss, and treating it more leniently is backwards.
The case for no, restated: on a multi-day culture run, a provider restart at 03:00 that latches the impeller stops stirring until someone re-arms. That is a culture-at-risk outcome from an event that, on the reference rig, has never coincided with a device actually being in a bad state. Latching converts every provider hiccup into a stop.
Where it lives: StateCache::poll_device's provider_lost gate and mark_provider_devices_unavailable. Flipping the answer means routing both through the reachability sink with UNAVAILABLE and rewriting ProviderUnavailableDoesNotFireTheSink / OutOfBandFailureDoesNotFireTheSink to pin the new invariant.
Evidence that would settle it: how often supervised restarts actually occur over a culture run, and whether any has ever coincided with a board in an unexpected state. journalctl -u anolis-runtime | grep -i 'restart' over a run.
2. Should sustained device loss stop the whole machine?
Current behaviour (#299): no. A lost actuating device is blocked from automation until MANUAL -> AUTO; the tree keeps running against every other device. On this machine that means: lose the dose pump, stirring and temperature control continue with one output inert.
The alternative: escalate to FAULT when a latched device stays unavailable past the health STALE bound, or when every actuating device of a provider latches in one cycle (the Category 0 signature). The whole tree stops; the culture loses stirring and heating until an operator intervenes. The design's step 5, written as "build last, or not at all", with the explicit note that the threshold there only governs how loudly an outage is announced, never whether the machine can restart — which is what makes it tunable-wrong rather than fatally-wrong.
The question, verbatim from the design: when a single actuating device is lost in AUTO for a long time, should the whole machine stop (FAULT — stirring stops, culture at risk) or should only that device be blocked (tree continues, one output inert)? If "block only", step 5 is dropped and the shipped behaviour is complete. If "stop", step 5 is built through the #279 FAULT path, with the poll-thread stall bound the design calls out.
Related: #284 (the runtime does not know a stop happened) would give a cleaner trigger than a duration heuristic; the Cat-0 signature detection above is what is possible without it.
Not in scope
The call() path's shared-status race in CallRouter (pre-existing; only feeds an HTTP status mapping), the config-time check that provider dark-read latency stays under providers[].timeout_ms (noted in #299), and #300.
Part of #283. Follows #299 (device-loss latch), which deliberately left two policy questions open. Both are about how much the machine should stop on events that are not a device losing power, and both are one-place changes once decided. Neither affects the correctness of what shipped.
1. Should provider loss latch its devices?
Current behaviour (#299): no. A supervised provider restart — the runtime's transport to the provider process failing, whether by crash, hang, or broken pipe — marks the provider's devices unavailable without engaging the loss latch. Automation resumes against them when the provider comes back. Rationale in the design: the boards did not lose power because their provider did, and supervised restarts happen in normal operation, so latching there would force an operator re-arm for events that are not stops.
The case for yes: a provider that hung because its I2C bus wedged leaves every board on that bus running its last command, and the runtime has no way to know what those commands were once the provider restarts. The reassert-on-return in #299 means the cost of latching is now a re-arm plus a safe-state drive, not a silent restart. Under that view, provider loss is at least as unknown a state as device loss, and treating it more leniently is backwards.
The case for no, restated: on a multi-day culture run, a provider restart at 03:00 that latches the impeller stops stirring until someone re-arms. That is a culture-at-risk outcome from an event that, on the reference rig, has never coincided with a device actually being in a bad state. Latching converts every provider hiccup into a stop.
Where it lives:
StateCache::poll_device'sprovider_lostgate andmark_provider_devices_unavailable. Flipping the answer means routing both through the reachability sink withUNAVAILABLEand rewritingProviderUnavailableDoesNotFireTheSink/OutOfBandFailureDoesNotFireTheSinkto pin the new invariant.Evidence that would settle it: how often supervised restarts actually occur over a culture run, and whether any has ever coincided with a board in an unexpected state.
journalctl -u anolis-runtime | grep -i 'restart'over a run.2. Should sustained device loss stop the whole machine?
Current behaviour (#299): no. A lost actuating device is blocked from automation until
MANUAL -> AUTO; the tree keeps running against every other device. On this machine that means: lose the dose pump, stirring and temperature control continue with one output inert.The alternative: escalate to FAULT when a latched device stays unavailable past the health STALE bound, or when every actuating device of a provider latches in one cycle (the Category 0 signature). The whole tree stops; the culture loses stirring and heating until an operator intervenes. The design's step 5, written as "build last, or not at all", with the explicit note that the threshold there only governs how loudly an outage is announced, never whether the machine can restart — which is what makes it tunable-wrong rather than fatally-wrong.
The question, verbatim from the design: when a single actuating device is lost in AUTO for a long time, should the whole machine stop (FAULT — stirring stops, culture at risk) or should only that device be blocked (tree continues, one output inert)? If "block only", step 5 is dropped and the shipped behaviour is complete. If "stop", step 5 is built through the #279 FAULT path, with the poll-thread stall bound the design calls out.
Related: #284 (the runtime does not know a stop happened) would give a cleaner trigger than a duration heuristic; the Cat-0 signature detection above is what is possible without it.
Not in scope
The
call()path's shared-status race inCallRouter(pre-existing; only feeds an HTTP status mapping), the config-time check that provider dark-read latency stays underproviders[].timeout_ms(noted in #299), and #300.