Epic: Implement Observability Mandate (State over Streams)

[kingdonb]

Overview

Following the "CloudWatch Lesson" and the recording of the Observability Mandate (State over Streams), we must transition Mecris from a system that relies on unstructured log files to a self-describing, state-driven architecture.

The goal is to move from "Grep for logs" to "Describe the state."

Philosophy: "If you need a log to know what happened, your system is opaque. If your system is observable, the state tells the story."

The Auditor's Corollary: "If you can't tell what happened from the log, your system is inscrutable to auditors. They will always use the logs."

Requirements

1. Database Schema Evolution

• Richer Heartbeats: Expand the scheduler_election table with the following columns:
  • last_status: (String) What the process did during its last turn (e.g., "Sent Walk Reminder", "Stood down (fresh Android heartbeat)", "Sync Complete").
  • last_error: (Text/JSON) Details of any failure that occurred.
  • intent: (String) What the process was trying to do.

2. Peer-to-Peer Reporting Responsibility

• Decentralized Status: Per the Peer Persistence model, every primary host (Local MCP, Fermyon Cloud, Akamai Functions) is responsible for reporting its own status directly to Neon.
• Resilience Telemetry: Verify that Local MCP can continue to record events/status in Neon even when Cloud APIs are unreachable.

3. Rust WASM Backend (Cloud Hub)

• Update sync-service/src/lib.rs to record every "Silent Decision" (especially stand-downs) into the scheduler_election.last_status field.
• Ensure API errors (Twilio, Beeminder) are captured in last_error so they appear in the system pulse without log access.

4. Local Host (Python MCP Peer)

• Update scheduler.py (Local Leader) to report its task-level status and intent.
• Ensure the Local MCP records its internal "Vacuumed Logic" fallbacks (e.g., "Cloud Timeout -> Falling back to legacy local logic").

5. Mobile Host & Standard Bus

• Update the Android HeartbeatWorker to report its internal state via the new status columns.
• Bus Standardization: Ensure the JSON responses on the "Standard Bus" for health/pulse include these new status and error fields.

6. Auditability & Persistence

• Structured Historical Logs: Ensure that the message_log and autonomous_turns tables serve as the definitive audit trail, capturing enough context that a third party can reconstruct system history without access to ephemeral stdout/stderr.

7. Observability Tools (The "Describe" Command)

• get_system_health: Update the MCP tool to return and format these new status/error fields.
• Android UI: Update the "System Pulse" screen/widget to display the last_status string for each modality (Fermyon, Akamai, Local).

References

• docs/OBSERVABILITY_MANDATE.md
• docs/AKAMAI_CRON_EVALUATION.md
• ARCHITECTURE.md (Peer Persistence Model)