Skip to content

autodev-runbook.md

Latest commit

 

History

History
132 lines (115 loc) · 10.7 KB

autodev-runbook.md

File metadata and controls

132 lines (115 loc) · 10.7 KB
Raw

autodev runbook

autodev turns a labelled GitHub issue into a PR written by Claude Code in a sandboxed LSF job, revises it on request, and merges it after a human approves. Design: agentic-dev-infrastructure.md.

How people use it

you do autodev does
add autodev:go to an issue (needs write access) runs an agent round, opens PR autodev/issue-N, labels the issue autodev:review
review with Request changes, or comment /revise <what> (write access) runs another round on the same branch (up to max_revisions)
approve the PR (or comment /approve <sha> under a personal token, after the latest round's summary) squash-merges that exact commit, closes the issue
close the PR stops; re-add autodev:go to start over (closing the issue does not stop a running workflow)
sees autodev:needs-human read the failure comment, fix, re-add autodev:go (an open PR is resumed)

Trust boundaries:

  • Only people with write access can trigger (:go), approve, or request changes, and only their comments reach the agent's prompt. The issue body is included as-is, so edits to it after labelling are a known ceiling.
  • Agent commits touching .github/ are refused.
  • Permanent failures stop the round immediately; transient GitHub/csub/git errors retry for up to 6 h.
  • Agent summaries and log tails are posted with secret-shaped strings (sk-ant-…, ghp_…, JWTs, the worker's own GitHub token) replaced by [redacted]; that is a pattern filter, not a guarantee.
  • The Temporal dev server's gRPC port and web UI are unauthenticated: anyone with a login on the host can start, signal or terminate workflows. Run autodev on a host you don't share with untrusted users.

Run it as yourself (testing)

  1. Prerequisites P1-P4 in the plan. csub is pinned to krokicki/csub branch deploy-all (has the claude request field); pixi install gets it.
  2. cp autodev.yaml.example autodev.yaml and set deployment: autodev-<you> (keeps your labels, branches and workflows apart from the service and from colleagues), paths.work_root under your csub allowed_roots. Keep work_root and paths.state_dir apart: work_root must not be an ancestor of state_dir, because work_root is bound read-write into the sandbox, and <state_dir>-host (the broker log) and temporal.state_dir from mia.yaml (the Temporal binary and database) must stay outside every writable bind.
  3. gh auth token > ~/.config/autodev/token && chmod 600 ~/.config/autodev/token (or a fine-grained PAT with Contents, Pull requests and Issues read/write, and Checks and Commit statuses read), and set github.token_file: ~/.config/autodev/token. The sandboxed worker reads the token from this file (the sandbox masks .env); host-side commands like autodev check also accept .env / $GITHUB_TOKEN.
  4. pixi run mia exec autodev check: the temporal row is BAD until step 5 has started the dev server; fix any other BAD row first.
  5. pixi run mia exec autodev init-labels, then pixi run mia up. It starts, in tmux, a Temporal dev server (session temporal) and the pixi env's csub-broker (reading your ~/.config/csub/broker.toml policy) on the host, and the worker inside the sandbox. Re-run pixi run mia exec autodev check: now every row ok.
  6. Open a small issue ("add a --version flag"), add autodev-<you>:go, watch pixi run mia attach <service>.

Manage the services listed in mia.yaml: pixi run mia ps, mia logs <service> [worker|broker] (-n N, -f to follow), mia logs temporal, mia attach <service>, mia down [<service>...], mia restart [<service>...], mia up [<service>...]. Without a service name each command acts on everything, Temporal first on the way up and last on the way down.

Files: the Temporal CLI binary, temporal.db and its log live in temporal.state_dir from mia.yaml (default ~/.local/state/temporal); the broker log is in <state_dir>-host/logs, which the sandbox never binds; worker logs are in <state_dir>/logs.

Upgrading a host from the old up-scripts: the dev server's database used to be <autodev state_dir>-host/temporal.db. To keep in-flight workflows, stop the old tmux sessions (tmux kill-session -t <deployment>) and copy that file into temporal.state_dir before the first mia up, or point temporal.state_dir at the old -host directory. A fresh database is fine on a dev host: the poll schedule is re-applied and open issues are picked up on the next poll.

Under a personal token you are both the bot and the reviewer, so GitHub won't let you approve your own PR: comment /approve <first 7+ chars of the head sha> instead. Only /approve comments posted after the bot's latest round summary count, so a short prefix approved earlier cannot approve a later head.

Run it as the group service

  1. Service account on the cluster with rootless podman / bwrap set up (agentic-sandbox README), a csub broker installed for it (csub deploy/INSTALL.md), and an LSF billing group in its broker policy.
  2. Broker policy (~svc/.config/csub/broker.toml): [agents] claude = ["api.anthropic.com", "claude.ai", "platform.claude.com"]; allowed_hosts ⊇ agent.allow_hosts; allowed_roots contains paths.work_root; the CPU [queues.<name>] tables and cpu_short_queue/cpu_default_queue are named after queues bqueues lists (at Janelia short and local; a made-up name fails every submit with No such queue). Config rejects GitHub hosts in agent.allow_hosts, and agent.extra_mounts must be absolute and end in :ro.
  3. Claude for the service account: claude auth login as that account (a team seat). agent.api_key_env is refused by mia up for now.
  4. GitHub App owned by the org, installed on the repo, permissions: Contents RW, Pull requests RW, Issues RW, Checks R, Commit statuses R, Metadata R. Not Workflows, not Administration. Private key at ~svc/.config/autodev/app.pem (mode 600); github.auth: app, app_id, installation_id, private_key_path. Step by step: docs/github-app.md.
  5. Branch protection / ruleset on main: 1 approving review required, stale approvals dismissed on push, the App not in the bypass list. This is the real gate; autodev's own approval check is a pre-check. Add .github/CODEOWNERS (inside .github/, so agents cannot edit it) covering autodev/, mia_ops/ and .github/, so changes to the gates need a named human.
  6. As the service account: pixi run mia exec autodev init-labels, pixi run mia up, then pixi run mia exec autodev check.

Known gaps (deliberate, for later)

  • No cancel command: temporal workflow terminate -w <id> stops a workflow but leaves its job; csub kill <job>.
  • The agent's tests need tools inside the job (e.g. pixi on path_prepend + extra_mounts, conda/pypi in allow_hosts); nothing installs them for you.
  • CI failures after approval only escalate; there is no automatic fix-CI round.
  • Claude sessions are not resumed across rounds; each round starts fresh from the PR diff + feedback.
  • Temporal is a dev server under the running account; a shared production server is a later step.
  • autodev/workflows.py and autodev/decide.py are workflow code: changing their decisions or command order breaks in-flight workflows on replay. test_replay_recorded_history replays a recorded run (tests/fixtures/autodev_issue_history.json) and fails on such a change. If the change is intentional, guard it with workflow.patched() (there is no Worker Versioning yet) so runs in flight keep their old path, then re-record: AUTODEV_RECORD_HISTORY=1 pixi run pytest tests/test_autodev_workflows.py -k recorded_history.
  • [bot] logins are treated as having no permission (never looked up), so no bot can trigger, approve or revise.
  • The live end-to-end run (issue, PR, /revise, /approve) has not been done yet.

mia-data

Prerequisites P1-P4 (plan): P1 repos mia-workflow-data and mia-snippets with the GitHub App installed; P2 shared dirs for state_dir, work_root, storage_root (all under paths) visible from compute nodes; P3 csub broker policy ([broker] in broker.toml: env_allow includes "MIA_HARNESS_*"; allowed_roots covers harness.dir, work_root and storage_root, none of them under $HOME, which csub never mounts; [agents], data hosts, redirect probe; CPU queue tables named after real bqueues queues; mia exec mia-data check names what is missing); P4 see the plan.

  1. cp mia_data.yaml.example mia_data.yaml, fill in paths, harness.dir and a github.token_file (or app key). paths.work_root and paths.storage_root take a path or {path, url}; with a url (the folder in Fileglancer) the issue comments link the round's workspace and the dataset's crops folder. The data repo needs at least one commit on its default branch: the galleries branch is created through the trees API, which answers 409 on an empty repo.
  2. pixi run mia exec mia-data check verifies repos, Temporal, csub, dirs and the harness pin (and that cd harness && pixi install was done in harness.dir). Run on the host, mia exec mia-data check and mia exec mia-data test-pr need the mia-data broker: CSUB_TRANSPORT=unix CSUB_SOCKET=/scratch/$USER/mia-data/csub.sock pixi run mia exec mia-data check.
  3. pixi run mia up starts Temporal (unless mia.yaml says temporal.managed: false, in which case it only checks that the configured cluster answers), then each service: it ensures the service's namespace, starts its broker and sandboxed worker, and applies the poll schedule. Re-running it while a session exists only re-applies that service's schedule, so a run that failed partway is repaired by running it again. Uses temporal from PATH; a managed server downloads one into temporal.state_dir/bin otherwise. Manage with mia ps, mia logs mia-data [worker|broker] (-n N, -f), mia attach mia-data, mia down mia-data, mia restart mia-data (needed after a config change; wait for running rounds to finish first). mia down with no service stops everything, Temporal last; naming a service never touches Temporal.
  4. pixi run mia exec mia-data init-labels creates the data:* labels; pixi run mia exec mia-data seed seeds dataset issues from the catalog.
  5. pixi run mia exec mia-data test-pr <pr> --binding <file> --record <id> (all required) runs the crop check on a snippet PR's head with that binding and catalog record, and reports on the PR.
  6. Labels on a dataset issue: data:go starts a run; data:approve approves the snippet PR / crops for publish.