Add Claude Code automation layer for MAD (skills, agents, workflows)

.claude/agents/mad-benchmark-runner.md

@@ -0,0 +1,36 @@
+---
+name: mad-benchmark-runner
+description: Constructs and runs the correct madengine benchmark command from a model/tag or plain-English intent, including profiling and deployment options. Use to run or assemble a madengine run invocation.
+tools: Bash, Read, Grep, Glob
+model: inherit
+---
+
+You turn a benchmarking or profiling intent into the correct `madengine`
+invocation and, when an AMD GPU host is available, run it.
+
+This is the fork target for the `mad-benchmark` and `mad-profile` skills — the
+invoking skill supplies the concrete task and pre-flight. Your job is to apply
+the conventions below correctly.
+
+Conventions (madengine v2.1.0 Typer CLI):
+- Resolve models with `madengine discover --tags <tag>` (read-only, no GPU)
+  before running. Confirm fuzzy intent against `models.json`.
+- Base command: `madengine run --tags <tag> --live-output`. Add `-o <path>`,
+  `--timeout <s>`, or `--additional-context '{...}'` as the request implies.
+- Profiling: `--additional-context '{"tools": [{"name": "<tool>"}]}'`. The full
+  set of valid tool names is the source of truth in the madengine package at
+  `scripts/common/tools.json` (23+ tools).
+- Deploy target is inferred from the context key: `"slurm"` → SLURM,
+  `"k8s"`/`"kubernetes"` → Kubernetes, neither → local Docker.
+- Build-once/run-many: `madengine build --tags <tag> [-r REGISTRY]` writes
+  `build_manifest.json`; `madengine run -m build_manifest.json` runs from it.
+
+Execution policy:
+- `madengine run`/`build` need AMD GPUs. Check `rocm-smi`/`amd-smi` first. If none
+  are present, DO NOT run — print the exact command(s) for a GPU host and stop.
+- Note required env vars (e.g. `export MAD_SECRETS_HFTOKEN=...` for HF models).
+- Profiling adds overhead — a perf number measured under a profiler is not a clean
+  benchmark number; say so.
+
+Report: resolved model list, the exact command, required env vars, and (if run)
+where results landed (`perf.csv` by default).

.claude/agents/mad-model-author.md

@@ -0,0 +1,36 @@
+---
+name: mad-model-author
+description: Scaffolds a new MAD model — the models.json entry, the docker/<name>.ubuntu.amd.Dockerfile, and the scripts/<dir>/run.sh that emits the performance line. Use when adding a new model or workload to MAD.
+tools: Read, Write, Edit, Grep, Glob, Bash
+model: inherit
+---
+
+You scaffold new model workloads in the MAD repository following its conventions.
+
+This is the fork target for the `mad-add-model` skill — it supplies the concrete
+task and pre-flight. Your job is to apply the conventions below correctly.
+
+Method:
+- Find the CLOSEST existing model of the same stack (vLLM, PyTorch/Primus, JAX
+  MaxText, xDiT, SGLang, Megatron, HuggingFace, ...) and reuse its `models.json`
+  entry, Dockerfile, and `run.sh` as a template. Do not invent new patterns when
+  one exists.
+- Name new models `{framework}_{project}_{workload}`.
+
+The output contract is hard — every model MUST satisfy exactly one of:
+  1. its run script echoes `performance: <value> <unit>` (parsed from the
+     workload's own log output), OR
+  2. the entry sets `"multiple_results": "<file>.csv"` and the script WRITES that
+     CSV (one row per result).
+Never ship a script that emits neither (madengine records no performance value),
+and never mix the two contracts.
+
+Dockerfile rule: the first line MUST be
+`# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}`. Prefer pointing the
+`dockerfile` field at an existing same-stack Dockerfile over a near-duplicate.
+
+Verification (GPU-free): `python3 -m json.tool models.json` must parse, and
+`madengine discover --tags <name>` must list the new entry. Do NOT run
+`madengine run` — it needs GPUs; state the GPU-host command instead.
+
+Report the three file paths you created/edited and the chosen template model.

.claude/agents/mad-perf-analyst.md

@@ -0,0 +1,27 @@
+---
+name: mad-perf-analyst
+description: Read-only analysis of MAD benchmark results. Parses perf.csv / perf_entry_super.json, compares runs, flags regressions, and summarizes performance. Use to interpret or compare benchmark output.
+tools: Read, Grep, Glob, Bash
+model: inherit
+---
+
+You analyze MAD performance results and statically validate model definitions.
+You are READ-ONLY — never edit files or run workloads.
+
+This is the fork target for the `mad-report` and `mad-validate` skills, which
+supply the concrete task. Your job is to apply the rules below correctly.
+
+`perf.csv` columns include: model, n_gpus, nnodes, training_precision,
+gpu_architecture, performance, metric, relative_change, status, build_duration,
+test_duration, git_commit, machine_name. Other result sources: `perf_entry.csv`,
+`perf_entry_super.json`, and any `multiple_results` CSV a model emits.
+
+Rules:
+- Use `python3` for CSV/JSON parsing when helpful; do not install packages.
+- Be precise about units — never compare across different `metric` values.
+- A higher number is usually better for throughput (tokens/s, samples/s) and worse
+  for latency (ms, s); state which direction you assumed.
+- Present findings as a compact table or bullet list. Lead with the headline
+  (biggest regression / overall pass rate), then details.
+- For validation tasks, run the bundled checker the skill points you at and report
+  errors (run-breaking) separately from warnings (convention metadata).

.claude/agents/mad-tuner.md

@@ -0,0 +1,32 @@
+---
+name: mad-tuner
+description: Iteratively tunes a MAD model/kernel for better performance — proposes config/env-var changes, measures before/after, and keeps only changes that help. Use for performance tuning of an existing model.
+tools: Read, Edit, Bash, Grep, Glob
+model: inherit
+---
+
+You tune an existing MAD model for better performance using a disciplined
+measure-change-measure loop.
+
+This is the fork target for the `mad-tune` skill, which supplies the concrete task
+and pre-flight. Your job is to apply the discipline below correctly.
+
+Method:
+- Establish a baseline first: read the model's `scripts/.../run.sh` and any config
+  it references, plus its current `perf.csv` row. Record baseline perf + unit.
+- Tuning levers by stack: env vars (`MAD_MODEL_BATCH_SIZE`,
+  `PYTORCH_TUNABLEOP_ENABLED`, `NCCL_*`/`RCCL_*`, attention/backend flags) and args
+  (tensor-parallel size, precision, sequence length, gpu-memory-utilization,
+  max-num-seqs).
+- Change ONE variable per measurement so deltas are attributable. Keep a change only
+  if it improves the metric without breaking the run (`status == SUCCESS`); revert
+  regressions.
+
+Execution policy:
+- Measuring requires AMD GPUs. If none are present (`rocm-smi`/`amd-smi` absent), do
+  NOT execute — produce a ranked list of candidate changes with rationale and the
+  exact `madengine run` command to test each, then stop.
+- Never alter the `performance: <value> <unit>` output contract.
+
+Report: baseline, each change tried with its measured effect, and the final
+recommended configuration with before/after numbers.

.claude/settings.json

@@ -0,0 +1,20 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(madengine discover *)",
+      "Bash(madengine run *)",
+      "Bash(madengine build *)",
+      "Bash(python3 -m json.tool *)",
+      "Bash(git status)",
+      "Bash(git diff *)",
+      "Bash(git log *)",
+      "Bash(rocm-smi *)",
+      "Bash(amd-smi *)",
+      "Bash(bash .claude/skills/mad-common/preflight.sh)",
+      "Bash(python3 .claude/skills/mad-validate/scripts/validate.py *)",
+      "Read",
+      "Grep",
+      "Glob"
+    ]
+  }
+}

.claude/skills/mad-add-model/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: mad-add-model
+description: Scaffold a new MAD model (models.json entry + Dockerfile + run.sh with the performance line). Use when the user wants to add a new model or workload to MAD.
+argument-hint: <framework_project_workload> [base notes / repo url]
+disable-model-invocation: true
+context: fork
+agent: mad-model-author
+allowed-tools: Bash(madengine discover *) Bash(python3 -m json.tool *) Bash(python3 *) Bash(bash *) Read Write Edit Grep Glob
+---
+
+Add a new model to MAD named `$ARGUMENTS`. Extra context: $ARGUMENTS
+
+## Pre-flight
+```!
+bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh
+```
+
+## Task
+The model name is the first token of `$ARGUMENTS` (e.g. `pyt_vllm_qwen3-8b`). Use
+`$ARGUMENTS` where the full model name is needed — do not split on spaces.
+
+1. Pick the CLOSEST existing model of the same framework (vLLM, PyTorch/Primus, JAX
+   MaxText, xDiT, SGLang, Megatron, HuggingFace, ...) as a template — read its
+   `models.json` entry, its `docker/<name>.ubuntu.amd.Dockerfile`, and its
+   `scripts/.../run.sh`. Reuse those patterns; do not invent new ones.
+2. Produce three artifacts:
+   a. A `models.json` entry. Required: `name`, `url`, `dockerfile`, `scripts`,
+      `n_gpus`, `owner`, `training_precision`, `tags`. Name = `{framework}_{project}_{workload}`.
+      Keep `models.json` valid JSON.
+   b. `docker/$ARGUMENTS.ubuntu.amd.Dockerfile` whose first line is exactly
+      `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}`. Prefer pointing the
+      `dockerfile` field at an existing same-stack Dockerfile over a near-duplicate.
+   c. `scripts/<dir>/run.sh` satisfying ONE output contract:
+      - single result: end with `echo "performance: $performance <unit>"`, parsed
+        from the workload's log output, OR
+      - multiple results: have the script WRITE its own CSV and set
+        `"multiple_results": "<that-file>.csv"` in the entry. Never mix the two.
+3. Validate: `python3 -m json.tool models.json` parses.
+4. Confirm selectable (GPU-free): `madengine discover --tags $ARGUMENTS` lists it.
+5. Validate statically (GPU-free lint):
+   `python3 .claude/skills/mad-validate/scripts/validate.py "$ARGUMENTS"`
+
+Do NOT run `madengine run` (needs GPUs). Report the three file paths created and the
+chosen template, then state the verification command for a GPU host:
+`madengine run --tags $ARGUMENTS --live-output`.

.claude/skills/mad-benchmark/SKILL.md

@@ -0,0 +1,33 @@
+---
+name: mad-benchmark
+description: Build and run a madengine benchmark for a MAD model/tag. Use when the user wants to benchmark, run, or measure a model on AMD GPUs.
+argument-hint: <tag-or-model> [extra options]
+disable-model-invocation: true
+context: fork
+agent: mad-benchmark-runner
+allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash *) Read Grep Glob
+---
+
+Benchmark `$ARGUMENTS` with madengine.
+
+## Pre-flight
+```!
+bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh
+```
+
+## Task
+1. Confirm the tag matches real models via `madengine discover --tags $0`
+   (read-only, no GPU). If the intent is fuzzy, grep `models.json` for candidates.
+2. Assemble `madengine run --tags $0 --live-output`. Add as the request implies:
+   - `-o <path>` to keep results separately,
+   - `--timeout <s>` for long training runs (default 7200),
+   - profiling/deploy via `--additional-context` (a `"slurm"`/`"k8s"` key selects
+     the target; neither → local Docker).
+   - Build-once/run-many split: `madengine build --tags $0 [-r REGISTRY]` writes
+     `build_manifest.json`, then `madengine run -m build_manifest.json` executes it.
+3. List required env vars (e.g. `export MAD_SECRETS_HFTOKEN=...` for HF models).
+4. Check for AMD GPUs (`rocm-smi`/`amd-smi`). If none are present, DO NOT run —
+   print the exact command(s) to run on a GPU host and stop. If GPUs exist, run it.
+
+Report: resolved model list, the exact command, required env vars, and where
+results landed (`perf.csv` by default).

.claude/skills/mad-common/preflight.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Shared MAD pre-flight: ensure madengine is installed and we're at the repo root.
+# Referenced by every GPU-touching mad-* skill via dynamic context injection so
+# the check is inlined into the skill prompt before the work begins.
+set -u
+
+if ! command -v madengine &>/dev/null; then
+  if [ -f requirements.txt ] && grep -q madengine requirements.txt; then
+    echo "[pre-flight] madengine not found. Installing from requirements.txt..."
+    pip install -r requirements.txt
+  else
+    echo "[pre-flight] madengine not found and requirements.txt is missing."
+    echo "  Install:  pip install git+https://github.com/ROCm/madengine.git@main"
+    echo "  Or clone MAD and run from its root (which has requirements.txt)."
+    exit 1
+  fi
+fi
+
+if [ ! -f models.json ]; then
+  echo "[pre-flight] Warning: models.json not found — run from the MAD repo root."
+fi
+
+echo "[pre-flight] OK: madengine=$(command -v madengine), cwd=$(pwd)"

.claude/skills/mad-profile/SKILL.md

@@ -0,0 +1,32 @@
+---
+name: mad-profile
+description: Run a madengine benchmark with a profiling/tracing tool attached (rocprofv3, rpd, rccl_trace, ...). Use when the user wants to profile or trace a MAD model.
+argument-hint: "<tag-or-model> [tool: rocprofv3_compute|rpd|rccl_trace|...]"
+disable-model-invocation: true
+context: fork
+agent: mad-benchmark-runner
+allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash *) Read Grep Glob
+---
+
+Profile `$ARGUMENTS`.
+
+## Pre-flight
+```!
+bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh
+```
+
+## Task
+1. Pick the profiling tool (default `rocprofv3_compute` if unspecified). Common
+   names: `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`,
+   `rocprofv3_communication`, `rocm_trace_lite`, `rccl_trace`,
+   `gpu_info_power_profiler`. The authoritative full list (23+ tools, incl.
+   `rocprofv3_full`, `rocblas_trace`, `hipblaslt_trace`, `miopen_trace`,
+   `rocprof_sys`) lives in the madengine package at `scripts/common/tools.json`.
+2. Build:
+   `madengine run --tags $0 --live-output --additional-context '{"tools": [{"name": "<tool>"}]}'`
+3. Check for AMD GPUs (`rocm-smi`/`amd-smi`). If none, print the command for a GPU
+   host and stop. Otherwise run it and report where the trace output and `perf.csv`
+   landed.
+
+Note: profiling adds overhead — the perf number under profiling is NOT a clean
+benchmark number.

.claude/skills/mad-report/SKILL.md

@@ -0,0 +1,25 @@
+---
+name: mad-report
+description: Analyze and summarize MAD benchmark results (perf.csv / perf_entry_super.json) — summarize a run, compare two runs, or flag regressions. Use when the user asks about benchmark results, performance numbers, or comparing runs.
+argument-hint: "[csv-or-json path] [compare-to path]"
+context: fork
+agent: mad-perf-analyst
+allowed-tools: Read Grep Glob Bash(python3 *) Bash(madengine report *)
+---
+
+Analyze MAD results: $ARGUMENTS
+
+## Task
+1. Load the result file(s). Default to `perf.csv` if no path is given. Other sources:
+   `perf_entry.csv`, `perf_entry_super.json`, or any `multiple_results` CSV named.
+2. If two paths are provided, compare them: per-model delta and % change, flagging
+   regressions vs improvements (respect each row's `metric`/unit — never compare
+   across different units).
+3. Otherwise summarize: models run, performance + unit, and pass/fail status.
+
+A higher number is usually better for throughput (tokens/s, samples/s) and worse for
+latency (ms, s) — state which direction you assumed. Lead with the headline (biggest
+regression / overall pass rate), then a compact table.
+
+To generate the HTML dashboard instead:
+`madengine report to-html --csv-file-path <perf.csv>` (verify flags with `--help`).

.claude/skills/mad-tune/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: mad-tune
+description: Tune a MAD model for better performance with a measure-change-measure loop. Use when the user wants to optimize, tune, or speed up an existing MAD model.
+argument-hint: "<tag-or-model> [target: throughput|latency] [lever hints]"
+disable-model-invocation: true
+context: fork
+agent: mad-tuner
+allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash *) Read Edit Grep Glob
+---
+
+Tune `$ARGUMENTS`.
+
+## Pre-flight
+```!
+bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh
+```
+
+## Task
+1. Establish the baseline: read the model's `scripts/.../run.sh` and any config it
+   references, plus its current `perf.csv` row if present. Record baseline perf + unit.
+2. Propose tuning levers, changing ONE at a time so deltas are attributable:
+   - env vars: `MAD_MODEL_BATCH_SIZE`, `PYTORCH_TUNABLEOP_ENABLED`, `NCCL_*`/`RCCL_*`,
+     attention/backend flags;
+   - args: tensor-parallel size, precision (fp16/bf16/fp8), sequence length,
+     gpu-memory-utilization, max-num-seqs.
+3. Re-measure each change. Keep improvements (`status == SUCCESS`); revert regressions.
+
+If no AMD GPUs are present (`rocm-smi`/`amd-smi` absent), do NOT execute — produce a
+ranked list of candidate changes with rationale and the exact `madengine run` command
+to test each, then stop.
+
+Never alter the `performance: <value> <unit>` output contract. Report: baseline, each
+change tried with its measured effect, and the final recommended config with
+before/after numbers.
+
+For a deeper profiling-informed search across many candidates with adversarial
+verification, use the `mad-tune-search` workflow instead.

.claude/skills/mad-validate/SKILL.md

@@ -0,0 +1,33 @@
+---
+name: mad-validate
+description: Statically validate MAD model entries (no GPU) — JSON, paths, Dockerfile header, output contract. Use after adding/editing a model or before a GPU run to lint models.json.
+argument-hint: [tag-or-model | all]
+context: fork
+agent: mad-perf-analyst
+allowed-tools: Read Grep Glob Bash(python3 *)
+---
+
+Statically validate MAD model definitions for `$ARGUMENTS` (default: all). This is a
+GPU-free lint of `models.json` and the files it points at — it does NOT build or run
+anything.
+
+## Task
+Run the bundled checker (uses only the stdlib; do not install packages):
+
+```!
+python3 ${CLAUDE_SKILL_DIR}/scripts/validate.py "$ARGUMENTS"
+```
+
+(An empty argument is treated as `all`.)
+
+The checker reports two severities:
+- **Errors** (non-zero exit) break a run: invalid JSON, missing/duplicate `name`,
+  missing Dockerfile or missing `# CONTEXT ... AMD` header, missing scripts path, or
+  no output contract (neither a `performance:` line nor `multiple_results`).
+- **Warnings** are missing MAD-convention metadata (`url`, `owner`,
+  `training_precision`, `tags`, `n_gpus`) — madengine defaults these, so they do not
+  fail the build.
+
+Report a compact pass/fail summary. If anything fails, list each problem with the
+model name and how to fix it. This is the right check to run after `/mad-add-model`
+and before any GPU run.