From ab89818f1b4367750a48a21c2a77f154d89d6942 Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Sun, 31 May 2026 12:51:09 -0400 Subject: [PATCH 01/13] Add Claude Code agent and workflow presets for MAD Add a foundation pack so common MAD tasks (benchmarking, adding models, tuning, development) can be driven through Claude Code with the repo's conventions baked in: - CLAUDE.md: models.json schema, 4-step add-model flow, the "performance: " stdout contract, madengine v2.1.0 CLI commands, deployment inference, and profiling. - .claude/agents/: mad-model-author, mad-perf-analyst, mad-benchmark-runner, mad-tuner. - .claude/commands/: mad-add-model, mad-benchmark, mad-profile, mad-report, mad-tune. - .claude/workflows/: mad-benchmark-sweep and mad-tune-search dynamic workflows (plan-only by default; execute:true on a GPU host). - .claude/settings.json: shared read-only/common command allowlist. Commands and docs verified against the installed madengine Typer CLI v2.1.0 (@main): top-level build/run/discover/report/database. Co-Authored-By: Claude Opus 4 --- .claude/agents/mad-benchmark-runner.md | 37 +++++++ .claude/agents/mad-model-author.md | 39 +++++++ .claude/agents/mad-perf-analyst.md | 29 +++++ .claude/agents/mad-tuner.md | 32 ++++++ .claude/commands/mad-add-model.md | 16 +++ .claude/commands/mad-benchmark.md | 15 +++ .claude/commands/mad-profile.md | 19 ++++ .claude/commands/mad-report.md | 16 +++ .claude/commands/mad-tune.md | 18 +++ .claude/settings.json | 16 +++ .claude/workflows/mad-benchmark-sweep.js | 79 +++++++++++++ .claude/workflows/mad-tune-search.js | 134 +++++++++++++++++++++++ CLAUDE.md | 130 ++++++++++++++++++++++ 13 files changed, 580 insertions(+) create mode 100644 .claude/agents/mad-benchmark-runner.md create mode 100644 .claude/agents/mad-model-author.md create mode 100644 .claude/agents/mad-perf-analyst.md create mode 100644 .claude/agents/mad-tuner.md create mode 100644 .claude/commands/mad-add-model.md create mode 100644 .claude/commands/mad-benchmark.md create mode 100644 .claude/commands/mad-profile.md create mode 100644 .claude/commands/mad-report.md create mode 100644 .claude/commands/mad-tune.md create mode 100644 .claude/settings.json create mode 100644 .claude/workflows/mad-benchmark-sweep.js create mode 100644 .claude/workflows/mad-tune-search.js create mode 100644 CLAUDE.md diff --git a/.claude/agents/mad-benchmark-runner.md b/.claude/agents/mad-benchmark-runner.md new file mode 100644 index 0000000..5001719 --- /dev/null +++ b/.claude/agents/mad-benchmark-runner.md @@ -0,0 +1,37 @@ +--- +name: mad-benchmark-runner +description: Constructs and runs the correct madengine benchmark command from a model/tag or plain-English intent, including profiling and deployment options. Use to run or assemble a madengine run invocation. +tools: Bash, Read, Grep, Glob +model: inherit +--- + +You turn a benchmarking intent into the correct `madengine` invocation and, when +a GPU host is available, run it. + +When invoked: +1. Resolve which models the user means. Use `madengine discover --tags ` + (read-only, no GPU) to confirm tags match real models in `models.json`. If + the intent is fuzzy, grep `models.json` for candidates and confirm. +2. Build the command (madengine v2.1.0 Typer CLI): + - Base: `madengine run --tags --live-output` (full build+run). + - Output file: `-o ` when the user wants results kept separately. + - Timeout: `--timeout ` for long training runs (default 7200). + - Profiling: `--additional-context '{"tools": [{"name": ""}]}'` + (e.g. `rocprofv3_compute`, `rpd`, `rccl_trace`). + - Multi-node: add a `"slurm": {...}` or `"k8s": {...}` key to + `--additional-context` — presence of the key selects the target. + - Build/run split (build once, run many): `madengine build --tags + [-r REGISTRY]` writes `build_manifest.json`, then `madengine run -m + build_manifest.json` executes from it (skips rebuild). +3. Note required env vars (e.g. `export MAD_SECRETS_HFTOKEN=...` for HF models). + +Execution policy: +- `madengine run`/`build` need AMD GPUs. Before running, check for GPUs + (`rocm-smi` or `amd-smi`). If none are present, DO NOT run — instead print the + exact command(s) the user should run on a GPU host, and stop. +- Smoke-test wiring with a single small tag before large sweeps. (There is no + `dummy` model in this repo's `models.json` — confirm a real tag with + `madengine discover`.) + +Report: the resolved model list, the exact command, required env vars, and +(if run) where results landed (`perf.csv` by default). diff --git a/.claude/agents/mad-model-author.md b/.claude/agents/mad-model-author.md new file mode 100644 index 0000000..f2047f8 --- /dev/null +++ b/.claude/agents/mad-model-author.md @@ -0,0 +1,39 @@ +--- +name: mad-model-author +description: Scaffolds a new MAD model — the models.json entry, the docker/.ubuntu.amd.Dockerfile, and the scripts//run.sh that emits the performance line. Use when adding a new model or workload to MAD. +tools: Read, Write, Edit, Grep, Glob +model: inherit +--- + +You scaffold new model workloads in the MAD repository following its conventions. + +When invoked: +1. Identify the framework/stack the new model belongs to (vLLM, PyTorch/Primus, + JAX MaxText, xDiT, SGLang, Megatron, HuggingFace, ...). +2. Find the CLOSEST existing model of that stack in `models.json` and read its + entry, its `docker/.ubuntu.amd.Dockerfile`, and its `scripts/.../run.sh`. + Reuse that as your template — do not invent new patterns when one exists. +3. Produce three artifacts: + + a. A new `models.json` entry. Required fields: `name`, `url`, `dockerfile`, + `scripts`, `n_gpus`, `owner`, `training_precision`, `tags`. Name follows + `{framework}_{project}_{workload}`. Add appropriate tags (framework + + precision + model family). Keep `models.json` valid JSON. + + b. `docker/.ubuntu.amd.Dockerfile`. First line MUST be: + `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}` + Prefer reusing an existing Dockerfile of the same stack (point the + `dockerfile` field at it) rather than creating a near-duplicate. + + c. `scripts//run.sh`. It MUST end by printing exactly: + `echo "performance: $performance "` + where `$performance` is parsed from the workload's own log output. Model + this parsing on the template script. + +Rules: +- The `performance: ` stdout line is the hard contract madengine + relies on. Never omit it. +- Validate the final `models.json` parses (`python3 -m json.tool models.json`). +- Do not run `madengine` (it needs GPUs). State the verification command the + user should run: `madengine run --tags --live-output`. +- Report the three file paths you created/edited and the chosen template model. diff --git a/.claude/agents/mad-perf-analyst.md b/.claude/agents/mad-perf-analyst.md new file mode 100644 index 0000000..c01932a --- /dev/null +++ b/.claude/agents/mad-perf-analyst.md @@ -0,0 +1,29 @@ +--- +name: mad-perf-analyst +description: Read-only analysis of MAD benchmark results. Parses perf.csv / perf_entry_super.json, compares runs, flags regressions, and summarizes performance. Use to interpret or compare benchmark output. +tools: Read, Grep, Glob, Bash +model: inherit +--- + +You analyze MAD performance results. You are READ-ONLY — never edit files or run +workloads. + +When invoked: +1. Locate result files: `perf.csv`, `perf_entry.csv`, `perf_entry_super.json`, + or any CSV the user names (model scripts may emit `multiple_results` CSVs). +2. Parse the data. `perf.csv` columns include: model, n_gpus, nnodes, + training_precision, gpu_architecture, performance, metric, relative_change, + status, build_duration, test_duration, git_commit, machine_name. +3. Answer the question asked — typically one of: + - Summarize: which models ran, their performance + unit, pass/fail status. + - Compare two result sets: per-model delta and % change; call out + regressions (slower) vs improvements clearly. + - Diagnose failures: surface `status != SUCCESS` rows and any error context. + +Rules: +- Use `python3` for CSV/JSON parsing when helpful; do not install packages. +- Be precise about units — never compare across different `metric` values. +- A higher number is usually better for throughput (tokens/s, samples/s) and + worse for latency (ms, s); state which direction you assumed. +- Present findings as a compact table or bullet list. Lead with the headline + (biggest regression / overall pass rate), then details. diff --git a/.claude/agents/mad-tuner.md b/.claude/agents/mad-tuner.md new file mode 100644 index 0000000..5f4c91e --- /dev/null +++ b/.claude/agents/mad-tuner.md @@ -0,0 +1,32 @@ +--- +name: mad-tuner +description: Iteratively tunes a MAD model/kernel for better performance — proposes config/env-var changes, measures before/after, and keeps only changes that help. Use for performance tuning of an existing model. +tools: Read, Edit, Bash, Grep, Glob +model: inherit +--- + +You tune an existing MAD model for better performance using a disciplined +measure-change-measure loop. + +When invoked: +1. Establish the baseline. Read the model's `scripts/.../run.sh` and any config + it references (YAML/JSON), plus its current `perf.csv` row if present. Record + the baseline performance + unit. +2. Identify tuning levers for the stack, e.g.: + - Env vars: batch size (`MAD_MODEL_BATCH_SIZE`), `HIP_VISIBLE_DEVICES`, + `NCCL_*`/`RCCL_*`, `PYTORCH_TUNABLEOP_ENABLED`, attention/backend flags. + - Script/config args: tensor-parallel size, precision (fp16/bf16/fp8), + sequence length, gpu-memory-utilization, max-num-seqs. +3. Propose ONE change at a time (or a small named set), explain the hypothesis, + apply it, and re-measure. +4. Keep a change only if it improves the metric without breaking the run + (`status == SUCCESS`). Revert regressions. + +Rules: +- Measuring requires AMD GPUs. If none are present (`rocm-smi`/`amd-smi` absent), + do NOT execute — instead produce a ranked list of candidate changes with + rationale and the exact `madengine run` commands to test each, then stop. +- Change one variable per measurement so deltas are attributable. +- Never alter the `performance: ` output contract. +- Report: baseline, each change tried, its measured effect, and the final + recommended configuration with before/after numbers. diff --git a/.claude/commands/mad-add-model.md b/.claude/commands/mad-add-model.md new file mode 100644 index 0000000..9765616 --- /dev/null +++ b/.claude/commands/mad-add-model.md @@ -0,0 +1,16 @@ +--- +description: Scaffold a new MAD model (models.json entry + Dockerfile + run.sh with the performance line) +argument-hint: [base notes / repo url] +--- + +Add a new model to MAD named `$1`. Extra context: $ARGUMENTS + +Use the `mad-model-author` subagent. It should: +1. Pick the closest existing model of the same framework as a template. +2. Create the `models.json` entry, `docker/$1.ubuntu.amd.Dockerfile` (with the + `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}` header), and + `scripts//run.sh` ending in `echo "performance: $performance "`. +3. Validate `models.json` with `python3 -m json.tool models.json`. + +Report the files created and the verification command +`madengine run --tags $1 --live-output` (requires a GPU host). diff --git a/.claude/commands/mad-benchmark.md b/.claude/commands/mad-benchmark.md new file mode 100644 index 0000000..28df6d0 --- /dev/null +++ b/.claude/commands/mad-benchmark.md @@ -0,0 +1,15 @@ +--- +description: Build and run a madengine benchmark for the given tag/model +argument-hint: [extra options] +--- + +Benchmark `$ARGUMENTS` with madengine. + +Use the `mad-benchmark-runner` subagent. It should: +1. Confirm the tag matches real models via `madengine discover --tags $1`. +2. Assemble the `madengine run --tags $1 --live-output` command (add `-o`, + `--timeout`, profiling `--additional-context`, or `slurm`/`k8s` keys as the + request implies), and list required env vars (e.g. `MAD_SECRETS_HFTOKEN`). +3. Check for GPUs first. If none, print the exact command(s) to run on a GPU + host instead of executing. If GPUs exist, run it and report where `perf.csv` + landed. diff --git a/.claude/commands/mad-profile.md b/.claude/commands/mad-profile.md new file mode 100644 index 0000000..b3214b7 --- /dev/null +++ b/.claude/commands/mad-profile.md @@ -0,0 +1,19 @@ +--- +description: Run a madengine benchmark with a profiling/tracing tool attached +argument-hint: [tool: rocprofv3_compute|rpd|rccl_trace|...] +--- + +Profile `$ARGUMENTS`. + +Use the `mad-benchmark-runner` subagent with profiling enabled. It should: +1. Pick the profiling tool (default `rocprofv3_compute` if unspecified). Valid + names include `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`, + `rocprofv3_communication`, `rocm_trace_lite`, `rccl_trace`, + `gpu_info_power_profiler`. +2. Build: + `madengine run --tags $1 --live-output --additional-context '{"tools": [{"name": ""}]}'` +3. Check for GPUs; if none, print the command for a GPU host. Otherwise run it + and report where the trace/profile output and `perf.csv` landed. + +Note: profiling adds overhead; the perf number under profiling is not a clean +benchmark number. diff --git a/.claude/commands/mad-report.md b/.claude/commands/mad-report.md new file mode 100644 index 0000000..5bbb74b --- /dev/null +++ b/.claude/commands/mad-report.md @@ -0,0 +1,16 @@ +--- +description: Analyze and summarize MAD benchmark results (perf.csv / perf_entry_super.json) +argument-hint: [csv-or-json path] [compare-to path] +--- + +Analyze MAD results: $ARGUMENTS + +Use the `mad-perf-analyst` subagent (read-only). It should: +1. Load the result file(s). Default to `perf.csv` if no path is given. +2. If two paths are provided, compare them: per-model delta and % change, + flagging regressions vs improvements (respecting each row's `metric`/unit). +3. Otherwise summarize: models run, performance + unit, and pass/fail status. + +Lead with the headline finding, then a compact table. To generate the HTML +dashboard instead, run `madengine report to-html --csv-file-path ` +(verify flags with `madengine report to-html --help`). diff --git a/.claude/commands/mad-tune.md b/.claude/commands/mad-tune.md new file mode 100644 index 0000000..185c307 --- /dev/null +++ b/.claude/commands/mad-tune.md @@ -0,0 +1,18 @@ +--- +description: Tune a MAD model for better performance with a measure-change-measure loop +argument-hint: [target: throughput|latency] [lever hints] +--- + +Tune `$ARGUMENTS`. + +Use the `mad-tuner` subagent. It should: +1. Establish the baseline (current `run.sh`/config + `perf.csv` row). +2. Propose tuning levers (env vars like `MAD_MODEL_BATCH_SIZE`, + `PYTORCH_TUNABLEOP_ENABLED`, `NCCL_*`/`RCCL_*`; or args like tensor-parallel + size, precision, gpu-memory-utilization), changing ONE at a time. +3. Re-measure each change; keep improvements, revert regressions. + +If no GPUs are present, produce a ranked list of candidate changes with rationale +and the exact `madengine run` command to test each, then stop. Report baseline, +changes tried with measured effect, and the final recommended config with +before/after numbers. diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..a4299f5 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,16 @@ +{ + "permissions": { + "allow": [ + "Bash(madengine discover *)", + "Bash(python3 -m json.tool *)", + "Bash(git status)", + "Bash(git diff *)", + "Bash(git log *)", + "Bash(rocm-smi *)", + "Bash(amd-smi *)", + "Read", + "Grep", + "Glob" + ] + } +} diff --git a/.claude/workflows/mad-benchmark-sweep.js b/.claude/workflows/mad-benchmark-sweep.js new file mode 100644 index 0000000..53d7e4d --- /dev/null +++ b/.claude/workflows/mad-benchmark-sweep.js @@ -0,0 +1,79 @@ +export const meta = { + name: 'mad-benchmark-sweep', + description: 'Fan out a benchmark matrix across MAD models/tags, run (or plan) each with madengine, parse results, and synthesize a comparison table', + phases: [ + { title: 'Resolve', detail: 'expand the sweep matrix into concrete madengine invocations' }, + { title: 'Run', detail: 'execute or plan each benchmark in parallel' }, + { title: 'Synthesize', detail: 'parse results into a comparison table' }, + ], +} + +// args: { tags: string[], precisions?: string[], nGpus?: string[], execute?: boolean, output?: string } +// Without execute:true (default) agents PLAN and validate commands only — safe on GPU-less hosts. +const cfg = args || {} +const tags = Array.isArray(cfg.tags) ? cfg.tags : (cfg.tags ? [cfg.tags] : ['bert']) +const precisions = cfg.precisions && cfg.precisions.length ? cfg.precisions : [null] +const nGpus = cfg.nGpus && cfg.nGpus.length ? cfg.nGpus : [null] +const execute = cfg.execute === true + +// Build the matrix of {tag, precision, nGpus} cells. +const matrix = [] +for (const tag of tags) + for (const precision of precisions) + for (const g of nGpus) + matrix.push({ tag, precision, nGpus: g }) + +log(`Sweep matrix: ${matrix.length} cell(s) across ${tags.length} tag(s). execute=${execute}`) + +const CELL_SCHEMA = { + type: 'object', + required: ['cell', 'command', 'status'], + properties: { + cell: { type: 'string' }, + command: { type: 'string' }, + status: { type: 'string', enum: ['planned', 'ran', 'skipped', 'error'] }, + performance: { type: ['number', 'null'] }, + metric: { type: ['string', 'null'] }, + notes: { type: 'string' }, + }, +} + +const results = await parallel(matrix.map((cell, i) => () => { + const label = `${cell.tag}${cell.precision ? '/' + cell.precision : ''}${cell.nGpus ? '/' + cell.nGpus + 'gpu' : ''}` + const flags = [ + '--tags ' + cell.tag, + '--live-output', + cfg.output ? '-o ' + cfg.output : '', + ].filter(Boolean).join(' ') + const ctxParts = [] + if (cell.nGpus) ctxParts.push(`"n_gpus": "${cell.nGpus}"`) + const ctx = ctxParts.length ? ` --additional-context '{${ctxParts.join(', ')}}'` : '' + const cmd = `madengine run ${flags}${ctx}` + + const action = execute + ? `If AMD GPUs are present (check rocm-smi/amd-smi), RUN this command and parse the "performance: " line from output. If no GPUs, set status "skipped".` + : `Do NOT execute. Validate the command is well-formed and the tag resolves via "madengine discover --tags ${cell.tag}". Set status "planned".` + + return agent( + `MAD benchmark sweep cell "${label}". +Command: ${cmd} +${action} +${cell.precision ? 'Intended precision: ' + cell.precision + ' (confirm the tag/model supports it).' : ''} +Return the cell label, the exact command, status, and performance/metric if available.`, + { label: `bench:${label}`, phase: 'Run', schema: CELL_SCHEMA } + ).then(r => ({ ...r, _label: label })) +})) + +phase('Synthesize') +const clean = results.filter(Boolean) +const summary = await agent( + `Synthesize a MAD benchmark sweep into a markdown comparison table. +Cells (JSON): ${JSON.stringify(clean)} +Produce: (1) a table with columns Cell | Command | Status | Performance | Metric; +(2) a short headline (how many planned/ran/skipped/errored); +(3) if any ran, the fastest and slowest cells (respecting that metric units differ). +Do not invent numbers — only use provided data.`, + { phase: 'Synthesize' } +) + +return { execute, cells: clean, report: summary } diff --git a/.claude/workflows/mad-tune-search.js b/.claude/workflows/mad-tune-search.js new file mode 100644 index 0000000..29a71ca --- /dev/null +++ b/.claude/workflows/mad-tune-search.js @@ -0,0 +1,134 @@ +export const meta = { + name: 'mad-tune-search', + description: 'Generate N candidate tuning configs for one MAD model, run (or plan) each, adversarially verify perf deltas, and synthesize the winning config', + phases: [ + { title: 'Baseline', detail: 'read the model + establish the baseline config' }, + { title: 'Propose', detail: 'generate distinct tuning candidates' }, + { title: 'Evaluate', detail: 'run or plan each candidate and verify its delta' }, + { title: 'Synthesize', detail: 'pick the winning configuration' }, + ], +} + +// args: { tag: string, target?: 'throughput'|'latency', candidates?: number, execute?: boolean } +const cfg = args || {} +const tag = cfg.tag || 'bert' +const target = cfg.target || 'throughput' +const nCandidates = Math.max(1, Math.min(cfg.candidates || 4, 8)) +const execute = cfg.execute === true + +phase('Baseline') +const BASELINE_SCHEMA = { + type: 'object', + required: ['model', 'scriptPath', 'baselineSummary'], + properties: { + model: { type: 'string' }, + scriptPath: { type: 'string' }, + baselineSummary: { type: 'string' }, + levers: { type: 'array', items: { type: 'string' } }, + }, +} +const baseline = await agent( + `In the MAD repo, establish the tuning baseline for tag "${tag}". +Find its models.json entry, its scripts/.../run.sh, and any config it references. +Summarize the current configuration and list the tuning levers available for this +stack (env vars like MAD_MODEL_BATCH_SIZE / PYTORCH_TUNABLEOP_ENABLED / NCCL_*, +or args like tensor-parallel size, precision, gpu-memory-utilization). +Optimization target: ${target}.`, + { phase: 'Baseline', schema: BASELINE_SCHEMA } +) + +phase('Propose') +const CANDS_SCHEMA = { + type: 'object', + required: ['candidates'], + properties: { + candidates: { + type: 'array', + items: { + type: 'object', + required: ['id', 'change', 'hypothesis'], + properties: { + id: { type: 'string' }, + change: { type: 'string' }, + hypothesis: { type: 'string' }, + }, + }, + }, + }, +} +const proposed = await agent( + `Propose ${nCandidates} DISTINCT tuning candidates to improve ${target} for model +"${baseline.model}". Baseline: ${baseline.baselineSummary} +Available levers: ${(baseline.levers || []).join(', ')} +Each candidate changes ONE lever (so its effect is attributable). Return id, +change (concrete value + how to apply it), and hypothesis.`, + { phase: 'Propose', schema: CANDS_SCHEMA } +) +const candidates = (proposed.candidates || []).slice(0, nCandidates) +log(`Evaluating ${candidates.length} candidate(s). execute=${execute}`) + +// Pipeline: evaluate each candidate, then adversarially verify its claimed delta. +const EVAL_SCHEMA = { + type: 'object', + required: ['id', 'status'], + properties: { + id: { type: 'string' }, + status: { type: 'string', enum: ['planned', 'ran', 'skipped', 'error'] }, + command: { type: 'string' }, + performance: { type: ['number', 'null'] }, + metric: { type: ['string', 'null'] }, + notes: { type: 'string' }, + }, +} +const VERDICT_SCHEMA = { + type: 'object', + required: ['id', 'trustworthy'], + properties: { + id: { type: 'string' }, + trustworthy: { type: 'boolean' }, + reason: { type: 'string' }, + }, +} + +const evaluated = await pipeline( + candidates, + (cand) => { + const action = execute + ? `If AMD GPUs are present, apply the change, run "madengine run --tags ${tag} --live-output", parse the performance line, then REVERT the change. If no GPUs, status "skipped".` + : `Do NOT execute. Produce the exact "madengine run --tags ${tag} ..." command (plus how to apply the change) and set status "planned".` + return agent( + `Evaluate tuning candidate ${cand.id} for "${baseline.model}". +Change: ${cand.change} +Hypothesis: ${cand.hypothesis} +${action} +Return id, status, command, and performance/metric if measured.`, + { label: `eval:${cand.id}`, phase: 'Evaluate', schema: EVAL_SCHEMA } + ) + }, + (evalResult, cand) => + agent( + `Adversarially review tuning candidate ${cand.id}. +Claimed result: ${JSON.stringify(evalResult)} +Hypothesis was: ${cand.hypothesis} +Is the conclusion trustworthy? Default to trustworthy=false if the change was +not actually measured (planned/skipped), if only one sample was taken, or if the +delta is within noise. Return id, trustworthy, reason.`, + { label: `verify:${cand.id}`, phase: 'Evaluate', schema: VERDICT_SCHEMA } + ).then(v => ({ ...evalResult, verdict: v })) +) + +phase('Synthesize') +const clean = evaluated.filter(Boolean) +const report = await agent( + `Synthesize a MAD tuning search for "${baseline.model}" (target: ${target}). +Baseline: ${baseline.baselineSummary} +Evaluated candidates (JSON, each with a verdict): ${JSON.stringify(clean)} +Produce: (1) a table Candidate | Change | Status | Performance | Trustworthy; +(2) the recommended configuration — only from candidates whose verdict is +trustworthy; if none ran, present the ranked plan to test on a GPU host; +(3) the exact next command(s) to run. +Do not invent measurements.`, + { phase: 'Synthesize' } +) + +return { model: baseline.model, target, execute, candidates: clean, report } diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..b40824c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,130 @@ +# CLAUDE.md + +Guidance for Claude Code when working in the MAD repository. + +## What MAD is + +MAD (Model Automation and Dashboarding) is a curated catalog of ~142 AI model +workloads that run on AMD Instinct GPUs via the **`madengine`** CLI. Each model +is one JSON entry in `models.json`. `madengine run --tags ` builds a Docker +image, runs the model in a container, parses a performance number from stdout, +and appends a row to `perf.csv`. + +The four common user tasks here are: **benchmarking**, **adding a new model**, +**tuning** a model/kernel for better perf, and **development** on the repo +itself. Dedicated subagents and `/mad-*` slash commands exist for each — see +`.claude/agents/` and `.claude/commands/`. + +## The performance contract (most important convention) + +A model's run script MUST print exactly one line to stdout: + +``` +performance: +``` + +madengine greps this line to populate the `performance` and `metric` columns of +`perf.csv`. A run that does not emit it is recorded with no performance value. +Real example (`scripts/huggingface_bert/run.sh`): + +```bash +performance=$(grep -Eo "train_samples_per_second':[^,]+" log.txt | sed "s/.*: //g" | head -n 1) +echo "performance: $performance samples_per_second" +``` + +If a script produces many results, set `multiple_results` in `models.json` to the +CSV filename the script writes (e.g. `"multiple_results": "perf_DeepSeek-R1.csv"`). + +## models.json schema + +One object per model. Required fields: `name`, `url`, `dockerfile`, `scripts`, +`n_gpus`, `owner`, `training_precision`, `tags`. Optional: `data`, `timeout`, +`multiple_results`, `args`, `skip_gpu_arch`. + +| Field | Notes | +|-------|-------| +| `name` | Unique. Convention: `{framework}_{project}_{workload}`, e.g. `pyt_vllm_deepseek-r1` | +| `url` | Git repo cloned into the container; `""` if none | +| `dockerfile` | Path prefix; engine appends `.ubuntu.amd.Dockerfile` → `docker/.ubuntu.amd.Dockerfile` | +| `scripts` | Path to a `run.sh` (or script dir) executed inside the container | +| `n_gpus` | String. `"-1"` = all available | +| `tags` | List; `--tags` matches any tag OR the exact `name` | +| `training_precision` | e.g. `fp16`, `bf16`, `fp8`, `fp32`, or `""` | +| `timeout` | Seconds; overrides the 7200s default. `-1` = no timeout | +| `skip_gpu_arch` | Skip on these archs, e.g. `"gfx942"` | +| `args` | Extra args appended to the run script | + +`models.json` must stay valid JSON — validate with `python3 -m json.tool models.json` +after editing. + +## Adding a new model (4 steps) + +1. **Name** it `{framework}_{project}_{workload}`. +2. **Add** an entry to `models.json` (copy the closest existing model of the same + framework as a template). +3. **Dockerfile** `docker/.ubuntu.amd.Dockerfile`. First line MUST be the + context header, then a base image: + ```dockerfile + # CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'} + ARG BASE_DOCKER=rocm/pytorch + FROM $BASE_DOCKER + ``` + Reuse an existing `docker/*.Dockerfile` of the same stack when possible rather + than inventing a new base. +4. **Script** `scripts//run.sh` that runs the workload and ends by echoing + the `performance: ` line. + +Verify (needs a GPU host): `madengine run --tags --live-output`. + +## madengine commands + +`requirements.txt` pins madengine to `@main` (Typer CLI, v2.1.0). It exposes +five commands: `build`, `run`, `discover`, `report`, `database`. Verify exact +flags with `madengine --help`. + +- `madengine discover --tags ` — list matching models (read-only, no GPU). +- `madengine build --tags [-r REGISTRY] [-a gfx942,gfx90a] [-l]` — build images, write `build_manifest.json`. `--use-image` skips the build and uses a prebuilt image. +- `madengine run --tags [-l/--live-output] [-o out.csv] [--timeout S] [--keep-alive] [--skip-model-run] [-c '']` — full build+run (or, with `-m manifest.json`, execution-only). Writes `perf.csv`. +- `madengine report to-html --csv-file-path perf.csv` — HTML report. +- `madengine database -f perf_entry_super.json --db DB -c COLLECTION` — upload CSV/JSON to MongoDB (`-k model,timestamp` sets the unique key; `--dry-run` to preview). + +`--skip-model-run` builds the image without running the workload. +`tools/run_models.py` is the **deprecated** legacy runner — prefer `madengine`. + +## Deployment target (convention over configuration) + +Pass deployment intent via `--additional-context` (a Python-dict/JSON string, +parsed with `ast.literal_eval` so Python dict syntax is fine). The target is +inferred from which key is present: + +- `"slurm"` key → SLURM. `"k8s"`/`"kubernetes"` key → Kubernetes. Neither → local Docker. + +## Profiling and tracing + +Add a `tools` list to `--additional-context`: + +```bash +madengine run --tags --additional-context '{"tools": [{"name": "rocprofv3_compute"}]}' +``` + +Common tool names: `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`, +`rocprofv3_communication`, `rocm_trace_lite`, `rccl_trace`, `gpu_info_power_profiler`. +Pre-built profiling context files live in the madengine package's +`examples/profiling-configs/`. + +## Outputs + +- `perf.csv` — one row per run (model, n_gpus, precision, performance, metric, status, durations, git_commit, gpu_architecture, ...). +- `perf_entry_super.json` — enriched record incl. a `configs` block; this is what gets pushed to MongoDB. + +## Environment variables + +`MAD_SECRETS_HFTOKEN` (HuggingFace token), `MAD_MODEL_NAME`, `MAD_RUNTIME_NGPUS`, +`MAD_SYSTEM_GPU_ARCHITECTURE`, `MAD_MODEL_BATCH_SIZE`. + +## Future (skeleton, not yet wired) + +`reference_db/mad_agent.db` (tables: `model_baselines`, `optimization_history`, +`best_configurations`, `learned_patterns`) and `knowledge_base/` exist as an +empty scaffold for a future persistent optimization-memory layer. Not populated +yet — do not assume data is present. From 771f129959beeca4ac545203aadba7290b47cbca Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Sun, 31 May 2026 22:43:32 +0000 Subject: [PATCH 02/13] Fix MAD automation bugs and add static model validator - mad-benchmark-sweep: drop the non-functional precision axis (precision is fixed per-model via training_precision or baked into the inference image, not a runtime flag) and give each cell its own perf_.csv to avoid clobbering a shared perf.csv under parallel execution; flag unresolved/errored cells. - mad-model-author: document the multiple_results output contract alongside the performance: stdout line, and confirm new entries resolve via discover. - Add /mad-validate: GPU-free static checker (JSON, paths, Dockerfile CONTEXT header, output contract) with errors vs convention-warning severities. - Point profiling agent/command at scripts/common/tools.json as the source of truth and document the deploy-key convention. Co-Authored-By: Claude Opus 4 --- .claude/agents/mad-benchmark-runner.md | 8 ++- .claude/agents/mad-model-author.md | 24 ++++--- .claude/commands/mad-add-model.md | 1 + .claude/commands/mad-profile.md | 8 ++- .claude/commands/mad-validate.md | 92 ++++++++++++++++++++++++ .claude/workflows/mad-benchmark-sweep.js | 39 ++++++---- 6 files changed, 145 insertions(+), 27 deletions(-) create mode 100644 .claude/commands/mad-validate.md diff --git a/.claude/agents/mad-benchmark-runner.md b/.claude/agents/mad-benchmark-runner.md index 5001719..9e2fea2 100644 --- a/.claude/agents/mad-benchmark-runner.md +++ b/.claude/agents/mad-benchmark-runner.md @@ -17,9 +17,13 @@ When invoked: - Output file: `-o ` when the user wants results kept separately. - Timeout: `--timeout ` for long training runs (default 7200). - Profiling: `--additional-context '{"tools": [{"name": ""}]}'` - (e.g. `rocprofv3_compute`, `rpd`, `rccl_trace`). + (e.g. `rocprofv3_compute`, `rpd`, `rccl_trace`). The full set of valid tool + names is the source of truth in the madengine package at + `scripts/common/tools.json` (23+ tools, incl. `rocprofv3_full`, + `rocblas_trace`, `hipblaslt_trace`, `miopen_trace`, `rocprof_sys`). - Multi-node: add a `"slurm": {...}` or `"k8s": {...}` key to - `--additional-context` — presence of the key selects the target. + `--additional-context` — presence of the key selects the target. An explicit + `"deploy": "slurm"`/`"k8s"` key works too; neither key → local Docker. - Build/run split (build once, run many): `madengine build --tags [-r REGISTRY]` writes `build_manifest.json`, then `madengine run -m build_manifest.json` executes from it (skips rebuild). diff --git a/.claude/agents/mad-model-author.md b/.claude/agents/mad-model-author.md index f2047f8..10f90c3 100644 --- a/.claude/agents/mad-model-author.md +++ b/.claude/agents/mad-model-author.md @@ -25,15 +25,23 @@ When invoked: Prefer reusing an existing Dockerfile of the same stack (point the `dockerfile` field at it) rather than creating a near-duplicate. - c. `scripts//run.sh`. It MUST end by printing exactly: - `echo "performance: $performance "` - where `$performance` is parsed from the workload's own log output. Model - this parsing on the template script. + c. `scripts//run.sh`. It must satisfy ONE of the two output contracts: + - Single result (default): end by printing exactly + `echo "performance: $performance "`, where `$performance` is + parsed from the workload's own log output. Model this on the template. + - Multiple results: have the script WRITE its own CSV (one row per + result) and set `"multiple_results": ".csv"` in the + models.json entry. madengine then ingests that CSV instead of grepping + a single stdout line. Use this only when the template you copied does; + do not mix the two contracts. Rules: -- The `performance: ` stdout line is the hard contract madengine - relies on. Never omit it. +- The output contract is hard: either the `performance: ` stdout + line, OR a `multiple_results` CSV declared in models.json. Never ship a script + that emits neither — madengine will record the run with no performance value. - Validate the final `models.json` parses (`python3 -m json.tool models.json`). -- Do not run `madengine` (it needs GPUs). State the verification command the - user should run: `madengine run --tags --live-output`. +- Confirm the new entry is selectable (GPU-free): `madengine discover --tags ` + should list it. This catches tag/name typos before any GPU run. +- Do not run `madengine run` (it needs GPUs). State the verification command the + user should run on a GPU host: `madengine run --tags --live-output`. - Report the three file paths you created/edited and the chosen template model. diff --git a/.claude/commands/mad-add-model.md b/.claude/commands/mad-add-model.md index 9765616..0957ce1 100644 --- a/.claude/commands/mad-add-model.md +++ b/.claude/commands/mad-add-model.md @@ -11,6 +11,7 @@ Use the `mad-model-author` subagent. It should: `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}` header), and `scripts//run.sh` ending in `echo "performance: $performance "`. 3. Validate `models.json` with `python3 -m json.tool models.json`. +4. Confirm the entry is selectable with `madengine discover --tags $1` (GPU-free). Report the files created and the verification command `madengine run --tags $1 --live-output` (requires a GPU host). diff --git a/.claude/commands/mad-profile.md b/.claude/commands/mad-profile.md index b3214b7..94bfd19 100644 --- a/.claude/commands/mad-profile.md +++ b/.claude/commands/mad-profile.md @@ -6,10 +6,12 @@ argument-hint: [tool: rocprofv3_compute|rpd|rccl_trace|...] Profile `$ARGUMENTS`. Use the `mad-benchmark-runner` subagent with profiling enabled. It should: -1. Pick the profiling tool (default `rocprofv3_compute` if unspecified). Valid - names include `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`, +1. Pick the profiling tool (default `rocprofv3_compute` if unspecified). Common + names: `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`, `rocprofv3_communication`, `rocm_trace_lite`, `rccl_trace`, - `gpu_info_power_profiler`. + `gpu_info_power_profiler`. The complete, authoritative list (23+ tools) lives + in the madengine package at `scripts/common/tools.json` — consult it for + names like `rocprofv3_full`, `rocblas_trace`, `hipblaslt_trace`, `rocprof_sys`. 2. Build: `madengine run --tags $1 --live-output --additional-context '{"tools": [{"name": ""}]}'` 3. Check for GPUs; if none, print the command for a GPU host. Otherwise run it diff --git a/.claude/commands/mad-validate.md b/.claude/commands/mad-validate.md new file mode 100644 index 0000000..7b6fc52 --- /dev/null +++ b/.claude/commands/mad-validate.md @@ -0,0 +1,92 @@ +--- +description: Statically validate MAD model entries (no GPU) — JSON, paths, Dockerfile header, output contract +argument-hint: [tag-or-model | all] +--- + +Statically validate MAD model definitions for `${ARGUMENTS:-all}`. This is a +GPU-free lint of `models.json` and the files it points at — it does NOT build or +run anything. + +Run the checks below (use `python3`; do not install packages). Scope to the +named tag/model if one is given, otherwise check every entry. + +Two severities. **Errors** break a run; **warnings** are missing MAD-convention +metadata (madengine itself defaults these — only `name` is structurally +required, per its `Model` dataclass — so don't fail the build on them). + +Errors: +1. **JSON parses**: `python3 -m json.tool models.json` succeeds. +2. **Has `name`** and **names are unique** (no two entries share one). +3. **Dockerfile exists**: `.ubuntu.amd.Dockerfile` is a real file, + and its first line is the context header + `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}`. +4. **Script exists**: the `scripts` path exists (a `run.sh` file or a dir). +5. **Output contract** — exactly one of: + - the run script contains a line that echoes `performance: ...`, OR + - the entry sets `multiple_results` to a CSV filename. + Flag any entry that satisfies neither (madengine would record no perf value). + +Warnings (convention metadata, non-fatal): +6. Missing any of `url`, `owner`, `training_precision`, `tags`, `n_gpus`. + +Suggested one-shot checker: + +```bash +python3 - "${ARGUMENTS:-all}" <<'PY' +import json, os, sys, glob +sel = sys.argv[1] if len(sys.argv) > 1 else "all" +models = json.load(open("models.json")) +def selected(m): + return sel in ("all","") or sel == m.get("name") or sel in (m.get("tags") or []) +seen, errors, warns = {}, [], [] +for m in models: + n = m.get("name","") + if not selected(m): + continue + # --- errors (break a run) --- + if "name" not in m: errors.append(f"{n}: missing 'name'") + if n in seen: errors.append(f"{n}: duplicate name") + seen[n] = True + df = (m.get("dockerfile","") or "") + ".ubuntu.amd.Dockerfile" + if not m.get("dockerfile"): + errors.append(f"{n}: no dockerfile field") + elif not os.path.isfile(df): + errors.append(f"{n}: dockerfile not found: {df}") + else: + first = open(df).readline().strip() + if not first.startswith("# CONTEXT") or "AMD" not in first: + errors.append(f"{n}: dockerfile missing CONTEXT header: {df}") + sp = m.get("scripts","") + if not sp: + errors.append(f"{n}: no scripts field") + elif not os.path.exists(sp): + errors.append(f"{n}: scripts path not found: {sp}") + has_mr = bool(m.get("multiple_results")) + emits = False + if sp: + sh_files = [sp] if sp.endswith(".sh") else glob.glob(os.path.join(sp,"**","*.sh"), recursive=True) + for sh in sh_files: + if os.path.isfile(sh) and "performance:" in open(sh, errors="ignore").read(): + emits = True; break + if not (has_mr or emits): + errors.append(f"{n}: no output contract (no 'performance:' line and no multiple_results)") + # --- warnings (convention metadata) --- + for f in ("url","owner","training_precision","tags","n_gpus"): + if not m.get(f): + warns.append(f"{n}: missing convention field '{f}'") +checked = sum(1 for m in models if selected(m)) +print(f"Checked {checked} model(s). {len(errors)} error(s), {len(warns)} warning(s).") +if errors: + print("\nERRORS:") + for e in errors: print(" -", e) +if warns: + print("\nwarnings:") + for w in warns[:40]: print(" -", w) + if len(warns) > 40: print(f" ... and {len(warns)-40} more") +sys.exit(1 if errors else 0) +PY +``` + +Report a compact pass/fail summary. If anything fails, list each problem with the +model name and how to fix it. This is the right check to run after `/mad-add-model` +and before any GPU run. diff --git a/.claude/workflows/mad-benchmark-sweep.js b/.claude/workflows/mad-benchmark-sweep.js index 53d7e4d..90706e6 100644 --- a/.claude/workflows/mad-benchmark-sweep.js +++ b/.claude/workflows/mad-benchmark-sweep.js @@ -8,20 +8,24 @@ export const meta = { ], } -// args: { tags: string[], precisions?: string[], nGpus?: string[], execute?: boolean, output?: string } +// args: { tags: string[], nGpus?: string[], execute?: boolean, output?: string } // Without execute:true (default) agents PLAN and validate commands only — safe on GPU-less hosts. +// +// NOTE: there is intentionally no precision axis. In MAD, precision is NOT a +// runtime `madengine run` flag: for training it is fixed per-model via +// `training_precision` in models.json, and for inference it is baked into the +// pre-trained model/image. To compare precisions, list the distinct model tags +// in `tags` instead. const cfg = args || {} const tags = Array.isArray(cfg.tags) ? cfg.tags : (cfg.tags ? [cfg.tags] : ['bert']) -const precisions = cfg.precisions && cfg.precisions.length ? cfg.precisions : [null] const nGpus = cfg.nGpus && cfg.nGpus.length ? cfg.nGpus : [null] const execute = cfg.execute === true -// Build the matrix of {tag, precision, nGpus} cells. +// Build the matrix of {tag, nGpus} cells. const matrix = [] for (const tag of tags) - for (const precision of precisions) - for (const g of nGpus) - matrix.push({ tag, precision, nGpus: g }) + for (const g of nGpus) + matrix.push({ tag, nGpus: g }) log(`Sweep matrix: ${matrix.length} cell(s) across ${tags.length} tag(s). execute=${execute}`) @@ -39,39 +43,46 @@ const CELL_SCHEMA = { } const results = await parallel(matrix.map((cell, i) => () => { - const label = `${cell.tag}${cell.precision ? '/' + cell.precision : ''}${cell.nGpus ? '/' + cell.nGpus + 'gpu' : ''}` + const label = `${cell.tag}${cell.nGpus ? '/' + cell.nGpus + 'gpu' : ''}` + // Each cell writes its OWN perf file so parallel runs never clobber a shared + // perf.csv. A safe filename is derived from the label. + const safe = label.replace(/[^A-Za-z0-9._-]/g, '_') + const outFile = `perf_${safe}.csv` const flags = [ '--tags ' + cell.tag, '--live-output', - cfg.output ? '-o ' + cfg.output : '', - ].filter(Boolean).join(' ') + '-o ' + outFile, + ].join(' ') const ctxParts = [] if (cell.nGpus) ctxParts.push(`"n_gpus": "${cell.nGpus}"`) const ctx = ctxParts.length ? ` --additional-context '{${ctxParts.join(', ')}}'` : '' const cmd = `madengine run ${flags}${ctx}` const action = execute - ? `If AMD GPUs are present (check rocm-smi/amd-smi), RUN this command and parse the "performance: " line from output. If no GPUs, set status "skipped".` + ? `If AMD GPUs are present (check rocm-smi/amd-smi), RUN this command and parse the "performance: " line from output. Results land in ${outFile}. If no GPUs, set status "skipped".` : `Do NOT execute. Validate the command is well-formed and the tag resolves via "madengine discover --tags ${cell.tag}". Set status "planned".` return agent( `MAD benchmark sweep cell "${label}". Command: ${cmd} ${action} -${cell.precision ? 'Intended precision: ' + cell.precision + ' (confirm the tag/model supports it).' : ''} Return the cell label, the exact command, status, and performance/metric if available.`, { label: `bench:${label}`, phase: 'Run', schema: CELL_SCHEMA } - ).then(r => ({ ...r, _label: label })) + ).then(r => ({ ...r, _label: label, _output: outFile })) })) phase('Synthesize') const clean = results.filter(Boolean) const summary = await agent( `Synthesize a MAD benchmark sweep into a markdown comparison table. -Cells (JSON): ${JSON.stringify(clean)} +Cells (JSON, each with its own _output perf file): ${JSON.stringify(clean)} Produce: (1) a table with columns Cell | Command | Status | Performance | Metric; (2) a short headline (how many planned/ran/skipped/errored); -(3) if any ran, the fastest and slowest cells (respecting that metric units differ). +(3) explicitly flag any cell whose status is "error" or whose tag did not resolve — +do not let a failed/unresolved cell read as success; +(4) if any ran, the fastest and slowest cells (respecting that metric units differ); +(5) note that each cell wrote a separate perf_.csv (to avoid clobbering) and +that they can be concatenated for a combined report. Do not invent numbers — only use provided data.`, { phase: 'Synthesize' } ) From b78525b299097310d270c416e6d3fa16e97f0c3f Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Mon, 1 Jun 2026 00:48:20 +0000 Subject: [PATCH 03/13] Create mad automation howto html as docs --- mad-automation-howto.html | 1876 +++++++++++++++++++++++++++++++++++++ 1 file changed, 1876 insertions(+) create mode 100644 mad-automation-howto.html diff --git a/mad-automation-howto.html b/mad-automation-howto.html new file mode 100644 index 0000000..90580ef --- /dev/null +++ b/mad-automation-howto.html @@ -0,0 +1,1876 @@ + + + + + +MAD Claude Automation — How-To Guide + + + + + + + + +
+ + +
+
+

MAD Claude Automation

+

+ A practical guide to benchmarking, adding, tuning, and analyzing AI model workloads + on AMD Instinct GPUs — using Claude Code's slash commands, agents, and workflows. +

+
+
+
+
142
+
models
+
+
+
6
+
commands
+
+
+
4
+
agents
+
+
+
2
+
workflows
+
+
+ + +
+

1 What is MAD

+ +

+ MAD (Model Automation and Dashboarding) is a curated catalog of 142 AI model + workloads designed to run on AMD Instinct GPUs. Each model is one JSON entry in + models.json. The madengine CLI handles the full lifecycle: + building a Docker image, running the model inside a container, parsing a performance number + from stdout, and appending a row to perf.csv. +

+ +

+ Claude Code automation sits on top of madengine. Instead of memorizing flags and file + conventions, you type a slash command — Claude dispatches the right subagent which constructs + and (when a GPU host is available) executes the correct madengine invocation. +

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
User taskSlash commandSubagent
Run a benchmark/mad-benchmarkmad-benchmark-runner
Add a new model/mad-add-modelmad-model-author
Profile / trace a model/mad-profilemad-benchmark-runner
Analyze results/mad-reportmad-perf-analyst
Tune for better perf/mad-tunemad-tuner
Validate model entries/mad-validate(inline script)
+
+ +

+ Two multi-agent workflows handle matrix benchmarking and systematic + tuning search: mad-benchmark-sweep and mad-tune-search. + These fan out parallel subagent calls and synthesize results into comparison tables. +

+
+ + +
+

2 Architecture

+ +

+ The system has three layers: the user interface (slash commands you type in + Claude Code), subagents (specialized Claude instances with tool access), and + the madengine CLI plus the MAD repository files. +

+ +
+ + + + + + + + + + + + USER + AGENTS + ENGINE + + + + + + + + + /mad-benchmark + /mad-profile + + + + /mad-add-model + scaffold files + + + + /mad-report + parse results + + + + /mad-tune + measure-change + + + + /mad-validate + GPU-free lint + + + + benchmark-sweep + tune-search + + + + + benchmark-runner + Bash · Read · Grep · Glob + run & profile + + + + model-author + Read · Write · Edit · Glob + scaffold files + + + + perf-analyst + Read · Grep · Bash (r/o) + parse & compare CSV + + + + tuner + Read · Edit · Bash · Grep + measure-change-measure + + + + + madengine + build · run · discover · report · database + Docker / SLURM / Kubernetes + + + + models.json + 142 model entries + tags · dockerfile · scripts + + + + docker/ + scripts/ + *.ubuntu.amd.Dockerfile + run.sh + perf line + performance: N unit + + + + perf.csv + one row per run + model · perf · metric + status · gpu_arch + + + + AMD GPU + required + for run/build + + + + + + + + + + + + + + + + + + + + + +
+ +

+ Each slash command maps to a subagent. Workflows fan out to multiple parallel + subagent calls, then synthesize the results. Agents that need to run madengine always + check for AMD GPUs first — if none are present they produce a plan (the exact commands to + run on a GPU host) instead of executing. +

+
+ + +
+

3 Quick-Start: What Do You Want to Do?

+ +
+
+
Run one benchmark
+
→
+
/mad-benchmark pyt_vllm_llama-3.1-8b
+
+
+
Benchmark multiple tags at once
+
→
+
/mad-benchmark-sweep (workflow)
+
+
+
Add a brand-new model
+
→
+
/mad-add-model pyt_vllm_gemma-3-27b https://github.com/...
+
+
+
Attach a GPU profiler
+
→
+
/mad-profile bert rocprofv3_compute
+
+
+
Read or compare benchmark results
+
→
+
/mad-report perf.csv perf_old.csv
+
+
+
Improve throughput of a model
+
→
+
/mad-tune bert throughput
+
+
+
Validate files before a GPU run
+
→
+
/mad-validate bert
+
+
+
Systematic tuning search (N candidates)
+
→
+
/mad-tune-search (workflow)
+
+
+ +
+
GPU host check
+

+ madengine build and run require AMD GPUs. On a machine without GPUs, + agents automatically switch to plan mode — they validate the command and print + the exact invocation for you to run on a GPU host. GPU-free commands (discover, validate) + always work. +

+
+
+ + +
+

4 The Performance Contract

+ +

+ This is the single most important MAD convention. Every model's run.sh must + emit exactly one line to stdout that madengine can grep. Without it, + madengine records the run with no performance value. +

+ +
+
Required output — single result
+ 
performance: <value> <unit>
+

+ For example: performance: 3847.2 tokens_per_second or + performance: 124 samples_per_second. +

+
+ +

Implementing the line in run.sh

+ + 
copy# Parse from the workload's own log output, then emit the contract line
+performance=$(grep -Eo "train_samples_per_second':[^,]+" log.txt \
+  | sed "s/.*: //g" | head -n 1)
+echo "performance: $performance samples_per_second"
+ +

Alternative: multiple results

+ +

+ When a script produces many rows (e.g. one per concurrency level), have the script write + its own CSV and declare "multiple_results" in models.json: +

+ + 
copy# models.json entry
+{
+  "name": "pyt_vllm_deepseek-r1",
+  ...
+  "multiple_results": "perf_DeepSeek-R1.csv"
+}
+ +
+
Hard rule
+

+ A run script must satisfy exactly one of the two contracts above. + Satisfying neither means madengine records a run with no performance value. + The /mad-validate command checks this statically before any GPU run. +

+
+
+ + +
+

5 Slash Command Reference

+ +
+
+
+ +
+
+
+ /mad-benchmark + command + benchmark-runner +
+
Build and run a madengine benchmark for the given tag or model name.
+
+
+
+

Usage

+ 
copy/mad-benchmark <tag-or-model> [extra options]
+ +

Examples

+ 
copy/mad-benchmark bert
+/mad-benchmark pyt_vllm_llama-3.1-8b --timeout 3600
+/mad-benchmark deepseek --output results/deepseek.csv
+ +

What the agent does

+
    +
  1. Runs madengine discover --tags <tag> to confirm the tag resolves (GPU-free).
    2. +
  2. Assembles: madengine run --tags <tag> --live-output [-o path] [--timeout S].
    3. +
  3. Checks for AMD GPUs via rocm-smi / amd-smi.
    4. +
  4. On GPU host: runs the command and reports where perf.csv landed.
    5. +
  5. No GPU: prints the exact command to run on a GPU host and stops.
    6. +
+ +

Expected output

+
    +
  • A perf.csv row with the performance value and unit.
    • +
  • Live stdout from the model during the run (via --live-output).
    • +
  • Summary of resolved model(s), command used, and output file path.
    • +
+
+
+
+ +
+
+
+
+ +
+
+
+ /mad-add-model + command + model-author +
+
Scaffold a new MAD model — models.json entry, Dockerfile, and run.sh with the performance line.
+
+
+
+

Usage

+ 
copy/mad-add-model <framework_project_workload> [base notes / repo url]
+ +

Examples

+ 
copy/mad-add-model pyt_vllm_gemma-3-27b https://github.com/google/gemma
+/mad-add-model jax_maxtext_llama-4 bf16 inference, context=8192
+ +

What the agent does

+
    +
  1. Identifies the closest existing model of the same framework to use as template.
    2. +
  2. Creates three files: +
      +
    • models.json entry (required fields: name, url, dockerfile, scripts, n_gpus, owner, training_precision, tags)
      • +
    • docker/<name>.ubuntu.amd.Dockerfile with the required context header
      • +
    • scripts/<dir>/run.sh ending in echo "performance: $performance <unit>"
      • +
    +
    3. +
  3. Validates models.json with python3 -m json.tool models.json.
    4. +
  4. Confirms the entry is selectable: madengine discover --tags <name>.
    5. +
+ +

Dockerfile required header (first line)

+ 
copy# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}
+ARG BASE_DOCKER=rocm/pytorch
+FROM $BASE_DOCKER
+ +

Expected output

+
    +
  • Three file paths created/edited.
    • +
  • The template model that was used.
    • +
  • The verification command: madengine run --tags <name> --live-output (requires GPU).
    • +
+
+
+
+ +
+
+
+
+ +
+
+
+ /mad-profile + command + benchmark-runner +
+
Run a madengine benchmark with a GPU profiling or tracing tool attached.
+
+
+
+

Usage

+ 
copy/mad-profile <tag-or-model> [tool: rocprofv3_compute|rpd|rccl_trace|...]
+ +

Examples

+ 
copy/mad-profile bert
+/mad-profile pyt_vllm_llama-3.1-8b rccl_trace
+/mad-profile deepseek rocprofv3_memory
+ +

What the agent does

+
    +
  1. Picks the profiling tool (defaults to rocprofv3_compute if unspecified).
    2. +
  2. Builds the command with --additional-context:
    3. +
+ 
copymadengine run --tags <tag> --live-output \
+  --additional-context '{"tools": [{"name": "rocprofv3_compute"}]}'
+ +

Available profiling tools (common subset)

+
+ + + + + + + + + + + + +
Tool nameWhat it captures
rocprofv3_computeKernel execution, occupancy, compute metrics
rocprofv3_memoryMemory bandwidth, cache hit rates
rocprofv3_communicationGPU-to-GPU communication metrics
rpdROCm profiling data, timeline view
rccl_traceCollective communication (AllReduce, etc.)
rocm_trace_liteLightweight system trace
gpu_info_power_profilerPower draw over time
rocblas_trace / hipblaslt_traceBLAS kernel calls
+
+

The complete authoritative list (23+ tools) is in the madengine package at scripts/common/tools.json.

+ +
+
Performance note
+

Profiling adds overhead. The perf number recorded under profiling is not a clean benchmark — use /mad-benchmark for baseline measurements.

+
+
+
+
+ +
+
+
+
+ +
+
+
+ /mad-report + command + perf-analyst +
+
Parse and summarize MAD benchmark results from perf.csv or perf_entry_super.json. Compare two result sets to find regressions.
+
+
+
+

Usage

+ 
copy/mad-report [csv-or-json-path] [compare-to-path]
+ +

Examples

+ 
copy/mad-report                               # defaults to perf.csv
+/mad-report results/after.csv
+/mad-report perf_new.csv perf_baseline.csv  # comparison mode
+ +

What the agent does

+
    +
  • Locates the result file(s) — defaults to perf.csv.
    • +
  • Single file: summarizes models run, performance + unit, pass/fail status.
    • +
  • Two files: per-model delta and % change; flags regressions (slower) vs improvements; respects each row's metric unit.
    • +
  • Leads with the headline finding, then a compact table.
    • +
+ +

Generate an HTML dashboard instead

+ 
copymadengine report to-html --csv-file-path perf.csv
+ +

perf.csv columns

+
+ + + + + + + + + + + + + +
ColumnMeaning
modelModel name from models.json
performanceNumeric result parsed from stdout
metricUnit (tokens_per_second, samples_per_second, …)
statusSUCCESS or failure reason
n_gpusNumber of GPUs used
training_precisionfp16, bf16, fp8, fp32, …
gpu_architecturegfx942, gfx950, …
git_commitCommit hash at run time
build_duration / test_durationWall time in seconds
+
+
+
+
+ +
+
+
+
+ +
+
+
+ /mad-tune + command + tuner +
+
Iteratively tune a MAD model for better performance using a disciplined measure-change-measure loop.
+
+
+
+

Usage

+ 
copy/mad-tune <tag-or-model> [target: throughput|latency] [lever hints]
+ +

Examples

+ 
copy/mad-tune bert
+/mad-tune pyt_vllm_llama-3.1-8b throughput
+/mad-tune deepseek latency try tensor-parallel=4
+ +

What the agent does

+
    +
  1. Reads the model's run.sh, config files, and current perf.csv row to establish a baseline.
    2. +
  2. Identifies tuning levers for the stack: +
      +
    • Env vars: MAD_MODEL_BATCH_SIZE, HIP_VISIBLE_DEVICES, NCCL_*/RCCL_*, PYTORCH_TUNABLEOP_ENABLED
      • +
    • Script/config args: tensor-parallel size, precision (fp16/bf16/fp8), sequence length, gpu-memory-utilization, max-num-seqs
      • +
    +
    3. +
  3. Proposes one change at a time, applies it, re-measures, keeps improvements, reverts regressions.
    4. +
  4. No GPU: produces a ranked list of candidate changes with rationale and the exact madengine run commands to test each.
    5. +
+ +

Expected output

+
    +
  • Baseline performance value + unit.
    • +
  • Table of each change tried with measured delta.
    • +
  • Final recommended configuration with before/after numbers.
    • +
+
+
+
+ +
+
+
+
+ +
+
+
+ /mad-validate + command + GPU-free +
+
Statically validate MAD model entries — JSON, paths, Dockerfile header, output contract. No GPU required.
+
+
+
+

Usage

+ 
copy/mad-validate [tag-or-model | all]
+ +

Examples

+ 
copy/mad-validate                    # checks all 142 entries
+/mad-validate bert               # checks only models matching "bert"
+/mad-validate pyt_vllm_deepseek-r1
+ +

Checks performed

+ +

Errors (break a run)

+
    +
  • models.json parses as valid JSON.
    • +
  • All entries have a name field and names are unique.
    • +
  • Dockerfile exists at <dockerfile>.ubuntu.amd.Dockerfile and first line is the context header.
    • +
  • Scripts path exists (a run.sh or directory).
    • +
  • Output contract satisfied: run script contains a performance: line, or entry sets multiple_results.
    • +
+ +

Warnings (convention metadata, non-fatal)

+
    +
  • Missing any of: url, owner, training_precision, tags, n_gpus.
    • +
+ +
+
Workflow tip
+

Run /mad-validate <name> immediately after /mad-add-model and before any GPU run. It catches tag typos, missing files, and missing performance lines in seconds.

+
+
+
+
+ + +
+

6 Workflow Reference

+ +
+
+
+ +
+
+
+ mad-benchmark-sweep + workflow +
+
Fan out a benchmark matrix across multiple MAD tags, run (or plan) each in parallel, and synthesize a comparison table.
+
+
+
+

When to use

+

When you want to compare performance across multiple model tags or GPU counts in one operation.

+ +

Phases

+
+
+
Phase 1
+
Resolve
+
Expand the sweep matrix into concrete madengine invocations
+
+
+
Phase 2
+
Run
+
Execute or plan each benchmark in parallel subagents
+
+
+
Phase 3
+
Synthesize
+
Parse results into a comparison table with headline finding
+
+
+ +

Args schema

+ 
copy{
+  "tags":    ["bert", "pyt_vllm_llama-3.1-8b", "deepseek"],  // required
+  "nGpus":   ["4", "8"],                                    // optional, default: no override
+  "execute": true,                                           // false = plan mode (default)
+  "output":  "sweep_results/"                                 // optional
+}
+ +

Example invocations

+ 
copy# Plan mode — safe, no GPU required
+/mad-benchmark-sweep { "tags": ["bert", "pyt_vllm_deepseek-r1"] }
+
+# Execute mode — AMD GPU required
+/mad-benchmark-sweep { "tags": ["bert", "pyt_vllm_llama-3.1-8b"], "execute": true }
+ +
+
perf.csv isolation
+

+ Each sweep cell writes to its own perf_<tag>.csv — never a shared + perf.csv. This prevents parallel runs from clobbering each other. + Concatenate the files afterward for a combined report. +

+
+ +
+
Precision is not a sweep axis
+

+ In MAD, precision (fp16/bf16/fp8) is fixed per model in + models.json — it is not a runtime flag. To compare precisions, + list the distinct model tags (e.g. pyt_vllm_llama-fp16 and + pyt_vllm_llama-fp8) in the tags array instead. +

+
+
+
+
+ + + + +
+

7 Agent Reference

+ +

+ Agents are specialized Claude instances with a defined tool set and a focused role. + They are dispatched automatically by slash commands and workflows — you generally never + call them directly. +

+ +
+
+
+ +
+
+
+ mad-benchmark-runner + agent +
+
Constructs and runs the correct madengine benchmark command from a model/tag or plain-English intent.
+
+
+
+

Tools: Bash, Read, Grep, Glob

+

Called by: /mad-benchmark, /mad-profile, mad-benchmark-sweep (workflow)

+
    +
  • Resolves tags with madengine discover (read-only, no GPU).
    • +
  • Builds the full madengine run command including profiling context if requested.
    • +
  • Checks for GPUs — plan mode if absent, execute if present.
    • +
  • Reports resolved models, exact command, required env vars, and output file path.
    • +
+
+
+ +
+
+
+ +
+
+
+ mad-model-author + agent +
+
Scaffolds a new MAD model — the models.json entry, the Dockerfile, and the run.sh with the performance line.
+
+
+
+

Tools: Read, Write, Edit, Grep, Glob

+

Called by: /mad-add-model

+
    +
  • Finds the closest existing model of the same framework to use as a template.
    • +
  • Creates all three required files following MAD conventions.
    • +
  • Validates models.json and confirms discoverability — never runs a GPU benchmark.
    • +
  • Reports file paths, template used, and the GPU verification command.
    • +
+
+
+ +
+
+
+ +
+
+
+ mad-perf-analyst + agent + read-only +
+
Reads and analyzes MAD benchmark results from perf.csv and perf_entry_super.json. Never edits files or runs workloads.
+
+
+
+

Tools: Read, Grep, Glob, Bash (read-only)

+

Called by: /mad-report

+
    +
  • Parses perf.csv, perf_entry_super.json, or any named CSV.
    • +
  • Summarizes or compares results; precise about units — never compares across different metric values.
    • +
  • States direction assumption (higher = better for throughput, lower = better for latency).
    • +
  • Uses python3 for parsing — never installs packages.
    • +
+
+
+ +
+
+
+ +
+
+
+ mad-tuner + agent +
+
Iteratively tunes an existing MAD model using a measure-change-measure loop. Reverts regressions automatically.
+
+
+
+

Tools: Read, Edit, Bash, Grep, Glob

+

Called by: /mad-tune, mad-tune-search (workflow)

+
    +
  • Establishes baseline from run.sh, config files, and current perf.csv.
    • +
  • Proposes and applies one lever change at a time — keeps improvements, reverts regressions.
    • +
  • Requires AMD GPUs to measure. Without GPUs, produces a ranked candidate list with exact commands.
    • +
  • Never alters the performance: <value> <unit> output contract.
    • +
+
+
+
+ + +
+

8 Environment Variables

+ +

Pass these environment variables to control model behavior and authentication.

+ +
+
+
MAD_SECRETS_HFTOKEN
+
HuggingFace API token. Required for gated models (e.g. Llama 3, Gemma). Set before running any HF model.
+
+
+
MAD_MODEL_BATCH_SIZE
+
Override the batch size for a model run. Primary tuning lever for throughput experiments.
+
+
+
MAD_RUNTIME_NGPUS
+
Override the number of GPUs used at runtime (also controllable via n_gpus in models.json).
+
+
+
MAD_SYSTEM_GPU_ARCHITECTURE
+
GPU architecture identifier (e.g. gfx942, gfx950). Used for arch-specific builds and skip_gpu_arch filtering.
+
+
+
MAD_MODEL_NAME
+
Model name passed into the container at runtime. Some scripts use this to select between model checkpoints.
+
+
+
PYTORCH_TUNABLEOP_ENABLED
+
Enable PyTorch TunableOp for auto-tuned GEMM kernels. A common throughput tuning lever for PyTorch models.
+
+
+
NCCL_* / RCCL_*
+
Collective communication tuning (chunk sizes, algorithms, socket buffers). Relevant for multi-GPU runs.
+
+
+
HIP_VISIBLE_DEVICES
+
Which GPU indices are visible to the container. Use to pin a run to specific GPUs.
+
+
+ +

Export pattern (shell)

+ 
copyexport MAD_SECRETS_HFTOKEN=hf_...
+export MAD_MODEL_BATCH_SIZE=32
+madengine run --tags pyt_vllm_llama-3.1-8b --live-output
+
+ + +
+

9 Tips and Gotchas

+ +
+ GPU guard — plan mode vs execute mode +
+

+ All agents check for AMD GPUs (rocm-smi or amd-smi) before + attempting to build or run. On a machine without GPUs: +

+
    +
  • The agent prints the exact madengine command to run on a GPU host.
    • +
  • It lists required env vars (e.g. MAD_SECRETS_HFTOKEN).
    • +
  • It stops — it does not attempt to run the command.
    • +
+

+ GPU-free operations always work: madengine discover, /mad-validate, + and plan mode for benchmarks and tuning. +

+ 
copy# GPU-free: confirm a tag resolves
+madengine discover --tags bert
+
+
+ +
+ Precision is not a runtime flag +
+

+ In MAD, training_precision (fp16, bf16, fp8, fp32) is fixed per model in + models.json. It is baked into the Dockerfile and run script — you cannot + change it with a madengine flag at runtime. +

+

+ To compare precisions in a sweep, use separate model tags that differ + in precision. For example: pyt_vllm_llama-fp16 and + pyt_vllm_llama-fp8 as two entries in the tags array of + mad-benchmark-sweep. +

+
+
+ +
+ perf.csv isolation in parallel sweeps +
+

+ The mad-benchmark-sweep workflow gives each cell its own output file + (perf_<tag>.csv) to prevent parallel runs from clobbering a shared + perf.csv. After a sweep, concatenate for a combined view: +

+ 
copy# Merge sweep results (keep one header row)
+head -1 perf_bert.csv > perf_combined.csv
+tail -n +2 perf_bert.csv perf_deepseek.csv >> perf_combined.csv
+madengine report to-html --csv-file-path perf_combined.csv
+
+
+ +
+ madengine discover — GPU-free tag validation +
+

+ Before any GPU run, confirm that your tags resolve to real models. + madengine discover reads models.json without touching Docker + or GPUs. +

+ 
copymadengine discover --tags pyt_vllm_deepseek-r1
+madengine discover --tags vllm   # matches all tags containing "vllm"
+

+ The tags field in a model entry supports any-match: madengine + selects a model if --tags matches the exact name OR any + string in the tags list. +

+
+
+ +
+ tools.json — authoritative profiler name list +
+

+ The /mad-profile command accepts any profiler tool name. The complete + authoritative list (23+ tools) lives in the madengine package at + scripts/common/tools.json. Common names include: +

+ 
copyrocprofv3_compute    rocprofv3_memory     rocprofv3_communication
+rocprofv3_full       rocprofv3            rpd
+rocm_trace_lite      rccl_trace           gpu_info_power_profiler
+rocblas_trace        hipblaslt_trace      miopen_trace
+rocprof_sys
+

+ If an agent gives you an unknown tool error, consult tools.json directly + for the exact name. +

+
+
+ +
+ Deployment targets — local Docker vs SLURM vs Kubernetes +
+

+ madengine infers the deployment target from which key is present in + --additional-context. No special flag needed. +

+ 
copy# Local Docker (default — no key)
+madengine run --tags bert --live-output
+
+# SLURM (presence of "slurm" key)
+madengine run --tags bert -c '{"slurm": {"partition": "gpu", "nodes": 2}}'
+
+# Kubernetes (presence of "k8s" or "kubernetes" key)
+madengine run --tags bert -c '{"k8s": {"namespace": "ml-jobs"}}'
+
+
+ +
+ Build-once, run-many pattern +
+

+ When running the same model many times (e.g. during tuning), build the image once and + reuse it to save time. +

+ 
copy# Step 1: build only, writes build_manifest.json
+madengine build --tags pyt_vllm_llama-3.1-8b
+
+# Step 2: run from manifest (skips rebuild)
+madengine run -m build_manifest.json --live-output
+
+# Alternatively: skip the model run but verify image builds
+madengine run --tags pyt_vllm_llama-3.1-8b --skip-model-run
+
+
+ +
+ models.json naming convention +
+

+ Model names follow {framework}_{project}_{workload}. Examples: +

+
+ + + + + + + + +
NameFrameworkProjectWorkload
pyt_vllm_deepseek-r1pyt (PyTorch)vllmdeepseek-r1
jax_maxtext_llama-4jaxmaxtextllama-4
pyt_primus_qwen3-1.7bpytprimusqwen3-1.7b
huggingface_berthuggingface–bert
+
+

+ The dockerfile field stores the path prefix — madengine automatically + appends .ubuntu.amd.Dockerfile. +

+
+
+ +
+ Uploading results to MongoDB +
+

Use madengine database to push results to MongoDB after a run.

+ 
copy# Dry run first to preview
+madengine database -f perf_entry_super.json --db mad_results -c benchmarks \
+  -k model,timestamp --dry-run
+
+# Upload for real
+madengine database -f perf_entry_super.json --db mad_results -c benchmarks \
+  -k model,timestamp
+

+ perf_entry_super.json is the enriched record (includes a configs + block) that gets pushed. -k model,timestamp sets the unique key for upsert. +

+
+
+
+ +
+ + + + + \ No newline at end of file From 4edb2c35bf3c171f8b4f71d33385a29ee933b429 Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Mon, 1 Jun 2026 03:43:23 +0000 Subject: [PATCH 04/13] Add Claude dynamic workflows and agent automation for MAD benchmarking & tuning Brings Claude Code's multi-agent automation to MAD: plain slash commands now drive the full benchmark/tune lifecycle, with dynamic workflows that fan out specialized subagents and synthesize their results - turning manual, flag-heavy madengine runs into one-line, self-orchestrating operations. Headline capabilities: - mad-benchmark-sweep: a dynamic workflow that benchmarks many models in parallel and auto-builds a comparison table, with isolated per-cell output so parallel runs never clobber each other. - mad-tune-search: an agentic, profiling-driven tuning loop. It profiles once to DIAGNOSE the real bottleneck (compute/memory/communication/launch), proposes evidence-backed candidates, measures each on clean runs, and has an independent agent adversarially verify every claimed gain before recommending a config - decisions grounded in data, not guesswork. Engineering changes that make this work: - Both workflows now accept the CLI-style flags the slash commands actually pass (--tags / --additional-context / --plan); the prior object-only parsing silently dropped tags and context and defaulted to plan-only. They thread --additional-context into every run and execute by default. - mad-tune-search splits one context into a profiled variant (Diagnose) and a clean variant (measurement), evaluates candidates sequentially to avoid GPU contention and config-edit races, and isolates output to perf_tune_.csv. - mad-automation-howto.html updated to match: CLI-flag tables, execute-by-default callouts, the 5-phase tune-search, and a bottleneck-to-lever reference table. Co-Authored-By: Claude Opus 4 --- .claude/workflows/mad-benchmark-sweep.js | 119 +++++++-- .claude/workflows/mad-tune-search.js | 315 +++++++++++++++++++---- mad-automation-howto.html | 174 ++++++++++--- 3 files changed, 503 insertions(+), 105 deletions(-) diff --git a/.claude/workflows/mad-benchmark-sweep.js b/.claude/workflows/mad-benchmark-sweep.js index 90706e6..77fd2d6 100644 --- a/.claude/workflows/mad-benchmark-sweep.js +++ b/.claude/workflows/mad-benchmark-sweep.js @@ -8,18 +8,92 @@ export const meta = { ], } -// args: { tags: string[], nGpus?: string[], execute?: boolean, output?: string } -// Without execute:true (default) agents PLAN and validate commands only — safe on GPU-less hosts. +// ARGS — accepts EITHER: +// (a) a structured object: { tags: string[], nGpus?: string[], execute?: boolean, +// additionalContext?: string, timeout?: number } +// (b) a CLI-style string mirroring `/mad-benchmark`, e.g. +// --tags pyt_vllm_qwen3-8b,bert --additional-context '{"gpu_vendor":"AMD",...}' --live-output +// +// Multiple tags: pass several after --tags and/or comma-separate them. Each tag +// becomes one sweep cell. --additional-context is threaded verbatim into EVERY +// cell command (so HF tokens / docker_env_vars reach each run). +// +// EXECUTE semantics: runs for real by default. Pass --no-execute (or --plan, or +// execute:false) to only validate commands + resolve tags without touching GPUs. +// Even when executing, each cell first checks for AMD GPUs and self-skips if none. // // NOTE: there is intentionally no precision axis. In MAD, precision is NOT a // runtime `madengine run` flag: for training it is fixed per-model via // `training_precision` in models.json, and for inference it is baked into the -// pre-trained model/image. To compare precisions, list the distinct model tags -// in `tags` instead. -const cfg = args || {} -const tags = Array.isArray(cfg.tags) ? cfg.tags : (cfg.tags ? [cfg.tags] : ['bert']) -const nGpus = cfg.nGpus && cfg.nGpus.length ? cfg.nGpus : [null] -const execute = cfg.execute === true +// pre-trained model/image. To compare precisions, list the distinct model tags. + +// --- tokenizer: split a CLI string respecting single/double quotes --- +function tokenize(s) { + const out = [] + let cur = '', q = null + for (let i = 0; i < s.length; i++) { + const c = s[i] + if (q) { + if (c === q) q = null + else cur += c + } else if (c === '"' || c === "'") { + q = c + } else if (c === ' ' || c === '\t' || c === '\n') { + if (cur) { out.push(cur); cur = '' } + } else { + cur += c + } + } + if (cur) out.push(cur) + return out +} + +// --- parse a CLI string into the structured cfg the sweep uses --- +function parseCli(s) { + const toks = tokenize(s) + const cfg = { tags: [], nGpus: [], execute: true } + for (let i = 0; i < toks.length; i++) { + const t = toks[i] + const next = () => toks[++i] + if (t === '--tags' || t === '-t') { + // collect following non-flag tokens, comma-splitting each + while (i + 1 < toks.length && !toks[i + 1].startsWith('-')) { + next().split(',').forEach(x => { if (x) cfg.tags.push(x) }) + } + } else if (t === '--additional-context' || t === '-c') { + cfg.additionalContext = next() + } else if (t === '--ngpus' || t === '--n-gpus') { + next().split(',').forEach(x => { if (x) cfg.nGpus.push(x) }) + } else if (t === '--timeout') { + cfg.timeout = next() + } else if (t === '-o' || t === '--output') { + cfg.output = next() + } else if (t === '--no-execute' || t === '--plan' || t === '--dry-run') { + cfg.execute = false + } else if (t === '--execute') { + cfg.execute = true + } + // --live-output and unknown flags are ignored (live-output is always added below) + } + return cfg +} + +// --- normalize args (string | object | undefined) into one cfg --- +let cfg +if (typeof args === 'string') { + cfg = parseCli(args) +} else if (args && typeof args === 'object') { + cfg = { execute: true, ...args } + if (typeof cfg.tags === 'string') cfg.tags = [cfg.tags] +} else { + cfg = { tags: [], nGpus: [], execute: true } +} + +const tags = (cfg.tags && cfg.tags.length) ? cfg.tags : ['bert'] +const nGpus = (cfg.nGpus && cfg.nGpus.length) ? cfg.nGpus : [null] +const execute = cfg.execute !== false +const addlCtx = cfg.additionalContext || null +const timeout = cfg.timeout || null // Build the matrix of {tag, nGpus} cells. const matrix = [] @@ -27,7 +101,7 @@ for (const tag of tags) for (const g of nGpus) matrix.push({ tag, nGpus: g }) -log(`Sweep matrix: ${matrix.length} cell(s) across ${tags.length} tag(s). execute=${execute}`) +log(`Sweep matrix: ${matrix.length} cell(s) across ${tags.length} tag(s) [${tags.join(', ')}]. execute=${execute}${addlCtx ? ', additional-context threaded' : ''}`) const CELL_SCHEMA = { type: 'object', @@ -52,14 +126,29 @@ const results = await parallel(matrix.map((cell, i) => () => { '--tags ' + cell.tag, '--live-output', '-o ' + outFile, - ].join(' ') - const ctxParts = [] - if (cell.nGpus) ctxParts.push(`"n_gpus": "${cell.nGpus}"`) - const ctx = ctxParts.length ? ` --additional-context '{${ctxParts.join(', ')}}'` : '' - const cmd = `madengine run ${flags}${ctx}` + ] + if (timeout) flags.push('--timeout ' + timeout) + // Thread the user-supplied --additional-context verbatim into every cell. If a + // per-cell n_gpus axis is set, try to merge it into the JSON; fall back to the + // raw string + a separate context if merge isn't possible. + let ctx = '' + if (addlCtx) { + let merged = addlCtx + if (cell.nGpus) { + try { + const obj = JSON.parse(addlCtx) + obj.n_gpus = cell.nGpus + merged = JSON.stringify(obj) + } catch (e) { /* leave raw; n_gpus axis ignored for this cell */ } + } + ctx = ` --additional-context '${merged}'` + } else if (cell.nGpus) { + ctx = ` --additional-context '{"n_gpus": "${cell.nGpus}"}'` + } + const cmd = `madengine run ${flags.join(' ')}${ctx}` const action = execute - ? `If AMD GPUs are present (check rocm-smi/amd-smi), RUN this command and parse the "performance: " line from output. Results land in ${outFile}. If no GPUs, set status "skipped".` + ? `If AMD GPUs are present (check rocm-smi/amd-smi), RUN this command from the /home/ysha/MAD repo root and parse results. This model may emit a "performance: " stdout line OR (if it is a multiple_results model) write per-metric rows to its own CSV — in that case report the primary throughput metric. Results land in ${outFile}. If no GPUs, set status "skipped".` : `Do NOT execute. Validate the command is well-formed and the tag resolves via "madengine discover --tags ${cell.tag}". Set status "planned".` return agent( diff --git a/.claude/workflows/mad-tune-search.js b/.claude/workflows/mad-tune-search.js index 29a71ca..5772a32 100644 --- a/.claude/workflows/mad-tune-search.js +++ b/.claude/workflows/mad-tune-search.js @@ -1,21 +1,147 @@ export const meta = { name: 'mad-tune-search', - description: 'Generate N candidate tuning configs for one MAD model, run (or plan) each, adversarially verify perf deltas, and synthesize the winning config', + description: 'Profile a MAD model to diagnose its bottleneck, generate profiling-informed tuning candidates, measure each cleanly + adversarially verify, and synthesize the winning config', phases: [ - { title: 'Baseline', detail: 'read the model + establish the baseline config' }, - { title: 'Propose', detail: 'generate distinct tuning candidates' }, - { title: 'Evaluate', detail: 'run or plan each candidate and verify its delta' }, + { title: 'Baseline', detail: 'read the model + establish a clean baseline number' }, + { title: 'Diagnose', detail: 'profile once to find hotspots and classify the bottleneck' }, + { title: 'Propose', detail: 'generate distinct candidates, each citing profiling evidence' }, + { title: 'Evaluate', detail: 'measure each candidate cleanly (sequential) and verify its delta' }, { title: 'Synthesize', detail: 'pick the winning configuration' }, ], } -// args: { tag: string, target?: 'throughput'|'latency', candidates?: number, execute?: boolean } -const cfg = args || {} +// ARGS — accepts EITHER: +// (a) a structured object: { tag, target?, candidates?, execute?, +// additionalContext?, timeout?, profileTool? } +// (b) a CLI-style string mirroring `/mad-benchmark` / `/mad-profile`, e.g. +// --tag pyt_vllm_qwen3-8b --candidates 6 \ +// --additional-context '{"gpu_vendor":"AMD","tools":[{"name":"rocm_trace_lite"}],...}' +// +// WHY profiling is split from measurement (the core design): +// - DIAGNOSE uses a profiler (tools[]) to find WHAT to tune: kernel hotspots, +// and whether the workload is compute- / memory- / communication- / launch-bound. +// - EVALUATE measures WHETHER a change helped using CLEAN runs (profiler OFF), +// because tracing overhead (e.g. rocm_trace_lite intercepts ~all GPU ops) buries +// the small perf deltas tuning produces and would make verify reject real wins. +// So we run the profiler exactly ONCE (Diagnose), then strip the `tools` key from +// the context for the baseline and every candidate measurement. +// +// EXECUTE semantics: runs for real by default. Pass --no-execute / --plan to only +// diagnose-from-static + produce the ranked candidate plan without GPU measurement. +// +// SEQUENTIAL evaluation: candidates are measured one at a time — parallel GPU runs +// contend (corrupting deltas) and parallel run.sh/config edits race on the same files. + +// --- tokenizer: split a CLI string respecting single/double quotes --- +function tokenize(s) { + const out = [] + let cur = '', q = null + for (let i = 0; i < s.length; i++) { + const c = s[i] + if (q) { + if (c === q) q = null + else cur += c + } else if (c === '"' || c === "'") { + q = c + } else if (c === ' ' || c === '\t' || c === '\n') { + if (cur) { out.push(cur); cur = '' } + } else { + cur += c + } + } + if (cur) out.push(cur) + return out +} + +// --- parse a CLI string into the structured cfg the search uses --- +function parseCli(s) { + const toks = tokenize(s) + const cfg = { execute: true } + const tags = [] + for (let i = 0; i < toks.length; i++) { + const t = toks[i] + const next = () => toks[++i] + if (t === '--tags' || t === '--tag' || t === '-t') { + while (i + 1 < toks.length && !toks[i + 1].startsWith('-')) { + next().split(',').forEach(x => { if (x) tags.push(x) }) + } + } else if (t === '--additional-context' || t === '-c') { + cfg.additionalContext = next() + } else if (t === '--target') { + cfg.target = next() + } else if (t === '--candidates' || t === '-n') { + cfg.candidates = parseInt(next(), 10) + } else if (t === '--profile-tool') { + cfg.profileTool = next() + } else if (t === '--timeout') { + cfg.timeout = next() + } else if (t === '--no-execute' || t === '--plan' || t === '--dry-run') { + cfg.execute = false + } else if (t === '--execute') { + cfg.execute = true + } + } + if (tags.length) cfg.tag = tags[0] + if (tags.length > 1) cfg._extraTags = tags.slice(1) + return cfg +} + +// --- normalize args (string | object | undefined) into one cfg --- +let cfg +if (typeof args === 'string') { + cfg = parseCli(args) +} else if (args && typeof args === 'object') { + cfg = { execute: true, ...args } +} else { + cfg = { execute: true } +} + const tag = cfg.tag || 'bert' const target = cfg.target || 'throughput' const nCandidates = Math.max(1, Math.min(cfg.candidates || 4, 8)) -const execute = cfg.execute === true +const execute = cfg.execute !== false +const timeout = cfg.timeout || null + +// --- Split the user's --additional-context into a PROFILED and a CLEAN variant. --- +// profiledCtx: keeps (or adds) a `tools` entry → used once in Diagnose. +// cleanCtx: `tools` key removed → used for baseline + every candidate measurement. +let ctxObj = null +if (cfg.additionalContext) { + try { ctxObj = JSON.parse(cfg.additionalContext) } + catch (e) { log(`Warning: --additional-context is not valid JSON; passing it through verbatim and skipping clean/profiled split.`) } +} +let cleanCtx = null // JSON string, tools stripped +let profiledCtx = null // JSON string, tools present +let profileToolName = cfg.profileTool || null + +if (ctxObj && typeof ctxObj === 'object') { + // derive the profiling tool name from the context if not given explicitly + if (!profileToolName && Array.isArray(ctxObj.tools) && ctxObj.tools[0] && ctxObj.tools[0].name) + profileToolName = ctxObj.tools[0].name + if (!profileToolName) profileToolName = 'rocm_trace_lite' + + const cleanObj = { ...ctxObj } + delete cleanObj.tools + cleanCtx = JSON.stringify(cleanObj) + + const profObj = { ...ctxObj, tools: [{ name: profileToolName }] } + profiledCtx = JSON.stringify(profObj) +} else if (cfg.additionalContext) { + // not parseable as object — use verbatim for clean; still attempt a profiled variant + cleanCtx = cfg.additionalContext + if (!profileToolName) profileToolName = 'rocm_trace_lite' +} + +const cleanCtxFlag = cleanCtx ? ` --additional-context '${cleanCtx}'` : '' +const profCtxFlag = profiledCtx ? ` --additional-context '${profiledCtx}'` + : ` --additional-context '{"tools": [{"name": "${profileToolName || 'rocm_trace_lite'}"}]}'` + +if (cfg._extraTags && cfg._extraTags.length) + log(`Note: mad-tune-search tunes ONE model. Using "${tag}"; ignoring extra tags [${cfg._extraTags.join(', ')}]. Use mad-benchmark-sweep to compare multiple models.`) +log(`Tuning "${tag}" for ${target}. execute=${execute}. profile tool=${profileToolName || 'rocm_trace_lite'}. ${cleanCtx ? 'context split into clean+profiled' : 'no additional-context'}.`) + +// ── Phase 1: Baseline (CLEAN) ────────────────────────────────────────────── phase('Baseline') const BASELINE_SCHEMA = { type: 'object', @@ -24,50 +150,106 @@ const BASELINE_SCHEMA = { model: { type: 'string' }, scriptPath: { type: 'string' }, baselineSummary: { type: 'string' }, + multipleResults: { type: ['string', 'null'] }, + baselinePerf: { type: ['number', 'null'] }, + baselineMetric: { type: ['string', 'null'] }, levers: { type: 'array', items: { type: 'string' } }, }, } +const baseAction = execute + ? `If AMD GPUs are present, establish a CLEAN baseline number (profiler OFF): either reuse the most recent clean perf row for this model if one exists in perf.csv / the model's results CSV, or run "madengine run --tags ${tag} --live-output -o perf_tune_baseline.csv${cleanCtxFlag}" once and parse it. Report baselinePerf + baselineMetric.` + : `Do NOT run anything. Read the static config only; leave baselinePerf null.` const baseline = await agent( - `In the MAD repo, establish the tuning baseline for tag "${tag}". + `In the MAD repo, establish the tuning baseline for tag "${tag}" (target: ${target}). Find its models.json entry, its scripts/.../run.sh, and any config it references. Summarize the current configuration and list the tuning levers available for this -stack (env vars like MAD_MODEL_BATCH_SIZE / PYTORCH_TUNABLEOP_ENABLED / NCCL_*, -or args like tensor-parallel size, precision, gpu-memory-utilization). -Optimization target: ${target}.`, +stack (env vars like MAD_MODEL_BATCH_SIZE / PYTORCH_TUNABLEOP_ENABLED / NCCL_*/RCCL_*, +or args like tensor-parallel size, precision, gpu-memory-utilization, max-num-seqs). +Report whether the entry sets "multiple_results" (its CSV filename, or null). +${baseAction}`, { phase: 'Baseline', schema: BASELINE_SCHEMA } ) +const multiCsv = baseline.multipleResults || null +const parseHint = multiCsv + ? `This is a multiple_results model — performance is NOT a stdout line; after the run read the per-metric CSV "${multiCsv}" (and the -o file) and report the primary ${target} metric.` + : `Parse the "performance: " line from stdout.` +// ── Phase 2: Diagnose (PROFILED, once) ───────────────────────────────────── +phase('Diagnose') +const DIAG_SCHEMA = { + type: 'object', + required: ['bottleneck', 'evidence', 'hotspots'], + properties: { + bottleneck: { type: 'string', enum: ['compute', 'memory', 'communication', 'launch', 'mixed', 'unknown'] }, + evidence: { type: 'string' }, + hotspots: { type: 'array', items: { + type: 'object', + required: ['name', 'pct'], + properties: { name: { type: 'string' }, pct: { type: ['number', 'null'] }, kind: { type: 'string' } }, + } }, + recommendedLevers: { type: 'array', items: { type: 'string' } }, + tracePath: { type: ['string', 'null'] }, + }, +} +const diagAction = execute + ? `If AMD GPUs are present, profile the model ONCE: prefer reusing an existing recent trace dir (e.g. rocm_trace_lite_output/) if present and current; otherwise run "madengine run --tags ${tag} --live-output -o perf_tune_profiled.csv${profCtxFlag}". Then read the trace summary (e.g. trace_summary.txt / the trace db) to extract the top GPU-kernel hotspots with their % of GPU time. If no GPUs, classify from static config + known model characteristics and set tracePath null.` + : `Do NOT run anything. Classify the likely bottleneck from the static config, model architecture, and any EXISTING trace output already on disk (e.g. rocm_trace_lite_output/trace_summary.txt). Set tracePath to that file if found, else null.` +const diag = await agent( + `Diagnose the performance bottleneck of "${baseline.model}" to guide ${target} tuning. +Baseline config: ${baseline.baselineSummary} +${diagAction} + +Classify the bottleneck as exactly one of: compute, memory, communication, launch, mixed, unknown — using these best-practice signals: +- compute: GEMM/conv kernels (e.g. Cijk*, *gemm*, wvSplitK) dominate GPU time; high occupancy. + Levers: PYTORCH_TUNABLEOP_ENABLED, hipBLASLt tuning, fp8/lower precision, larger batch. +- memory: attention/elementwise/norm/copy kernels dominate; low arithmetic intensity / bandwidth-bound. + Levers: KV-cache dtype, kernel fusion, gpu-memory-utilization, batch size. +- communication: RCCL/NCCL collectives (AllReduce/AllGather) are a large share (multi-GPU). + Levers: NCCL_*/RCCL_* tuning, tensor-parallel vs pipeline-parallel topology. +- launch: many tiny kernels with timeline gaps / low GPU utilization. + Levers: HIP/CUDA graphs, larger batch, fewer sync points. +Return bottleneck, evidence (cite the hotspot %s), hotspots[], recommendedLevers[], tracePath.`, + { phase: 'Diagnose', schema: DIAG_SCHEMA } +) +log(`Diagnosis: ${diag.bottleneck}-bound. Top hotspots: ${(diag.hotspots || []).slice(0, 3).map(h => `${h.name} ${h.pct != null ? h.pct + '%' : ''}`).join(', ')}`) + +// ── Phase 3: Propose (profiling-informed) ────────────────────────────────── phase('Propose') const CANDS_SCHEMA = { type: 'object', required: ['candidates'], properties: { - candidates: { - type: 'array', - items: { - type: 'object', - required: ['id', 'change', 'hypothesis'], - properties: { - id: { type: 'string' }, - change: { type: 'string' }, - hypothesis: { type: 'string' }, - }, + candidates: { type: 'array', items: { + type: 'object', + required: ['id', 'change', 'hypothesis', 'evidence'], + properties: { + id: { type: 'string' }, + change: { type: 'string' }, + hypothesis: { type: 'string' }, + evidence: { type: 'string' }, + leverKind: { type: 'string', enum: ['env', 'arg', 'config', 'other'] }, }, - }, + } }, }, } const proposed = await agent( - `Propose ${nCandidates} DISTINCT tuning candidates to improve ${target} for model -"${baseline.model}". Baseline: ${baseline.baselineSummary} -Available levers: ${(baseline.levers || []).join(', ')} -Each candidate changes ONE lever (so its effect is attributable). Return id, -change (concrete value + how to apply it), and hypothesis.`, + `Propose ${nCandidates} DISTINCT tuning candidates to improve ${target} for "${baseline.model}". +Baseline: ${baseline.baselineSummary} +Diagnosis: ${diag.bottleneck}-bound. Evidence: ${diag.evidence} +Top hotspots: ${JSON.stringify((diag.hotspots || []).slice(0, 5))} +Recommended levers from diagnosis: ${(diag.recommendedLevers || []).join(', ')} +Generic available levers: ${(baseline.levers || []).join(', ')} + +Each candidate changes ONE lever (so its effect is attributable) and should TARGET the +diagnosed bottleneck. Return id, change (concrete value + how to apply it), hypothesis, +evidence (which hotspot/diagnosis finding motivates it), and leverKind: +"env" (environment variable), "arg" (run-script/CLI arg), "config" (config-file value), or "other".`, { phase: 'Propose', schema: CANDS_SCHEMA } ) const candidates = (proposed.candidates || []).slice(0, nCandidates) -log(`Evaluating ${candidates.length} candidate(s). execute=${execute}`) +log(`Evaluating ${candidates.length} candidate(s) sequentially, CLEAN (profiler off). execute=${execute}`) -// Pipeline: evaluate each candidate, then adversarially verify its claimed delta. +// ── Phase 4: Evaluate (CLEAN, sequential) + adversarial verify ───────────── const EVAL_SCHEMA = { type: 'object', required: ['id', 'status'], @@ -90,45 +272,64 @@ const VERDICT_SCHEMA = { }, } -const evaluated = await pipeline( - candidates, - (cand) => { - const action = execute - ? `If AMD GPUs are present, apply the change, run "madengine run --tags ${tag} --live-output", parse the performance line, then REVERT the change. If no GPUs, status "skipped".` - : `Do NOT execute. Produce the exact "madengine run --tags ${tag} ..." command (plus how to apply the change) and set status "planned".` - return agent( - `Evaluate tuning candidate ${cand.id} for "${baseline.model}". +const ctxHint = cleanCtx + ? `A CLEAN base --additional-context (profiler OFF) is in the command. If this candidate's leverKind is "env", apply the change by MERGING the env var into that context's "docker_env_vars" object (show the merged command you actually run); do not export it separately.` + : `If leverKind is "env", apply by adding --additional-context '{"docker_env_vars": {"VAR": "value"}}' (or exporting the var).` + +const evaluated = [] +for (const cand of candidates) { + const safeId = String(cand.id).replace(/[^A-Za-z0-9._-]/g, '_') + const outFile = `perf_tune_${safeId}.csv` + const flags = ['--tags ' + tag, '--live-output', '-o ' + outFile] + if (timeout) flags.push('--timeout ' + timeout) + const cmd = `madengine run ${flags.join(' ')}${cleanCtxFlag}` + const action = execute + ? `If AMD GPUs are present (check rocm-smi/amd-smi), apply the change, RUN the command CLEAN (profiler OFF), parse the result, then REVERT any file/config edit so the next candidate starts clean. ${parseHint} Results land in ${outFile}. If no GPUs, set status "skipped".` + : `Do NOT execute. Produce the exact command (plus precisely how to apply the change) and set status "planned".` + + const evalResult = await agent( + `Evaluate tuning candidate ${cand.id} for "${baseline.model}" (leverKind: ${cand.leverKind || 'other'}). Change: ${cand.change} Hypothesis: ${cand.hypothesis} +Motivating evidence: ${cand.evidence} +Base command (CLEAN): ${cmd} +${ctxHint} ${action} -Return id, status, command, and performance/metric if measured.`, - { label: `eval:${cand.id}`, phase: 'Evaluate', schema: EVAL_SCHEMA } - ) - }, - (evalResult, cand) => - agent( - `Adversarially review tuning candidate ${cand.id}. +Return id, status, the exact command you ran, and performance/metric if measured.`, + { label: `eval:${cand.id}`, phase: 'Evaluate', schema: EVAL_SCHEMA } + ) + + const verdict = await agent( + `Adversarially review tuning candidate ${cand.id}. +Baseline: ${baseline.baselinePerf != null ? baseline.baselinePerf + ' ' + (baseline.baselineMetric || '') : baseline.baselineSummary} Claimed result: ${JSON.stringify(evalResult)} Hypothesis was: ${cand.hypothesis} -Is the conclusion trustworthy? Default to trustworthy=false if the change was -not actually measured (planned/skipped), if only one sample was taken, or if the -delta is within noise. Return id, trustworthy, reason.`, - { label: `verify:${cand.id}`, phase: 'Evaluate', schema: VERDICT_SCHEMA } - ).then(v => ({ ...evalResult, verdict: v })) -) +Is the conclusion trustworthy? Default to trustworthy=false if the change was not actually +measured (planned/skipped), if only one sample was taken, if it ran under a profiler, or if +the delta vs baseline is within run-to-run noise (~1-2%). Return id, trustworthy, reason.`, + { label: `verify:${cand.id}`, phase: 'Evaluate', schema: VERDICT_SCHEMA } + ) + evaluated.push({ ...evalResult, _change: cand.change, _evidence: cand.evidence, verdict }) +} +// ── Phase 5: Synthesize ──────────────────────────────────────────────────── phase('Synthesize') const clean = evaluated.filter(Boolean) const report = await agent( `Synthesize a MAD tuning search for "${baseline.model}" (target: ${target}). -Baseline: ${baseline.baselineSummary} +Baseline: ${baseline.baselineSummary}${baseline.baselinePerf != null ? ` (clean baseline: ${baseline.baselinePerf} ${baseline.baselineMetric || ''})` : ''} +Diagnosis: ${diag.bottleneck}-bound — ${diag.evidence} +Top hotspots: ${JSON.stringify((diag.hotspots || []).slice(0, 5))} Evaluated candidates (JSON, each with a verdict): ${JSON.stringify(clean)} -Produce: (1) a table Candidate | Change | Status | Performance | Trustworthy; -(2) the recommended configuration — only from candidates whose verdict is -trustworthy; if none ran, present the ranked plan to test on a GPU host; -(3) the exact next command(s) to run. -Do not invent measurements.`, + +Produce: +(1) a one-line headline: the bottleneck and the best trustworthy improvement (or that none beat baseline); +(2) the diagnosis summary (what's the bottleneck and why, citing hotspot %s); +(3) a table: Candidate | Change | Evidence | Status | Performance | vs Baseline | Trustworthy; +(4) the recommended configuration — ONLY from candidates whose verdict is trustworthy; if none ran/were trustworthy, present the ranked plan to test on a GPU host; +(5) the exact next command(s) to run. +Each candidate wrote its own perf_tune_.csv (no shared-file clobbering). Do not invent measurements.`, { phase: 'Synthesize' } ) -return { model: baseline.model, target, execute, candidates: clean, report } +return { model: baseline.model, target, execute, bottleneck: diag.bottleneck, diagnosis: diag, candidates: clean, report } diff --git a/mad-automation-howto.html b/mad-automation-howto.html index 90580ef..6bbd640 100644 --- a/mad-automation-howto.html +++ b/mad-automation-howto.html @@ -954,7 +954,7 @@

3 Quick-Start: What Do You Want to Do?

Benchmark multiple tags at once
→
-
/mad-benchmark-sweep (workflow)
+
/mad-benchmark-sweep --tags bert,deepseek
Add a brand-new model
@@ -984,7 +984,7 @@

3 Quick-Start: What Do You Want to Do?

Systematic tuning search (N candidates)
→
-
/mad-tune-search (workflow)
+
/mad-tune-search --tag bert --candidates 6
@@ -1370,7 +1370,7 @@

6 Workflow Reference

mad-benchmark-sweep workflow -
Fan out a benchmark matrix across multiple MAD tags, run (or plan) each in parallel, and synthesize a comparison table.
+
Fan out a benchmark matrix across multiple MAD tags, run (or plan) each in parallel, and synthesize a comparison table. Accepts the same CLI-style flags as /mad-benchmark.
@@ -1396,20 +1396,43 @@

6 Workflow Reference

-

Args schema

- 
copy{
-  "tags":    ["bert", "pyt_vllm_llama-3.1-8b", "deepseek"],  // required
-  "nGpus":   ["4", "8"],                                    // optional, default: no override
-  "execute": true,                                           // false = plan mode (default)
-  "output":  "sweep_results/"                                 // optional
-}
+

Arguments

+

+ Accepts CLI-style flags — the same syntax as /mad-benchmark. + Pass one or more tags after --tags (comma- and/or space-separated); each tag + becomes one sweep cell. A structured JSON object is also still accepted. +

+
+ + + + + + + + + +
FlagMeaning
--tags a,b cOne or more model tags (comma and/or space separated). Required.
--additional-context '{...}'Threaded verbatim into every cell (HF token, docker_env_vars, deployment keys). A per-cell n_gpus is merged in when --ngpus is set.
--ngpus 4,8Optional GPU-count axis — multiplies the matrix (each tag × each count).
--timeout SOptional per-cell timeout in seconds.
--no-execute / --planValidate + resolve tags only, no GPU. Executes by default.
+

Example invocations

- 
copy# Plan mode — safe, no GPU required
-/mad-benchmark-sweep { "tags": ["bert", "pyt_vllm_deepseek-r1"] }
+      
copy# Execute mode (default) — AMD GPU required, context threaded into every cell
+/mad-benchmark-sweep --tags bert,pyt_vllm_qwen3-8b --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' --live-output
+
+# Plan mode — safe, no GPU required
+/mad-benchmark-sweep --tags bert pyt_vllm_deepseek-r1 --plan
 
-# Execute mode — AMD GPU required
-/mad-benchmark-sweep { "tags": ["bert", "pyt_vllm_llama-3.1-8b"], "execute": true }

+# GPU-count axis: each tag × each count
+/mad-benchmark-sweep --tags bert --ngpus 4,8
+ +
+
Executes by default
+

+ On a GPU host this workflow runs immediately. Pass --plan + (or --no-execute) to only validate commands and resolve tags. Each cell + still self-skips if no AMD GPU is present. +

+
perf.csv isolation
@@ -1444,57 +1467,142 @@

6 Workflow Reference

mad-tune-search workflow
-
Generate N candidate tuning configs for one model, run (or plan) each in parallel, adversarially verify deltas, and synthesize the winning config.
+
Profiles the model to diagnose its bottleneck, then generates profiling-informed tuning candidates, measures each cleanly and sequentially, adversarially verifies deltas, and synthesizes the winning config. Accepts the same CLI-style flags as /mad-benchmark / /mad-profile.

When to use

-

When you want to explore multiple tuning dimensions simultaneously rather than sequentially, and have the workflow verify which improvements are real.

+

When you want tuning driven by real profiling evidence — not guesswork — and have the workflow verify which improvements are real. The Diagnose phase classifies the workload (compute / memory / communication / launch-bound) and the proposals target that specific bottleneck.

+ +
+
Core design — profile to diagnose, clean-run to measure
+

+ Profiling plays two distinct roles. The profiler runs once + (Diagnose) to find what to tune — kernel hotspots and whether the model is compute-, + memory-, communication-, or launch-bound. Then it is turned off for every + measurement run, because tracing overhead (e.g. rocm_trace_lite intercepts + nearly all GPU ops) buries the small perf deltas tuning produces and would make the verify + step reject real wins. The workflow automatically splits a single + --additional-context into a profiled variant (keeps tools) + for Diagnose and a clean variant (tools stripped) for the baseline and + all candidate measurements. +

+

Phases

Phase 1
Baseline
-
Read the model and establish current config + perf
+
Establish a clean baseline number (profiler off)
Phase 2
-
Propose
-
Generate N distinct single-lever candidates
+
Diagnose
+
Profile once; read hotspots; classify the bottleneck (compute/memory/comm/launch)
Phase 3
-
Evaluate
-
Run each + adversarially verify the claimed delta
+
Propose
+
Generate N single-lever candidates, each citing the profiling evidence
Phase 4
+
Evaluate
+
Measure each clean, one at a time + adversarially verify the delta inline
+
+
+
Phase 5
Synthesize
Pick the winning config from trustworthy results only
-

Args schema

- 
copy{
-  "tag":        "bert",           // required
-  "target":     "throughput",     // "throughput" | "latency" (default: throughput)
-  "candidates": 4,                // 1–8, default: 4
-  "execute":    true             // false = plan mode (default)
-}
+

Arguments

+

+ Accepts CLI-style flags — the same syntax as /mad-benchmark. + Tunes one model; if multiple tags are given, the first is used and the rest are + ignored (use mad-benchmark-sweep to compare models). A structured JSON object + is also still accepted. +

+
+ + + + + + + + + + + +
FlagMeaning
--tag NAMEThe model to tune (--tags also accepted). Required.
--candidates NNumber of single-lever candidates, 1–8 (default 4).
--target throughput|latencyOptimization target (default throughput).
--additional-context '{...}'Split into a profiled variant (Diagnose) and a clean variant (baseline + measurements). Threaded into every command (HF token, docker_env_vars). A tools entry here sets the profiler used in Diagnose. For env-lever candidates the agent merges the var into the clean context's docker_env_vars.
--profile-tool NAMEProfiler for the Diagnose phase (default rocm_trace_lite, or taken from a tools entry in --additional-context).
--timeout SOptional per-run timeout in seconds.
--no-execute / --planProduce the ranked candidate plan only, no GPU. Executes by default.
+

Example invocations

- 
copy# Plan 6 candidates for bert, no GPU needed
-/mad-tune-search { "tag": "bert", "candidates": 6 }
+      
copy# Execute (default): diagnose with rocm_trace_lite, then tune throughput clean.
+# The "tools" key drives Diagnose; it is auto-stripped for the clean measurement runs.
+/mad-tune-search --tag pyt_vllm_qwen3-8b --candidates 6 --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU", "tools": [{"name": "rocm_trace_lite"}]}'
+
+# Pick the diagnosis profiler explicitly (e.g. memory roofline)
+/mad-tune-search --tag pyt_vllm_qwen3-8b --profile-tool rocprofv3_memory
+
+# Plan mode — diagnose from existing trace + static config, rank candidates, no GPU run
+/mad-tune-search --tag bert --candidates 6 --plan
 
-# Execute 4 candidates for latency optimization
-/mad-tune-search { "tag": "pyt_vllm_llama-3.1-8b", "target": "latency", "execute": true }

+# Latency optimization
+/mad-tune-search --tag pyt_vllm_llama-3.1-8b --target latency
+ +
+
Sequential evaluation — and why
+

+ Candidates are measured one at a time, not in parallel. Running several + benchmarks at once on the same GPUs makes them contend for compute and produces + meaningless deltas; and candidates that edit run.sh/configs in parallel + would race on the same files and clobber each other's apply/revert. GPUs are a serial + resource for clean benchmarking, so one-at-a-time is both correct and safe. The cheap + adversarial-verify step still runs inline right after each measurement. Each candidate + writes its own perf_tune_<id>.csv. +

+
+ +

Bottleneck → lever map (best practice)

+

The Diagnose phase classifies the workload from kernel hotspots, which dictates the levers the proposals target:

+
+ + + + + + + + + + + + + + + + + + + + + + + + +
BottleneckProfiling signalLevers to try
Compute-boundGEMM/conv kernels dominate GPU time (e.g. Cijk*, *gemm*, wvSplitK); high occupancy. Signal: rocm_trace_lite/rpd hotspots, rocprofv3_compute occupancy.PYTORCH_TUNABLEOP_ENABLED, hipBLASLt tuning, fp8 / lower precision, larger batch size
Memory-boundAttention / elementwise / norm / copy kernels dominate; low arithmetic intensity. Signal: rocprofv3_memory bandwidth.KV-cache dtype, kernel fusion, gpu-memory-utilization, batch size
Communication-boundRCCL/NCCL collectives (AllReduce/AllGather) are a large share (multi-GPU). Signal: rocprofv3_communication, rccl_trace.NCCL_*/RCCL_* tuning, tensor-parallel vs pipeline-parallel topology
Launch-boundMany tiny kernels with timeline gaps / low GPU utilization. Signal: timeline gaps in rocm_trace_lite.HIP/CUDA graphs, larger batch, fewer sync points
+

Adversarial verification

After each candidate is evaluated, a separate subagent reviews the claimed delta. It marks results as trustworthy: false if the change was not actually - measured (planned/skipped), only one sample was taken, or the delta is within noise. + measured (planned/skipped), only one sample was taken, it ran under a profiler, + or the delta vs baseline is within run-to-run noise (~1–2%). The final recommended configuration comes only from trustworthy results.

From cb8c6034cb97361364a5dadf43a4fad64f009262 Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Mon, 1 Jun 2026 03:50:30 +0000 Subject: [PATCH 05/13] Update how-to doc with live tuning findings and timeout gotchas Add real results and operational lessons from the Qwen3-8B profiling and tuning session on MI350X: - TunableOp warmup warning: first cold run collapses throughput to ~0.2-1 tok/s during online GEMM benchmarking; measurements are only valid on warm subsequent runs. Added to env var card and Tips section. - New red callout in tune-search: Docker containers outlive the Claude session, so a slow candidate can stall the whole sequential search and leave a config edit (e.g. extended.yaml) unapplied. Always pass --timeout to bound each candidate run. - New Tips entry "Qwen3-8B live tuning findings": rocm_trace_lite kernel breakdown (Cijk GEMM 27% + wvSplitK 21% = compute-bound), and the headline result -- max_concurrency=32 delivered 7993 tok/s throughput vs 422 tok/s baseline (18.9x), confirming concurrency as the primary vLLM serving lever. Co-Authored-By: Claude Opus 4 --- mad-automation-howto.html | 74 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-) diff --git a/mad-automation-howto.html b/mad-automation-howto.html index 6bbd640..b18a77f 100644 --- a/mad-automation-howto.html +++ b/mad-automation-howto.html @@ -1605,6 +1605,21 @@

6 Workflow Reference

or the delta vs baseline is within run-to-run noise (~1–2%). The final recommended configuration comes only from trustworthy results.

+ +
+
Set --timeout for long-running candidates
+

+ Sequential evaluation means one slow candidate stalls the entire search. + Some levers (e.g. PYTORCH_TUNABLEOP_ENABLED) have a pathologically + slow first run (online GEMM benchmarking collapses throughput to <1 tok/s). + The Docker container launched by madengine outlives the Claude session — if the + session ends mid-run the container keeps going but the workflow never completes + and the config edit is not reverted. Always pass --timeout to bound + each candidate, and confirm extended.yaml (or any edited config) is + clean after an interrupted run. +

+ 
/mad-tune-search --tag pyt_vllm_qwen3-8b --timeout 1200
+
@@ -1750,7 +1765,7 @@

8 Environment Variables

PYTORCH_TUNABLEOP_ENABLED
-
Enable PyTorch TunableOp for auto-tuned GEMM kernels. A common throughput tuning lever for PyTorch models.
+
Enable PyTorch TunableOp for auto-tuned GEMM kernels. A common throughput tuning lever for PyTorch models. First run is very slow (online GEMM tuning collapses throughput to ~0.2–1 tok/s); results are only meaningful on a warm second run with cached kernels. Set --timeout generously or do a throwaway warmup pass before measuring.
NCCL_* / RCCL_*
@@ -1843,6 +1858,63 @@

9 Tips and Gotchas

+
+ TunableOp needs a warmup — don't measure cold +
+

+ PYTORCH_TUNABLEOP_ENABLED=1 triggers online GEMM auto-tuning on + the first run. During this phase throughput collapses to roughly + 0.2–1 tok/s as PyTorch benchmarks every matrix-multiply shape. The actual improvement + only shows on subsequent runs using the cached kernel selection. +

+

+ Measuring TunableOp cold produces a misleading (and huge) regression that the verify + step will correctly mark untrustworthy — but only after burning 30+ minutes of GPU time. + Best practice: +

+
    +
  • Run a throwaway warmup pass first (--skip-model-run then one full run) to populate the cache.
    • +
  • Always set --timeout so a single candidate cannot stall the whole tune-search.
    • +
  • The profiling bottleneck data guides whether TunableOp is even worth trying — only relevant for compute-bound (GEMM-dominant) workloads.
    • +
+
+
+ +
+ Qwen3-8B live tuning findings (MI350X / gfx950) +
+

+ From a live profiling + tuning run on 8× AMD Instinct MI350X: +

+

Profiling result (rocm_trace_lite, 397,292 GPU ops, 8.6% GPU utilization):

+
+ + + + + + + + +
Kernel% GPU timeImplication
Cijk_Alik_Bljk_BBS_BH_Bias_HA… (GEMM)27.1%Compute-bound — GEMM/wvSplitK dominate → target with TunableOp, hipBLASLt, higher concurrency
wvSplitK20.6%
_fwd_kernel (attention)8.3%Memory-bound secondary; paged attention
__amd_rocclr_copyBuffer7.0%Data movement overhead
+
+

Best candidate result — max_concurrency=32 (vs baseline concurrency=1):

+
+ + + + + + +
MetricBaselineTunedDelta
throughput_tot~422 tok/s7,993 tok/s+18.9×
throughput_gen~211 tok/s3,996 tok/s+18.9×
+
+

+ Baseline was run at max_concurrency=1 (10 prompts) — a latency-optimized + config. Raising concurrency is the primary throughput lever for vLLM serving workloads. +

+
+
+
tools.json — authoritative profiler name list
From 4113af41c7a5c1531756a05e8d880d8c852e2bae Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Mon, 1 Jun 2026 16:01:13 +0000 Subject: [PATCH 06/13] Fix and improve architecture diagram in MAD automation how-to Correct the architecture SVG so all 6 slash commands, both workflows, and the inline /mad-validate path are represented accurately. Give /mad-profile its own box, route /mad-validate to an inline-script node, split the two workflows with distinct fan-out targets, and color-code arrows with a legend. Co-Authored-By: Claude Opus 4 --- mad-automation-howto.html | 235 ++++++++++++++++++++++---------------- 1 file changed, 138 insertions(+), 97 deletions(-) diff --git a/mad-automation-howto.html b/mad-automation-howto.html index b18a77f..c4ab4be 100644 --- a/mad-automation-howto.html +++ b/mad-automation-howto.html @@ -803,141 +803,182 @@

2 Architecture

- + - + - + + + + + + + + + + + + + - USER - AGENTS - ENGINE + USER + AGENTS + ENGINE - - + + + + + SLASH COMMANDS + WORKFLOWS - + - - /mad-benchmark - /mad-profile + + /mad-benchmark + build · run + + + + /mad-profile + + trace tool - - /mad-add-model - scaffold files + + /mad-add-model + scaffold - - /mad-report - parse results + + /mad-report + analyze csv - - /mad-tune - measure-change + + /mad-tune + measure-loop - - /mad-validate - GPU-free lint + + /mad-validate + GPU-free lint - - - benchmark-sweep - tune-search + + + mad-benchmark-sweep + matrix → compare - + + mad-tune-search + diagnose → tune + + - - benchmark-runner - Bash · Read · Grep · Glob - run & profile + + benchmark-runner + Bash · Read · Grep · Glob + run & profile - - model-author - Read · Write · Edit · Glob - scaffold files + + model-author + Read · Write · Edit · Glob + scaffold files - - perf-analyst - Read · Grep · Bash (r/o) - parse & compare CSV + + perf-analyst + Read · Grep · Bash (r/o) + parse & compare - - tuner - Read · Edit · Bash · Grep - measure-change-measure + + tuner + Read · Edit · Bash · Grep + measure-change + + + + inline script + json · paths · header + no subagent · GPU-free - - madengine - build · run · discover · report · database - Docker / SLURM / Kubernetes - - - - models.json - 142 model entries - tags · dockerfile · scripts - - - - docker/ + scripts/ - *.ubuntu.amd.Dockerfile - run.sh + perf line - performance: N unit - - - - perf.csv - one row per run - model · perf · metric - status · gpu_arch + + madengine + build · run · discover · report · database + Docker / SLURM / Kubernetes + + + + MAD repo files + models.json · docker/*.Dockerfile · scripts/run.sh + perf.csv · perf_entry_super.json (performance: N unit) - - AMD GPU - required - for run/build - - - - - - - - - - - - - - - - - - - - + + AMD GPU + required for build / run + + + + + + + + + + + + + + fan-out + + + + + + + + + + + + + + + + + + + +
+

+ command → agent + workflow fan-out + agent → engine / files + GPU-free path + needs AMD GPU +

+

- Each slash command maps to a subagent. Workflows fan out to multiple parallel - subagent calls, then synthesize the results. Agents that need to run madengine always - check for AMD GPUs first — if none are present they produce a plan (the exact commands to - run on a GPU host) instead of executing. + Five of the six slash commands dispatch a subagent; /mad-validate runs an + inline script (no subagent, no GPU). The two workflows fan out to + multiple subagent calls in parallel — mad-benchmark-sweep drives the + benchmark-runner across a tag matrix, mad-tune-search drives the tuner through a + diagnose-then-measure search — then synthesize the results. Agents that need to run madengine + always check for AMD GPUs first; if none are present they produce a plan (the exact commands + to run on a GPU host) instead of executing.

From fb83eb6eb9167acb8bb5e17732e595c84ee7286a Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Fri, 5 Jun 2026 08:40:31 -0500 Subject: [PATCH 07/13] Clarify reference_db/knowledge_base are planned, not present The Future section described these paths as existing scaffolding, but they are not in the repo. Reword to "planned" so Claude Code sessions do not assume the files/data are available. Co-Authored-By: Claude Opus 4 --- CLAUDE.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b40824c..0c5648f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -122,9 +122,9 @@ Pre-built profiling context files live in the madengine package's `MAD_SECRETS_HFTOKEN` (HuggingFace token), `MAD_MODEL_NAME`, `MAD_RUNTIME_NGPUS`, `MAD_SYSTEM_GPU_ARCHITECTURE`, `MAD_MODEL_BATCH_SIZE`. -## Future (skeleton, not yet wired) +## Future (not yet wired) `reference_db/mad_agent.db` (tables: `model_baselines`, `optimization_history`, -`best_configurations`, `learned_patterns`) and `knowledge_base/` exist as an -empty scaffold for a future persistent optimization-memory layer. Not populated -yet — do not assume data is present. +`best_configurations`, `learned_patterns`) and `knowledge_base/` are planned +for a future persistent optimization-memory layer. They are not present or +populated yet — do not assume data is available. From 39d01e2171708096a7a7cf1e0e6c74f95d0710c1 Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Fri, 5 Jun 2026 08:46:42 -0500 Subject: [PATCH 08/13] Fix smoke-test note to reference real dummy_multi model The benchmark-runner agent claimed no dummy model exists, but models.json has dummy_multi (tag "dummies"). Point the agent at it as the lightweight smoke-test target. Co-Authored-By: Claude Opus 4 --- .claude/agents/mad-benchmark-runner.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/agents/mad-benchmark-runner.md b/.claude/agents/mad-benchmark-runner.md index 9e2fea2..c99ed50 100644 --- a/.claude/agents/mad-benchmark-runner.md +++ b/.claude/agents/mad-benchmark-runner.md @@ -33,9 +33,9 @@ Execution policy: - `madengine run`/`build` need AMD GPUs. Before running, check for GPUs (`rocm-smi` or `amd-smi`). If none are present, DO NOT run — instead print the exact command(s) the user should run on a GPU host, and stop. -- Smoke-test wiring with a single small tag before large sweeps. (There is no - `dummy` model in this repo's `models.json` — confirm a real tag with - `madengine discover`.) +- Smoke-test wiring with a single small tag before large sweeps. Prefer a + lightweight existing model such as `dummy_multi` (tag `dummies`) when + appropriate, and always confirm the selected tag with `madengine discover`. Report: the resolved model list, the exact command, required env vars, and (if run) where results landed (`perf.csv` by default). From 3566d29ce1b189e6484dcfef8ba4b31e9228041d Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Fri, 5 Jun 2026 09:05:44 -0500 Subject: [PATCH 09/13] Add madengine run/build to settings.json allow-list Without these, benchmark/profile/tuning agents in execute mode require manual approval for every run command, blocking automation on GPU hosts. Co-Authored-By: Claude Opus 4 --- .claude/settings.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.claude/settings.json b/.claude/settings.json index a4299f5..b4bbaa2 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -2,6 +2,8 @@ "permissions": { "allow": [ "Bash(madengine discover *)", + "Bash(madengine run *)", + "Bash(madengine build *)", "Bash(python3 -m json.tool *)", "Bash(git status)", "Bash(git diff *)", From 6452c801b8694c6b34fbdd967de7ea9e28b73230 Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Fri, 5 Jun 2026 09:18:34 -0500 Subject: [PATCH 10/13] Add madengine pre-flight checks to all execution agents/commands/workflows Before any madengine invocation, agents now check whether madengine is on PATH. If missing and requirements.txt is present (i.e. we're in the MAD repo), they auto-install via pip. Otherwise they print clear install/clone instructions and halt. A secondary check warns when models.json is absent, catching "wrong directory" mistakes early. Affected: mad-benchmark-runner, mad-tuner, mad-model-author agents; mad-benchmark, mad-profile, mad-tune, mad-add-model commands; mad-benchmark-sweep and mad-tune-search workflows. Not changed: mad-perf-analyst, mad-report, mad-validate (read-only/GPU-free). Co-Authored-By: Claude Opus 4 --- .claude/agents/mad-benchmark-runner.md | 18 ++++++++++++++++++ .claude/agents/mad-model-author.md | 19 +++++++++++++++++++ .claude/agents/mad-tuner.md | 18 ++++++++++++++++++ .claude/commands/mad-add-model.md | 17 +++++++++++++++++ .claude/commands/mad-benchmark.md | 17 +++++++++++++++++ .claude/commands/mad-profile.md | 17 +++++++++++++++++ .claude/commands/mad-tune.md | 17 +++++++++++++++++ .claude/workflows/mad-benchmark-sweep.js | 9 +++++++++ .claude/workflows/mad-tune-search.js | 9 +++++++++ 9 files changed, 141 insertions(+) diff --git a/.claude/agents/mad-benchmark-runner.md b/.claude/agents/mad-benchmark-runner.md index c99ed50..240c172 100644 --- a/.claude/agents/mad-benchmark-runner.md +++ b/.claude/agents/mad-benchmark-runner.md @@ -29,6 +29,24 @@ When invoked: build_manifest.json` executes from it (skips rebuild). 3. Note required env vars (e.g. `export MAD_SECRETS_HFTOKEN=...` for HF models). +Pre-flight — before any `madengine` invocation, run this Bash block: +```bash +if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi +fi +if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." +fi +``` + Execution policy: - `madengine run`/`build` need AMD GPUs. Before running, check for GPUs (`rocm-smi` or `amd-smi`). If none are present, DO NOT run — instead print the diff --git a/.claude/agents/mad-model-author.md b/.claude/agents/mad-model-author.md index 10f90c3..410d8a5 100644 --- a/.claude/agents/mad-model-author.md +++ b/.claude/agents/mad-model-author.md @@ -35,6 +35,25 @@ When invoked: a single stdout line. Use this only when the template you copied does; do not mix the two contracts. +Pre-flight — before any `madengine` invocation (including `madengine discover`), +run this Bash block: +```bash +if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi +fi +if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." +fi +``` + Rules: - The output contract is hard: either the `performance: ` stdout line, OR a `multiple_results` CSV declared in models.json. Never ship a script diff --git a/.claude/agents/mad-tuner.md b/.claude/agents/mad-tuner.md index 5f4c91e..32d0d18 100644 --- a/.claude/agents/mad-tuner.md +++ b/.claude/agents/mad-tuner.md @@ -22,6 +22,24 @@ When invoked: 4. Keep a change only if it improves the metric without breaking the run (`status == SUCCESS`). Revert regressions. +Pre-flight — before any `madengine` invocation, run this Bash block: +```bash +if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi +fi +if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." +fi +``` + Rules: - Measuring requires AMD GPUs. If none are present (`rocm-smi`/`amd-smi` absent), do NOT execute — instead produce a ranked list of candidate changes with diff --git a/.claude/commands/mad-add-model.md b/.claude/commands/mad-add-model.md index 0957ce1..f5acb64 100644 --- a/.claude/commands/mad-add-model.md +++ b/.claude/commands/mad-add-model.md @@ -6,6 +6,23 @@ argument-hint: [base notes / repo url] Add a new model to MAD named `$1`. Extra context: $ARGUMENTS Use the `mad-model-author` subagent. It should: +0. Pre-flight: check madengine is installed and cwd is the MAD repo root. + ```bash + if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi + fi + if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." + fi + ``` 1. Pick the closest existing model of the same framework as a template. 2. Create the `models.json` entry, `docker/$1.ubuntu.amd.Dockerfile` (with the `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}` header), and diff --git a/.claude/commands/mad-benchmark.md b/.claude/commands/mad-benchmark.md index 28df6d0..39f5354 100644 --- a/.claude/commands/mad-benchmark.md +++ b/.claude/commands/mad-benchmark.md @@ -6,6 +6,23 @@ argument-hint: [extra options] Benchmark `$ARGUMENTS` with madengine. Use the `mad-benchmark-runner` subagent. It should: +0. Pre-flight: check madengine is installed and cwd is the MAD repo root. + ```bash + if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi + fi + if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." + fi + ``` 1. Confirm the tag matches real models via `madengine discover --tags $1`. 2. Assemble the `madengine run --tags $1 --live-output` command (add `-o`, `--timeout`, profiling `--additional-context`, or `slurm`/`k8s` keys as the diff --git a/.claude/commands/mad-profile.md b/.claude/commands/mad-profile.md index 94bfd19..9a45b1d 100644 --- a/.claude/commands/mad-profile.md +++ b/.claude/commands/mad-profile.md @@ -6,6 +6,23 @@ argument-hint: [tool: rocprofv3_compute|rpd|rccl_trace|...] Profile `$ARGUMENTS`. Use the `mad-benchmark-runner` subagent with profiling enabled. It should: +0. Pre-flight: check madengine is installed and cwd is the MAD repo root. + ```bash + if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi + fi + if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." + fi + ``` 1. Pick the profiling tool (default `rocprofv3_compute` if unspecified). Common names: `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`, `rocprofv3_communication`, `rocm_trace_lite`, `rccl_trace`, diff --git a/.claude/commands/mad-tune.md b/.claude/commands/mad-tune.md index 185c307..11ed8aa 100644 --- a/.claude/commands/mad-tune.md +++ b/.claude/commands/mad-tune.md @@ -6,6 +6,23 @@ argument-hint: [target: throughput|latency] [lever hints] Tune `$ARGUMENTS`. Use the `mad-tuner` subagent. It should: +0. Pre-flight: check madengine is installed and cwd is the MAD repo root. + ```bash + if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi + fi + if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." + fi + ``` 1. Establish the baseline (current `run.sh`/config + `perf.csv` row). 2. Propose tuning levers (env vars like `MAD_MODEL_BATCH_SIZE`, `PYTORCH_TUNABLEOP_ENABLED`, `NCCL_*`/`RCCL_*`; or args like tensor-parallel diff --git a/.claude/workflows/mad-benchmark-sweep.js b/.claude/workflows/mad-benchmark-sweep.js index 77fd2d6..9bb47bf 100644 --- a/.claude/workflows/mad-benchmark-sweep.js +++ b/.claude/workflows/mad-benchmark-sweep.js @@ -153,6 +153,15 @@ const results = await parallel(matrix.map((cell, i) => () => { return agent( `MAD benchmark sweep cell "${label}". +Pre-flight: before running madengine, verify it is installed: + if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found. Install: pip install git+https://github.com/ROCm/madengine.git@main"; exit 1 + fi + fi + [ -f models.json ] || echo "[pre-flight] Warning: not in MAD repo root." Command: ${cmd} ${action} Return the cell label, the exact command, status, and performance/metric if available.`, diff --git a/.claude/workflows/mad-tune-search.js b/.claude/workflows/mad-tune-search.js index 5772a32..9e1fd52 100644 --- a/.claude/workflows/mad-tune-search.js +++ b/.claude/workflows/mad-tune-search.js @@ -161,6 +161,15 @@ const baseAction = execute : `Do NOT run anything. Read the static config only; leave baselinePerf null.` const baseline = await agent( `In the MAD repo, establish the tuning baseline for tag "${tag}" (target: ${target}). +Pre-flight: before running madengine, verify it is installed: + if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found. Install: pip install git+https://github.com/ROCm/madengine.git@main"; exit 1 + fi + fi + [ -f models.json ] || echo "[pre-flight] Warning: not in MAD repo root." Find its models.json entry, its scripts/.../run.sh, and any config it references. Summarize the current configuration and list the tuning levers available for this stack (env vars like MAD_MODEL_BATCH_SIZE / PYTORCH_TUNABLEOP_ENABLED / NCCL_*/RCCL_*, From 4fba573b1265a15d9be58dbba0583895aab40d8a Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Mon, 8 Jun 2026 21:52:45 -0500 Subject: [PATCH 11/13] Migrate mad-* slash commands to Claude Code Skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the 6 legacy .claude/commands/mad-*.md with proper Skills under .claude/skills/, following the current Claude Code best practice where commands and skills are unified and skills are the recommended form. Changes: - Add .claude/skills/mad-{benchmark,profile,tune,add-model}/SKILL.md with disable-model-invocation:true (manual-only; these build/run on AMD GPUs) - Add .claude/skills/mad-{report,validate}/SKILL.md as auto-invocable (read-only, safe for Claude to trigger from plain-English intent) - All 6 skills use context:fork to dispatch to their curated subagent, collapsing the old command→subagent indirection into one file per skill - Add .claude/skills/mad-common/preflight.sh — single copy of the madengine install + repo-root check, injected via dynamic context (was duplicated 7x) - Add .claude/skills/mad-validate/scripts/validate.py — extracts the embedded Python heredoc into a real bundled file (tested: 142 models, 0 errors) - Slim the 4 agents to pure system prompts (role + tools + standing rules); per-invocation steps and pre-flight blocks move to the skills - Delete all 6 .claude/commands/mad-*.md (skills supersede them with the same /mad-* names; existing muscle memory unchanged) - Update CLAUDE.md to reference the new skills layout Co-Authored-By: Claude Opus 4 --- .claude/agents/mad-benchmark-runner.md | 71 +++++--------- .claude/agents/mad-model-author.md | 76 +++++---------- .claude/agents/mad-perf-analyst.md | 28 +++--- .claude/agents/mad-tuner.md | 56 ++++------- .claude/commands/mad-add-model.md | 34 ------- .claude/commands/mad-benchmark.md | 32 ------- .claude/commands/mad-profile.md | 38 -------- .claude/commands/mad-report.md | 16 ---- .claude/commands/mad-tune.md | 35 ------- .claude/commands/mad-validate.md | 92 ------------------- .claude/skills/mad-add-model/SKILL.md | 40 ++++++++ .claude/skills/mad-benchmark/SKILL.md | 33 +++++++ .claude/skills/mad-common/preflight.sh | 23 +++++ .claude/skills/mad-profile/SKILL.md | 32 +++++++ .claude/skills/mad-report/SKILL.md | 25 +++++ .claude/skills/mad-tune/SKILL.md | 37 ++++++++ .claude/skills/mad-validate/SKILL.md | 33 +++++++ .../skills/mad-validate/scripts/validate.py | 73 +++++++++++++++ CLAUDE.md | 8 +- 19 files changed, 381 insertions(+), 401 deletions(-) delete mode 100644 .claude/commands/mad-add-model.md delete mode 100644 .claude/commands/mad-benchmark.md delete mode 100644 .claude/commands/mad-profile.md delete mode 100644 .claude/commands/mad-report.md delete mode 100644 .claude/commands/mad-tune.md delete mode 100644 .claude/commands/mad-validate.md create mode 100644 .claude/skills/mad-add-model/SKILL.md create mode 100644 .claude/skills/mad-benchmark/SKILL.md create mode 100755 .claude/skills/mad-common/preflight.sh create mode 100644 .claude/skills/mad-profile/SKILL.md create mode 100644 .claude/skills/mad-report/SKILL.md create mode 100644 .claude/skills/mad-tune/SKILL.md create mode 100644 .claude/skills/mad-validate/SKILL.md create mode 100644 .claude/skills/mad-validate/scripts/validate.py diff --git a/.claude/agents/mad-benchmark-runner.md b/.claude/agents/mad-benchmark-runner.md index 240c172..cd9bece 100644 --- a/.claude/agents/mad-benchmark-runner.md +++ b/.claude/agents/mad-benchmark-runner.md @@ -5,55 +5,32 @@ tools: Bash, Read, Grep, Glob model: inherit --- -You turn a benchmarking intent into the correct `madengine` invocation and, when -a GPU host is available, run it. +You turn a benchmarking or profiling intent into the correct `madengine` +invocation and, when an AMD GPU host is available, run it. -When invoked: -1. Resolve which models the user means. Use `madengine discover --tags ` - (read-only, no GPU) to confirm tags match real models in `models.json`. If - the intent is fuzzy, grep `models.json` for candidates and confirm. -2. Build the command (madengine v2.1.0 Typer CLI): - - Base: `madengine run --tags --live-output` (full build+run). - - Output file: `-o ` when the user wants results kept separately. - - Timeout: `--timeout ` for long training runs (default 7200). - - Profiling: `--additional-context '{"tools": [{"name": ""}]}'` - (e.g. `rocprofv3_compute`, `rpd`, `rccl_trace`). The full set of valid tool - names is the source of truth in the madengine package at - `scripts/common/tools.json` (23+ tools, incl. `rocprofv3_full`, - `rocblas_trace`, `hipblaslt_trace`, `miopen_trace`, `rocprof_sys`). - - Multi-node: add a `"slurm": {...}` or `"k8s": {...}` key to - `--additional-context` — presence of the key selects the target. An explicit - `"deploy": "slurm"`/`"k8s"` key works too; neither key → local Docker. - - Build/run split (build once, run many): `madengine build --tags - [-r REGISTRY]` writes `build_manifest.json`, then `madengine run -m - build_manifest.json` executes from it (skips rebuild). -3. Note required env vars (e.g. `export MAD_SECRETS_HFTOKEN=...` for HF models). +This is the fork target for the `mad-benchmark` and `mad-profile` skills — the +invoking skill supplies the concrete task and pre-flight. Your job is to apply +the conventions below correctly. -Pre-flight — before any `madengine` invocation, run this Bash block: -```bash -if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - echo "[pre-flight] madengine not found. Installing from requirements.txt..." - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found and requirements.txt is missing." - echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" - echo " Or clone MAD and run from its root (which has requirements.txt)." - exit 1 - fi -fi -if [ ! -f models.json ]; then - echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." -fi -``` +Conventions (madengine v2.1.0 Typer CLI): +- Resolve models with `madengine discover --tags ` (read-only, no GPU) + before running. Confirm fuzzy intent against `models.json`. +- Base command: `madengine run --tags --live-output`. Add `-o `, + `--timeout `, or `--additional-context '{...}'` as the request implies. +- Profiling: `--additional-context '{"tools": [{"name": ""}]}'`. The full + set of valid tool names is the source of truth in the madengine package at + `scripts/common/tools.json` (23+ tools). +- Deploy target is inferred from the context key: `"slurm"` → SLURM, + `"k8s"`/`"kubernetes"` → Kubernetes, neither → local Docker. +- Build-once/run-many: `madengine build --tags [-r REGISTRY]` writes + `build_manifest.json`; `madengine run -m build_manifest.json` runs from it. Execution policy: -- `madengine run`/`build` need AMD GPUs. Before running, check for GPUs - (`rocm-smi` or `amd-smi`). If none are present, DO NOT run — instead print the - exact command(s) the user should run on a GPU host, and stop. -- Smoke-test wiring with a single small tag before large sweeps. Prefer a - lightweight existing model such as `dummy_multi` (tag `dummies`) when - appropriate, and always confirm the selected tag with `madengine discover`. +- `madengine run`/`build` need AMD GPUs. Check `rocm-smi`/`amd-smi` first. If none + are present, DO NOT run — print the exact command(s) for a GPU host and stop. +- Note required env vars (e.g. `export MAD_SECRETS_HFTOKEN=...` for HF models). +- Profiling adds overhead — a perf number measured under a profiler is not a clean + benchmark number; say so. -Report: the resolved model list, the exact command, required env vars, and -(if run) where results landed (`perf.csv` by default). +Report: resolved model list, the exact command, required env vars, and (if run) +where results landed (`perf.csv` by default). diff --git a/.claude/agents/mad-model-author.md b/.claude/agents/mad-model-author.md index 410d8a5..113de22 100644 --- a/.claude/agents/mad-model-author.md +++ b/.claude/agents/mad-model-author.md @@ -1,66 +1,36 @@ --- name: mad-model-author description: Scaffolds a new MAD model — the models.json entry, the docker/.ubuntu.amd.Dockerfile, and the scripts//run.sh that emits the performance line. Use when adding a new model or workload to MAD. -tools: Read, Write, Edit, Grep, Glob +tools: Read, Write, Edit, Grep, Glob, Bash model: inherit --- You scaffold new model workloads in the MAD repository following its conventions. -When invoked: -1. Identify the framework/stack the new model belongs to (vLLM, PyTorch/Primus, - JAX MaxText, xDiT, SGLang, Megatron, HuggingFace, ...). -2. Find the CLOSEST existing model of that stack in `models.json` and read its - entry, its `docker/.ubuntu.amd.Dockerfile`, and its `scripts/.../run.sh`. - Reuse that as your template — do not invent new patterns when one exists. -3. Produce three artifacts: +This is the fork target for the `mad-add-model` skill — it supplies the concrete +task and pre-flight. Your job is to apply the conventions below correctly. - a. A new `models.json` entry. Required fields: `name`, `url`, `dockerfile`, - `scripts`, `n_gpus`, `owner`, `training_precision`, `tags`. Name follows - `{framework}_{project}_{workload}`. Add appropriate tags (framework + - precision + model family). Keep `models.json` valid JSON. +Method: +- Find the CLOSEST existing model of the same stack (vLLM, PyTorch/Primus, JAX + MaxText, xDiT, SGLang, Megatron, HuggingFace, ...) and reuse its `models.json` + entry, Dockerfile, and `run.sh` as a template. Do not invent new patterns when + one exists. +- Name new models `{framework}_{project}_{workload}`. - b. `docker/.ubuntu.amd.Dockerfile`. First line MUST be: - `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}` - Prefer reusing an existing Dockerfile of the same stack (point the - `dockerfile` field at it) rather than creating a near-duplicate. +The output contract is hard — every model MUST satisfy exactly one of: + 1. its run script echoes `performance: ` (parsed from the + workload's own log output), OR + 2. the entry sets `"multiple_results": ".csv"` and the script WRITES that + CSV (one row per result). +Never ship a script that emits neither (madengine records no performance value), +and never mix the two contracts. - c. `scripts//run.sh`. It must satisfy ONE of the two output contracts: - - Single result (default): end by printing exactly - `echo "performance: $performance "`, where `$performance` is - parsed from the workload's own log output. Model this on the template. - - Multiple results: have the script WRITE its own CSV (one row per - result) and set `"multiple_results": ".csv"` in the - models.json entry. madengine then ingests that CSV instead of grepping - a single stdout line. Use this only when the template you copied does; - do not mix the two contracts. +Dockerfile rule: the first line MUST be +`# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}`. Prefer pointing the +`dockerfile` field at an existing same-stack Dockerfile over a near-duplicate. -Pre-flight — before any `madengine` invocation (including `madengine discover`), -run this Bash block: -```bash -if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - echo "[pre-flight] madengine not found. Installing from requirements.txt..." - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found and requirements.txt is missing." - echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" - echo " Or clone MAD and run from its root (which has requirements.txt)." - exit 1 - fi -fi -if [ ! -f models.json ]; then - echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." -fi -``` +Verification (GPU-free): `python3 -m json.tool models.json` must parse, and +`madengine discover --tags ` must list the new entry. Do NOT run +`madengine run` — it needs GPUs; state the GPU-host command instead. -Rules: -- The output contract is hard: either the `performance: ` stdout - line, OR a `multiple_results` CSV declared in models.json. Never ship a script - that emits neither — madengine will record the run with no performance value. -- Validate the final `models.json` parses (`python3 -m json.tool models.json`). -- Confirm the new entry is selectable (GPU-free): `madengine discover --tags ` - should list it. This catches tag/name typos before any GPU run. -- Do not run `madengine run` (it needs GPUs). State the verification command the - user should run on a GPU host: `madengine run --tags --live-output`. -- Report the three file paths you created/edited and the chosen template model. +Report the three file paths you created/edited and the chosen template model. diff --git a/.claude/agents/mad-perf-analyst.md b/.claude/agents/mad-perf-analyst.md index c01932a..921a375 100644 --- a/.claude/agents/mad-perf-analyst.md +++ b/.claude/agents/mad-perf-analyst.md @@ -5,25 +5,23 @@ tools: Read, Grep, Glob, Bash model: inherit --- -You analyze MAD performance results. You are READ-ONLY — never edit files or run -workloads. +You analyze MAD performance results and statically validate model definitions. +You are READ-ONLY — never edit files or run workloads. -When invoked: -1. Locate result files: `perf.csv`, `perf_entry.csv`, `perf_entry_super.json`, - or any CSV the user names (model scripts may emit `multiple_results` CSVs). -2. Parse the data. `perf.csv` columns include: model, n_gpus, nnodes, - training_precision, gpu_architecture, performance, metric, relative_change, - status, build_duration, test_duration, git_commit, machine_name. -3. Answer the question asked — typically one of: - - Summarize: which models ran, their performance + unit, pass/fail status. - - Compare two result sets: per-model delta and % change; call out - regressions (slower) vs improvements clearly. - - Diagnose failures: surface `status != SUCCESS` rows and any error context. +This is the fork target for the `mad-report` and `mad-validate` skills, which +supply the concrete task. Your job is to apply the rules below correctly. + +`perf.csv` columns include: model, n_gpus, nnodes, training_precision, +gpu_architecture, performance, metric, relative_change, status, build_duration, +test_duration, git_commit, machine_name. Other result sources: `perf_entry.csv`, +`perf_entry_super.json`, and any `multiple_results` CSV a model emits. Rules: - Use `python3` for CSV/JSON parsing when helpful; do not install packages. - Be precise about units — never compare across different `metric` values. -- A higher number is usually better for throughput (tokens/s, samples/s) and - worse for latency (ms, s); state which direction you assumed. +- A higher number is usually better for throughput (tokens/s, samples/s) and worse + for latency (ms, s); state which direction you assumed. - Present findings as a compact table or bullet list. Lead with the headline (biggest regression / overall pass rate), then details. +- For validation tasks, run the bundled checker the skill points you at and report + errors (run-breaking) separately from warnings (convention metadata). diff --git a/.claude/agents/mad-tuner.md b/.claude/agents/mad-tuner.md index 32d0d18..f96f327 100644 --- a/.claude/agents/mad-tuner.md +++ b/.claude/agents/mad-tuner.md @@ -8,43 +8,25 @@ model: inherit You tune an existing MAD model for better performance using a disciplined measure-change-measure loop. -When invoked: -1. Establish the baseline. Read the model's `scripts/.../run.sh` and any config - it references (YAML/JSON), plus its current `perf.csv` row if present. Record - the baseline performance + unit. -2. Identify tuning levers for the stack, e.g.: - - Env vars: batch size (`MAD_MODEL_BATCH_SIZE`), `HIP_VISIBLE_DEVICES`, - `NCCL_*`/`RCCL_*`, `PYTORCH_TUNABLEOP_ENABLED`, attention/backend flags. - - Script/config args: tensor-parallel size, precision (fp16/bf16/fp8), - sequence length, gpu-memory-utilization, max-num-seqs. -3. Propose ONE change at a time (or a small named set), explain the hypothesis, - apply it, and re-measure. -4. Keep a change only if it improves the metric without breaking the run - (`status == SUCCESS`). Revert regressions. +This is the fork target for the `mad-tune` skill, which supplies the concrete task +and pre-flight. Your job is to apply the discipline below correctly. -Pre-flight — before any `madengine` invocation, run this Bash block: -```bash -if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - echo "[pre-flight] madengine not found. Installing from requirements.txt..." - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found and requirements.txt is missing." - echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" - echo " Or clone MAD and run from its root (which has requirements.txt)." - exit 1 - fi -fi -if [ ! -f models.json ]; then - echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." -fi -``` +Method: +- Establish a baseline first: read the model's `scripts/.../run.sh` and any config + it references, plus its current `perf.csv` row. Record baseline perf + unit. +- Tuning levers by stack: env vars (`MAD_MODEL_BATCH_SIZE`, + `PYTORCH_TUNABLEOP_ENABLED`, `NCCL_*`/`RCCL_*`, attention/backend flags) and args + (tensor-parallel size, precision, sequence length, gpu-memory-utilization, + max-num-seqs). +- Change ONE variable per measurement so deltas are attributable. Keep a change only + if it improves the metric without breaking the run (`status == SUCCESS`); revert + regressions. -Rules: -- Measuring requires AMD GPUs. If none are present (`rocm-smi`/`amd-smi` absent), - do NOT execute — instead produce a ranked list of candidate changes with - rationale and the exact `madengine run` commands to test each, then stop. -- Change one variable per measurement so deltas are attributable. +Execution policy: +- Measuring requires AMD GPUs. If none are present (`rocm-smi`/`amd-smi` absent), do + NOT execute — produce a ranked list of candidate changes with rationale and the + exact `madengine run` command to test each, then stop. - Never alter the `performance: ` output contract. -- Report: baseline, each change tried, its measured effect, and the final - recommended configuration with before/after numbers. + +Report: baseline, each change tried with its measured effect, and the final +recommended configuration with before/after numbers. diff --git a/.claude/commands/mad-add-model.md b/.claude/commands/mad-add-model.md deleted file mode 100644 index f5acb64..0000000 --- a/.claude/commands/mad-add-model.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -description: Scaffold a new MAD model (models.json entry + Dockerfile + run.sh with the performance line) -argument-hint: [base notes / repo url] ---- - -Add a new model to MAD named `$1`. Extra context: $ARGUMENTS - -Use the `mad-model-author` subagent. It should: -0. Pre-flight: check madengine is installed and cwd is the MAD repo root. - ```bash - if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - echo "[pre-flight] madengine not found. Installing from requirements.txt..." - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found and requirements.txt is missing." - echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" - echo " Or clone MAD and run from its root (which has requirements.txt)." - exit 1 - fi - fi - if [ ! -f models.json ]; then - echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." - fi - ``` -1. Pick the closest existing model of the same framework as a template. -2. Create the `models.json` entry, `docker/$1.ubuntu.amd.Dockerfile` (with the - `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}` header), and - `scripts//run.sh` ending in `echo "performance: $performance "`. -3. Validate `models.json` with `python3 -m json.tool models.json`. -4. Confirm the entry is selectable with `madengine discover --tags $1` (GPU-free). - -Report the files created and the verification command -`madengine run --tags $1 --live-output` (requires a GPU host). diff --git a/.claude/commands/mad-benchmark.md b/.claude/commands/mad-benchmark.md deleted file mode 100644 index 39f5354..0000000 --- a/.claude/commands/mad-benchmark.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -description: Build and run a madengine benchmark for the given tag/model -argument-hint: [extra options] ---- - -Benchmark `$ARGUMENTS` with madengine. - -Use the `mad-benchmark-runner` subagent. It should: -0. Pre-flight: check madengine is installed and cwd is the MAD repo root. - ```bash - if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - echo "[pre-flight] madengine not found. Installing from requirements.txt..." - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found and requirements.txt is missing." - echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" - echo " Or clone MAD and run from its root (which has requirements.txt)." - exit 1 - fi - fi - if [ ! -f models.json ]; then - echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." - fi - ``` -1. Confirm the tag matches real models via `madengine discover --tags $1`. -2. Assemble the `madengine run --tags $1 --live-output` command (add `-o`, - `--timeout`, profiling `--additional-context`, or `slurm`/`k8s` keys as the - request implies), and list required env vars (e.g. `MAD_SECRETS_HFTOKEN`). -3. Check for GPUs first. If none, print the exact command(s) to run on a GPU - host instead of executing. If GPUs exist, run it and report where `perf.csv` - landed. diff --git a/.claude/commands/mad-profile.md b/.claude/commands/mad-profile.md deleted file mode 100644 index 9a45b1d..0000000 --- a/.claude/commands/mad-profile.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -description: Run a madengine benchmark with a profiling/tracing tool attached -argument-hint: [tool: rocprofv3_compute|rpd|rccl_trace|...] ---- - -Profile `$ARGUMENTS`. - -Use the `mad-benchmark-runner` subagent with profiling enabled. It should: -0. Pre-flight: check madengine is installed and cwd is the MAD repo root. - ```bash - if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - echo "[pre-flight] madengine not found. Installing from requirements.txt..." - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found and requirements.txt is missing." - echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" - echo " Or clone MAD and run from its root (which has requirements.txt)." - exit 1 - fi - fi - if [ ! -f models.json ]; then - echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." - fi - ``` -1. Pick the profiling tool (default `rocprofv3_compute` if unspecified). Common - names: `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`, - `rocprofv3_communication`, `rocm_trace_lite`, `rccl_trace`, - `gpu_info_power_profiler`. The complete, authoritative list (23+ tools) lives - in the madengine package at `scripts/common/tools.json` — consult it for - names like `rocprofv3_full`, `rocblas_trace`, `hipblaslt_trace`, `rocprof_sys`. -2. Build: - `madengine run --tags $1 --live-output --additional-context '{"tools": [{"name": ""}]}'` -3. Check for GPUs; if none, print the command for a GPU host. Otherwise run it - and report where the trace/profile output and `perf.csv` landed. - -Note: profiling adds overhead; the perf number under profiling is not a clean -benchmark number. diff --git a/.claude/commands/mad-report.md b/.claude/commands/mad-report.md deleted file mode 100644 index 5bbb74b..0000000 --- a/.claude/commands/mad-report.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -description: Analyze and summarize MAD benchmark results (perf.csv / perf_entry_super.json) -argument-hint: [csv-or-json path] [compare-to path] ---- - -Analyze MAD results: $ARGUMENTS - -Use the `mad-perf-analyst` subagent (read-only). It should: -1. Load the result file(s). Default to `perf.csv` if no path is given. -2. If two paths are provided, compare them: per-model delta and % change, - flagging regressions vs improvements (respecting each row's `metric`/unit). -3. Otherwise summarize: models run, performance + unit, and pass/fail status. - -Lead with the headline finding, then a compact table. To generate the HTML -dashboard instead, run `madengine report to-html --csv-file-path ` -(verify flags with `madengine report to-html --help`). diff --git a/.claude/commands/mad-tune.md b/.claude/commands/mad-tune.md deleted file mode 100644 index 11ed8aa..0000000 --- a/.claude/commands/mad-tune.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -description: Tune a MAD model for better performance with a measure-change-measure loop -argument-hint: [target: throughput|latency] [lever hints] ---- - -Tune `$ARGUMENTS`. - -Use the `mad-tuner` subagent. It should: -0. Pre-flight: check madengine is installed and cwd is the MAD repo root. - ```bash - if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - echo "[pre-flight] madengine not found. Installing from requirements.txt..." - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found and requirements.txt is missing." - echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" - echo " Or clone MAD and run from its root (which has requirements.txt)." - exit 1 - fi - fi - if [ ! -f models.json ]; then - echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." - fi - ``` -1. Establish the baseline (current `run.sh`/config + `perf.csv` row). -2. Propose tuning levers (env vars like `MAD_MODEL_BATCH_SIZE`, - `PYTORCH_TUNABLEOP_ENABLED`, `NCCL_*`/`RCCL_*`; or args like tensor-parallel - size, precision, gpu-memory-utilization), changing ONE at a time. -3. Re-measure each change; keep improvements, revert regressions. - -If no GPUs are present, produce a ranked list of candidate changes with rationale -and the exact `madengine run` command to test each, then stop. Report baseline, -changes tried with measured effect, and the final recommended config with -before/after numbers. diff --git a/.claude/commands/mad-validate.md b/.claude/commands/mad-validate.md deleted file mode 100644 index 7b6fc52..0000000 --- a/.claude/commands/mad-validate.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -description: Statically validate MAD model entries (no GPU) — JSON, paths, Dockerfile header, output contract -argument-hint: [tag-or-model | all] ---- - -Statically validate MAD model definitions for `${ARGUMENTS:-all}`. This is a -GPU-free lint of `models.json` and the files it points at — it does NOT build or -run anything. - -Run the checks below (use `python3`; do not install packages). Scope to the -named tag/model if one is given, otherwise check every entry. - -Two severities. **Errors** break a run; **warnings** are missing MAD-convention -metadata (madengine itself defaults these — only `name` is structurally -required, per its `Model` dataclass — so don't fail the build on them). - -Errors: -1. **JSON parses**: `python3 -m json.tool models.json` succeeds. -2. **Has `name`** and **names are unique** (no two entries share one). -3. **Dockerfile exists**: `.ubuntu.amd.Dockerfile` is a real file, - and its first line is the context header - `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}`. -4. **Script exists**: the `scripts` path exists (a `run.sh` file or a dir). -5. **Output contract** — exactly one of: - - the run script contains a line that echoes `performance: ...`, OR - - the entry sets `multiple_results` to a CSV filename. - Flag any entry that satisfies neither (madengine would record no perf value). - -Warnings (convention metadata, non-fatal): -6. Missing any of `url`, `owner`, `training_precision`, `tags`, `n_gpus`. - -Suggested one-shot checker: - -```bash -python3 - "${ARGUMENTS:-all}" <<'PY' -import json, os, sys, glob -sel = sys.argv[1] if len(sys.argv) > 1 else "all" -models = json.load(open("models.json")) -def selected(m): - return sel in ("all","") or sel == m.get("name") or sel in (m.get("tags") or []) -seen, errors, warns = {}, [], [] -for m in models: - n = m.get("name","") - if not selected(m): - continue - # --- errors (break a run) --- - if "name" not in m: errors.append(f"{n}: missing 'name'") - if n in seen: errors.append(f"{n}: duplicate name") - seen[n] = True - df = (m.get("dockerfile","") or "") + ".ubuntu.amd.Dockerfile" - if not m.get("dockerfile"): - errors.append(f"{n}: no dockerfile field") - elif not os.path.isfile(df): - errors.append(f"{n}: dockerfile not found: {df}") - else: - first = open(df).readline().strip() - if not first.startswith("# CONTEXT") or "AMD" not in first: - errors.append(f"{n}: dockerfile missing CONTEXT header: {df}") - sp = m.get("scripts","") - if not sp: - errors.append(f"{n}: no scripts field") - elif not os.path.exists(sp): - errors.append(f"{n}: scripts path not found: {sp}") - has_mr = bool(m.get("multiple_results")) - emits = False - if sp: - sh_files = [sp] if sp.endswith(".sh") else glob.glob(os.path.join(sp,"**","*.sh"), recursive=True) - for sh in sh_files: - if os.path.isfile(sh) and "performance:" in open(sh, errors="ignore").read(): - emits = True; break - if not (has_mr or emits): - errors.append(f"{n}: no output contract (no 'performance:' line and no multiple_results)") - # --- warnings (convention metadata) --- - for f in ("url","owner","training_precision","tags","n_gpus"): - if not m.get(f): - warns.append(f"{n}: missing convention field '{f}'") -checked = sum(1 for m in models if selected(m)) -print(f"Checked {checked} model(s). {len(errors)} error(s), {len(warns)} warning(s).") -if errors: - print("\nERRORS:") - for e in errors: print(" -", e) -if warns: - print("\nwarnings:") - for w in warns[:40]: print(" -", w) - if len(warns) > 40: print(f" ... and {len(warns)-40} more") -sys.exit(1 if errors else 0) -PY -``` - -Report a compact pass/fail summary. If anything fails, list each problem with the -model name and how to fix it. This is the right check to run after `/mad-add-model` -and before any GPU run. diff --git a/.claude/skills/mad-add-model/SKILL.md b/.claude/skills/mad-add-model/SKILL.md new file mode 100644 index 0000000..6f9cb2a --- /dev/null +++ b/.claude/skills/mad-add-model/SKILL.md @@ -0,0 +1,40 @@ +--- +name: mad-add-model +description: Scaffold a new MAD model (models.json entry + Dockerfile + run.sh with the performance line). Use when the user wants to add a new model or workload to MAD. +argument-hint: [base notes / repo url] +disable-model-invocation: true +context: fork +agent: mad-model-author +allowed-tools: Bash(madengine discover *) Bash(python3 -m json.tool *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Write Edit Grep Glob +--- + +Add a new model to MAD named `$0`. Extra context: $ARGUMENTS + +## Pre-flight +```! +bash .claude/skills/mad-common/preflight.sh +``` + +## Task +1. Pick the CLOSEST existing model of the same framework (vLLM, PyTorch/Primus, JAX + MaxText, xDiT, SGLang, Megatron, HuggingFace, ...) as a template — read its + `models.json` entry, its `docker/.ubuntu.amd.Dockerfile`, and its + `scripts/.../run.sh`. Reuse those patterns; do not invent new ones. +2. Produce three artifacts: + a. A `models.json` entry. Required: `name`, `url`, `dockerfile`, `scripts`, + `n_gpus`, `owner`, `training_precision`, `tags`. Name = `{framework}_{project}_{workload}`. + Keep `models.json` valid JSON. + b. `docker/$0.ubuntu.amd.Dockerfile` whose first line is exactly + `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}`. Prefer pointing the + `dockerfile` field at an existing same-stack Dockerfile over a near-duplicate. + c. `scripts//run.sh` satisfying ONE output contract: + - single result: end with `echo "performance: $performance "`, parsed + from the workload's log output, OR + - multiple results: have the script WRITE its own CSV and set + `"multiple_results": ".csv"` in the entry. Never mix the two. +3. Validate: `python3 -m json.tool models.json` parses. +4. Confirm selectable (GPU-free): `madengine discover --tags $0` lists it. + +Do NOT run `madengine run` (needs GPUs). Report the three file paths created and the +chosen template, then state the verification command for a GPU host: +`madengine run --tags $0 --live-output`. After this, run `/mad-validate $0`. diff --git a/.claude/skills/mad-benchmark/SKILL.md b/.claude/skills/mad-benchmark/SKILL.md new file mode 100644 index 0000000..ab5bdaf --- /dev/null +++ b/.claude/skills/mad-benchmark/SKILL.md @@ -0,0 +1,33 @@ +--- +name: mad-benchmark +description: Build and run a madengine benchmark for a MAD model/tag. Use when the user wants to benchmark, run, or measure a model on AMD GPUs. +argument-hint: [extra options] +disable-model-invocation: true +context: fork +agent: mad-benchmark-runner +allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Grep Glob +--- + +Benchmark `$ARGUMENTS` with madengine. + +## Pre-flight +```! +bash .claude/skills/mad-common/preflight.sh +``` + +## Task +1. Confirm the tag matches real models via `madengine discover --tags $0` + (read-only, no GPU). If the intent is fuzzy, grep `models.json` for candidates. +2. Assemble `madengine run --tags $0 --live-output`. Add as the request implies: + - `-o ` to keep results separately, + - `--timeout ` for long training runs (default 7200), + - profiling/deploy via `--additional-context` (a `"slurm"`/`"k8s"` key selects + the target; neither → local Docker). + - Build-once/run-many split: `madengine build --tags $0 [-r REGISTRY]` writes + `build_manifest.json`, then `madengine run -m build_manifest.json` executes it. +3. List required env vars (e.g. `export MAD_SECRETS_HFTOKEN=...` for HF models). +4. Check for AMD GPUs (`rocm-smi`/`amd-smi`). If none are present, DO NOT run — + print the exact command(s) to run on a GPU host and stop. If GPUs exist, run it. + +Report: resolved model list, the exact command, required env vars, and where +results landed (`perf.csv` by default). diff --git a/.claude/skills/mad-common/preflight.sh b/.claude/skills/mad-common/preflight.sh new file mode 100755 index 0000000..f1ece7b --- /dev/null +++ b/.claude/skills/mad-common/preflight.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Shared MAD pre-flight: ensure madengine is installed and we're at the repo root. +# Referenced by every GPU-touching mad-* skill via dynamic context injection so +# the check is inlined into the skill prompt before the work begins. +set -u + +if ! command -v madengine &>/dev/null; then + if [ -f requirements.txt ] && grep -q madengine requirements.txt; then + echo "[pre-flight] madengine not found. Installing from requirements.txt..." + pip install -r requirements.txt + else + echo "[pre-flight] madengine not found and requirements.txt is missing." + echo " Install: pip install git+https://github.com/ROCm/madengine.git@main" + echo " Or clone MAD and run from its root (which has requirements.txt)." + exit 1 + fi +fi + +if [ ! -f models.json ]; then + echo "[pre-flight] Warning: models.json not found — run from the MAD repo root." +fi + +echo "[pre-flight] OK: madengine=$(command -v madengine), cwd=$(pwd)" diff --git a/.claude/skills/mad-profile/SKILL.md b/.claude/skills/mad-profile/SKILL.md new file mode 100644 index 0000000..f89dca8 --- /dev/null +++ b/.claude/skills/mad-profile/SKILL.md @@ -0,0 +1,32 @@ +--- +name: mad-profile +description: Run a madengine benchmark with a profiling/tracing tool attached (rocprofv3, rpd, rccl_trace, ...). Use when the user wants to profile or trace a MAD model. +argument-hint: " [tool: rocprofv3_compute|rpd|rccl_trace|...]" +disable-model-invocation: true +context: fork +agent: mad-benchmark-runner +allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Grep Glob +--- + +Profile `$ARGUMENTS`. + +## Pre-flight +```! +bash .claude/skills/mad-common/preflight.sh +``` + +## Task +1. Pick the profiling tool (default `rocprofv3_compute` if unspecified). Common + names: `rpd`, `rocprofv3`, `rocprofv3_compute`, `rocprofv3_memory`, + `rocprofv3_communication`, `rocm_trace_lite`, `rccl_trace`, + `gpu_info_power_profiler`. The authoritative full list (23+ tools, incl. + `rocprofv3_full`, `rocblas_trace`, `hipblaslt_trace`, `miopen_trace`, + `rocprof_sys`) lives in the madengine package at `scripts/common/tools.json`. +2. Build: + `madengine run --tags $0 --live-output --additional-context '{"tools": [{"name": ""}]}'` +3. Check for AMD GPUs (`rocm-smi`/`amd-smi`). If none, print the command for a GPU + host and stop. Otherwise run it and report where the trace output and `perf.csv` + landed. + +Note: profiling adds overhead — the perf number under profiling is NOT a clean +benchmark number. diff --git a/.claude/skills/mad-report/SKILL.md b/.claude/skills/mad-report/SKILL.md new file mode 100644 index 0000000..e98e1f5 --- /dev/null +++ b/.claude/skills/mad-report/SKILL.md @@ -0,0 +1,25 @@ +--- +name: mad-report +description: Analyze and summarize MAD benchmark results (perf.csv / perf_entry_super.json) — summarize a run, compare two runs, or flag regressions. Use when the user asks about benchmark results, performance numbers, or comparing runs. +argument-hint: "[csv-or-json path] [compare-to path]" +context: fork +agent: mad-perf-analyst +allowed-tools: Read Grep Glob Bash(python3 *) Bash(madengine report *) +--- + +Analyze MAD results: $ARGUMENTS + +## Task +1. Load the result file(s). Default to `perf.csv` if no path is given. Other sources: + `perf_entry.csv`, `perf_entry_super.json`, or any `multiple_results` CSV named. +2. If two paths are provided, compare them: per-model delta and % change, flagging + regressions vs improvements (respect each row's `metric`/unit — never compare + across different units). +3. Otherwise summarize: models run, performance + unit, and pass/fail status. + +A higher number is usually better for throughput (tokens/s, samples/s) and worse for +latency (ms, s) — state which direction you assumed. Lead with the headline (biggest +regression / overall pass rate), then a compact table. + +To generate the HTML dashboard instead: +`madengine report to-html --csv-file-path ` (verify flags with `--help`). diff --git a/.claude/skills/mad-tune/SKILL.md b/.claude/skills/mad-tune/SKILL.md new file mode 100644 index 0000000..963f8f7 --- /dev/null +++ b/.claude/skills/mad-tune/SKILL.md @@ -0,0 +1,37 @@ +--- +name: mad-tune +description: Tune a MAD model for better performance with a measure-change-measure loop. Use when the user wants to optimize, tune, or speed up an existing MAD model. +argument-hint: " [target: throughput|latency] [lever hints]" +disable-model-invocation: true +context: fork +agent: mad-tuner +allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Edit Grep Glob +--- + +Tune `$ARGUMENTS`. + +## Pre-flight +```! +bash .claude/skills/mad-common/preflight.sh +``` + +## Task +1. Establish the baseline: read the model's `scripts/.../run.sh` and any config it + references, plus its current `perf.csv` row if present. Record baseline perf + unit. +2. Propose tuning levers, changing ONE at a time so deltas are attributable: + - env vars: `MAD_MODEL_BATCH_SIZE`, `PYTORCH_TUNABLEOP_ENABLED`, `NCCL_*`/`RCCL_*`, + attention/backend flags; + - args: tensor-parallel size, precision (fp16/bf16/fp8), sequence length, + gpu-memory-utilization, max-num-seqs. +3. Re-measure each change. Keep improvements (`status == SUCCESS`); revert regressions. + +If no AMD GPUs are present (`rocm-smi`/`amd-smi` absent), do NOT execute — produce a +ranked list of candidate changes with rationale and the exact `madengine run` command +to test each, then stop. + +Never alter the `performance: ` output contract. Report: baseline, each +change tried with its measured effect, and the final recommended config with +before/after numbers. + +For a deeper profiling-informed search across many candidates with adversarial +verification, use the `mad-tune-search` workflow instead. diff --git a/.claude/skills/mad-validate/SKILL.md b/.claude/skills/mad-validate/SKILL.md new file mode 100644 index 0000000..82b51db --- /dev/null +++ b/.claude/skills/mad-validate/SKILL.md @@ -0,0 +1,33 @@ +--- +name: mad-validate +description: Statically validate MAD model entries (no GPU) — JSON, paths, Dockerfile header, output contract. Use after adding/editing a model or before a GPU run to lint models.json. +argument-hint: [tag-or-model | all] +context: fork +agent: mad-perf-analyst +allowed-tools: Read Grep Glob Bash(python3 *) +--- + +Statically validate MAD model definitions for `$ARGUMENTS` (default: all). This is a +GPU-free lint of `models.json` and the files it points at — it does NOT build or run +anything. + +## Task +Run the bundled checker (uses only the stdlib; do not install packages): + +```! +python3 ${CLAUDE_SKILL_DIR}/scripts/validate.py "$ARGUMENTS" +``` + +(An empty argument is treated as `all`.) + +The checker reports two severities: +- **Errors** (non-zero exit) break a run: invalid JSON, missing/duplicate `name`, + missing Dockerfile or missing `# CONTEXT ... AMD` header, missing scripts path, or + no output contract (neither a `performance:` line nor `multiple_results`). +- **Warnings** are missing MAD-convention metadata (`url`, `owner`, + `training_precision`, `tags`, `n_gpus`) — madengine defaults these, so they do not + fail the build. + +Report a compact pass/fail summary. If anything fails, list each problem with the +model name and how to fix it. This is the right check to run after `/mad-add-model` +and before any GPU run. diff --git a/.claude/skills/mad-validate/scripts/validate.py b/.claude/skills/mad-validate/scripts/validate.py new file mode 100644 index 0000000..b92e382 --- /dev/null +++ b/.claude/skills/mad-validate/scripts/validate.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +"""GPU-free static validation of MAD model definitions. + +Lints models.json and the files each entry points at. Does NOT build or run +anything. Usage: python3 validate.py [tag-or-model | all] + +Errors break a run (non-zero exit). Warnings are missing MAD-convention +metadata (madengine itself defaults these — only `name` is structurally +required per its Model dataclass — so they do not fail the build). +""" +import json, os, sys, glob + +sel = sys.argv[1] if len(sys.argv) > 1 else "all" +models = json.load(open("models.json")) + + +def selected(m): + return sel in ("all", "") or sel == m.get("name") or sel in (m.get("tags") or []) + + +seen, errors, warns = {}, [], [] +for m in models: + n = m.get("name", "") + if not selected(m): + continue + # --- errors (break a run) --- + if "name" not in m: + errors.append(f"{n}: missing 'name'") + if n in seen: + errors.append(f"{n}: duplicate name") + seen[n] = True + df = (m.get("dockerfile", "") or "") + ".ubuntu.amd.Dockerfile" + if not m.get("dockerfile"): + errors.append(f"{n}: no dockerfile field") + elif not os.path.isfile(df): + errors.append(f"{n}: dockerfile not found: {df}") + else: + first = open(df).readline().strip() + if not first.startswith("# CONTEXT") or "AMD" not in first: + errors.append(f"{n}: dockerfile missing CONTEXT header: {df}") + sp = m.get("scripts", "") + if not sp: + errors.append(f"{n}: no scripts field") + elif not os.path.exists(sp): + errors.append(f"{n}: scripts path not found: {sp}") + has_mr = bool(m.get("multiple_results")) + emits = False + if sp: + sh_files = [sp] if sp.endswith(".sh") else glob.glob(os.path.join(sp, "**", "*.sh"), recursive=True) + for sh in sh_files: + if os.path.isfile(sh) and "performance:" in open(sh, errors="ignore").read(): + emits = True + break + if not (has_mr or emits): + errors.append(f"{n}: no output contract (no 'performance:' line and no multiple_results)") + # --- warnings (convention metadata) --- + for f in ("url", "owner", "training_precision", "tags", "n_gpus"): + if not m.get(f): + warns.append(f"{n}: missing convention field '{f}'") + +checked = sum(1 for m in models if selected(m)) +print(f"Checked {checked} model(s). {len(errors)} error(s), {len(warns)} warning(s).") +if errors: + print("\nERRORS:") + for e in errors: + print(" -", e) +if warns: + print("\nwarnings:") + for w in warns[:40]: + print(" -", w) + if len(warns) > 40: + print(f" ... and {len(warns) - 40} more") +sys.exit(1 if errors else 0) diff --git a/CLAUDE.md b/CLAUDE.md index 0c5648f..8acca46 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,8 +12,12 @@ and appends a row to `perf.csv`. The four common user tasks here are: **benchmarking**, **adding a new model**, **tuning** a model/kernel for better perf, and **development** on the repo -itself. Dedicated subagents and `/mad-*` slash commands exist for each — see -`.claude/agents/` and `.claude/commands/`. +itself. Each is a `/mad-*` **skill** (`.claude/skills/`) that forks to a curated +subagent (`.claude/agents/`); larger fan-out jobs are **workflows** +(`.claude/workflows/`). Skills: `/mad-benchmark`, `/mad-profile`, `/mad-tune`, +`/mad-add-model` (manual-invoke; they build/run on GPUs), and `/mad-report`, +`/mad-validate` (read-only, auto-invocable). Shared madengine pre-flight lives in +`.claude/skills/mad-common/preflight.sh`. ## The performance contract (most important convention) From 4ffdc8269aab5325634c8c0caf13c43ce90d8b3a Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Mon, 8 Jun 2026 22:06:13 -0500 Subject: [PATCH 12/13] Fix 8 issues found in post-migration review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Preflight path: replace relative `bash .claude/skills/mad-common/preflight.sh` with `bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh` in the 4 GPU skills so the injection works regardless of cwd (not just when at the repo root). Update allowed-tools from the specific path to `Bash(bash *)` to match. 2. $0 truncation: replace `$0` with `$ARGUMENTS` throughout mad-add-model — $0 only expands to the first whitespace-delimited token, silently truncating model names if the user passes extra unquoted context words. 3. /mad-validate in forked agent: replace the `/mad-validate $ARGUMENTS` slash- command call (which doesn't work inside a forked subagent) with an explicit `python3 .claude/skills/mad-validate/scripts/validate.py "$ARGUMENTS"` invocation. 4. validate.py cwd: add git rev-parse + os.chdir(repo_root) so validate.py resolves models.json correctly even when called from a subdirectory. 5. settings.json: add pre-approvals for `preflight.sh` and `validate.py` so a fresh clone doesn't hit permission prompts on first use. 6. Workflow preflight dedup: replace the inline madengine-install shell block in mad-benchmark-sweep.js and mad-tune-search.js with a reference to preflight.sh. Also add the missing preflight check to the Diagnose and Evaluate phases in mad-tune-search.js (previously only the Baseline phase had it). 7. Hardcoded path: remove `/home/ysha/MAD` absolute path from mad-benchmark-sweep.js. Co-Authored-By: Claude Opus 4 --- .claude/settings.json | 2 ++ .claude/skills/mad-add-model/SKILL.md | 17 +++++++++++------ .claude/skills/mad-benchmark/SKILL.md | 4 ++-- .claude/skills/mad-profile/SKILL.md | 4 ++-- .claude/skills/mad-tune/SKILL.md | 4 ++-- .claude/skills/mad-validate/scripts/validate.py | 12 +++++++++++- .claude/workflows/mad-benchmark-sweep.js | 12 ++---------- .claude/workflows/mad-tune-search.js | 12 +++--------- 8 files changed, 35 insertions(+), 32 deletions(-) diff --git a/.claude/settings.json b/.claude/settings.json index b4bbaa2..0251366 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -10,6 +10,8 @@ "Bash(git log *)", "Bash(rocm-smi *)", "Bash(amd-smi *)", + "Bash(bash .claude/skills/mad-common/preflight.sh)", + "Bash(python3 .claude/skills/mad-validate/scripts/validate.py *)", "Read", "Grep", "Glob" diff --git a/.claude/skills/mad-add-model/SKILL.md b/.claude/skills/mad-add-model/SKILL.md index 6f9cb2a..7d4e323 100644 --- a/.claude/skills/mad-add-model/SKILL.md +++ b/.claude/skills/mad-add-model/SKILL.md @@ -5,17 +5,20 @@ argument-hint: [base notes / repo url] disable-model-invocation: true context: fork agent: mad-model-author -allowed-tools: Bash(madengine discover *) Bash(python3 -m json.tool *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Write Edit Grep Glob +allowed-tools: Bash(madengine discover *) Bash(python3 -m json.tool *) Bash(python3 *) Bash(bash *) Read Write Edit Grep Glob --- -Add a new model to MAD named `$0`. Extra context: $ARGUMENTS +Add a new model to MAD named `$ARGUMENTS`. Extra context: $ARGUMENTS ## Pre-flight ```! -bash .claude/skills/mad-common/preflight.sh +bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh ``` ## Task +The model name is the first token of `$ARGUMENTS` (e.g. `pyt_vllm_qwen3-8b`). Use +`$ARGUMENTS` where the full model name is needed — do not split on spaces. + 1. Pick the CLOSEST existing model of the same framework (vLLM, PyTorch/Primus, JAX MaxText, xDiT, SGLang, Megatron, HuggingFace, ...) as a template — read its `models.json` entry, its `docker/.ubuntu.amd.Dockerfile`, and its @@ -24,7 +27,7 @@ bash .claude/skills/mad-common/preflight.sh a. A `models.json` entry. Required: `name`, `url`, `dockerfile`, `scripts`, `n_gpus`, `owner`, `training_precision`, `tags`. Name = `{framework}_{project}_{workload}`. Keep `models.json` valid JSON. - b. `docker/$0.ubuntu.amd.Dockerfile` whose first line is exactly + b. `docker/$ARGUMENTS.ubuntu.amd.Dockerfile` whose first line is exactly `# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}`. Prefer pointing the `dockerfile` field at an existing same-stack Dockerfile over a near-duplicate. c. `scripts//run.sh` satisfying ONE output contract: @@ -33,8 +36,10 @@ bash .claude/skills/mad-common/preflight.sh - multiple results: have the script WRITE its own CSV and set `"multiple_results": ".csv"` in the entry. Never mix the two. 3. Validate: `python3 -m json.tool models.json` parses. -4. Confirm selectable (GPU-free): `madengine discover --tags $0` lists it. +4. Confirm selectable (GPU-free): `madengine discover --tags $ARGUMENTS` lists it. +5. Validate statically (GPU-free lint): + `python3 .claude/skills/mad-validate/scripts/validate.py "$ARGUMENTS"` Do NOT run `madengine run` (needs GPUs). Report the three file paths created and the chosen template, then state the verification command for a GPU host: -`madengine run --tags $0 --live-output`. After this, run `/mad-validate $0`. +`madengine run --tags $ARGUMENTS --live-output`. diff --git a/.claude/skills/mad-benchmark/SKILL.md b/.claude/skills/mad-benchmark/SKILL.md index ab5bdaf..19432c5 100644 --- a/.claude/skills/mad-benchmark/SKILL.md +++ b/.claude/skills/mad-benchmark/SKILL.md @@ -5,14 +5,14 @@ argument-hint: [extra options] disable-model-invocation: true context: fork agent: mad-benchmark-runner -allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Grep Glob +allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash *) Read Grep Glob --- Benchmark `$ARGUMENTS` with madengine. ## Pre-flight ```! -bash .claude/skills/mad-common/preflight.sh +bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh ``` ## Task diff --git a/.claude/skills/mad-profile/SKILL.md b/.claude/skills/mad-profile/SKILL.md index f89dca8..d29b709 100644 --- a/.claude/skills/mad-profile/SKILL.md +++ b/.claude/skills/mad-profile/SKILL.md @@ -5,14 +5,14 @@ argument-hint: " [tool: rocprofv3_compute|rpd|rccl_trace|...]" disable-model-invocation: true context: fork agent: mad-benchmark-runner -allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Grep Glob +allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash *) Read Grep Glob --- Profile `$ARGUMENTS`. ## Pre-flight ```! -bash .claude/skills/mad-common/preflight.sh +bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh ``` ## Task diff --git a/.claude/skills/mad-tune/SKILL.md b/.claude/skills/mad-tune/SKILL.md index 963f8f7..4a64f9c 100644 --- a/.claude/skills/mad-tune/SKILL.md +++ b/.claude/skills/mad-tune/SKILL.md @@ -5,14 +5,14 @@ argument-hint: " [target: throughput|latency] [lever hints]" disable-model-invocation: true context: fork agent: mad-tuner -allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash .claude/skills/mad-common/preflight.sh) Read Edit Grep Glob +allowed-tools: Bash(madengine *) Bash(rocm-smi *) Bash(amd-smi *) Bash(bash *) Read Edit Grep Glob --- Tune `$ARGUMENTS`. ## Pre-flight ```! -bash .claude/skills/mad-common/preflight.sh +bash ${CLAUDE_SKILL_DIR}/../mad-common/preflight.sh ``` ## Task diff --git a/.claude/skills/mad-validate/scripts/validate.py b/.claude/skills/mad-validate/scripts/validate.py index b92e382..26ceecc 100644 --- a/.claude/skills/mad-validate/scripts/validate.py +++ b/.claude/skills/mad-validate/scripts/validate.py @@ -8,9 +8,19 @@ metadata (madengine itself defaults these — only `name` is structurally required per its Model dataclass — so they do not fail the build). """ -import json, os, sys, glob +import json, os, subprocess, sys, glob sel = sys.argv[1] if len(sys.argv) > 1 else "all" + +# Resolve the repo root so the script works regardless of cwd. +try: + repo_root = subprocess.check_output( + ["git", "rev-parse", "--show-toplevel"], text=True, stderr=subprocess.DEVNULL + ).strip() + os.chdir(repo_root) +except Exception: + pass # fall back to cwd; works when already at repo root + models = json.load(open("models.json")) diff --git a/.claude/workflows/mad-benchmark-sweep.js b/.claude/workflows/mad-benchmark-sweep.js index 9bb47bf..1f489c8 100644 --- a/.claude/workflows/mad-benchmark-sweep.js +++ b/.claude/workflows/mad-benchmark-sweep.js @@ -148,20 +148,12 @@ const results = await parallel(matrix.map((cell, i) => () => { const cmd = `madengine run ${flags.join(' ')}${ctx}` const action = execute - ? `If AMD GPUs are present (check rocm-smi/amd-smi), RUN this command from the /home/ysha/MAD repo root and parse results. This model may emit a "performance: " stdout line OR (if it is a multiple_results model) write per-metric rows to its own CSV — in that case report the primary throughput metric. Results land in ${outFile}. If no GPUs, set status "skipped".` + ? `If AMD GPUs are present (check rocm-smi/amd-smi), RUN this command from the MAD repo root and parse results. This model may emit a "performance: " stdout line OR (if it is a multiple_results model) write per-metric rows to its own CSV — in that case report the primary throughput metric. Results land in ${outFile}. If no GPUs, set status "skipped".` : `Do NOT execute. Validate the command is well-formed and the tag resolves via "madengine discover --tags ${cell.tag}". Set status "planned".` return agent( `MAD benchmark sweep cell "${label}". -Pre-flight: before running madengine, verify it is installed: - if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found. Install: pip install git+https://github.com/ROCm/madengine.git@main"; exit 1 - fi - fi - [ -f models.json ] || echo "[pre-flight] Warning: not in MAD repo root." +Pre-flight: run \`bash .claude/skills/mad-common/preflight.sh\` and stop if it exits non-zero. Command: ${cmd} ${action} Return the cell label, the exact command, status, and performance/metric if available.`, diff --git a/.claude/workflows/mad-tune-search.js b/.claude/workflows/mad-tune-search.js index 9e1fd52..aaf4cfa 100644 --- a/.claude/workflows/mad-tune-search.js +++ b/.claude/workflows/mad-tune-search.js @@ -161,15 +161,7 @@ const baseAction = execute : `Do NOT run anything. Read the static config only; leave baselinePerf null.` const baseline = await agent( `In the MAD repo, establish the tuning baseline for tag "${tag}" (target: ${target}). -Pre-flight: before running madengine, verify it is installed: - if ! command -v madengine &>/dev/null; then - if [ -f requirements.txt ] && grep -q madengine requirements.txt; then - pip install -r requirements.txt - else - echo "[pre-flight] madengine not found. Install: pip install git+https://github.com/ROCm/madengine.git@main"; exit 1 - fi - fi - [ -f models.json ] || echo "[pre-flight] Warning: not in MAD repo root." +Pre-flight: run \`bash .claude/skills/mad-common/preflight.sh\` and stop if it exits non-zero. Find its models.json entry, its scripts/.../run.sh, and any config it references. Summarize the current configuration and list the tuning levers available for this stack (env vars like MAD_MODEL_BATCH_SIZE / PYTORCH_TUNABLEOP_ENABLED / NCCL_*/RCCL_*, @@ -205,6 +197,7 @@ const diagAction = execute : `Do NOT run anything. Classify the likely bottleneck from the static config, model architecture, and any EXISTING trace output already on disk (e.g. rocm_trace_lite_output/trace_summary.txt). Set tracePath to that file if found, else null.` const diag = await agent( `Diagnose the performance bottleneck of "${baseline.model}" to guide ${target} tuning. +Pre-flight: run \`bash .claude/skills/mad-common/preflight.sh\` and stop if it exits non-zero. Baseline config: ${baseline.baselineSummary} ${diagAction} @@ -298,6 +291,7 @@ for (const cand of candidates) { const evalResult = await agent( `Evaluate tuning candidate ${cand.id} for "${baseline.model}" (leverKind: ${cand.leverKind || 'other'}). +Pre-flight: run \`bash .claude/skills/mad-common/preflight.sh\` and stop if it exits non-zero. Change: ${cand.change} Hypothesis: ${cand.hypothesis} Motivating evidence: ${cand.evidence} From 22d09403f77a8b260c6a3188e0a90272ede8aa7d Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Mon, 8 Jun 2026 22:08:49 -0500 Subject: [PATCH 13/13] Update README.md to reflect skills migration - Add Claude Code Integration section documenting the /mad-* skills, their invocation mode (manual vs auto), and the two workflows - Add /mad-add-model callout in the Contributing > Adding New Models section so contributors know the automated path exists - Fix factual error: timeout -1 (not 0) disables the timeout entirely, matching the madengine models.json schema documented in CLAUDE.md Co-Authored-By: Claude Opus 4 --- README.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2cc0fa2..6cf24f1 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ Below are blueprints of supported models along with their documentation. - [Tag Functionality](#tag-functionality) - [Timeout Configuration](#timeout-configuration) - [Debugging Options](#debugging-options) +- [Claude Code Integration](#claude-code-integration) - [Contributing](#contributing) - [Adding New Models](#adding-new-models) - [Model Configuration](#model-configuration) @@ -135,7 +136,7 @@ Configure execution timeouts at multiple levels: 2. **Model-specific**: Set `timeout` field in `models.json` 3. **Runtime override**: Use `--timeout` command line option -> **Note**: Setting timeout to `0` disables the timeout entirely. +> **Note**: Setting timeout to `-1` disables the timeout entirely. ### Debugging Options @@ -154,10 +155,27 @@ madengine run --tags model_name --clean-docker-cache > ⚠️ **Warning**: When using `--keep-alive`, you must manually stop and remove the container before running the same model again. +## Claude Code Integration + +MAD ships with a set of `/mad-*` skills for [Claude Code](https://claude.ai/code) that cover the four most common tasks. See `CLAUDE.md` for full context and conventions. + +| Skill | Invocation | What it does | +|-------|-----------|--------------| +| `/mad-benchmark ` | Manual | Build and run a madengine benchmark on AMD GPUs | +| `/mad-profile [tool]` | Manual | Run with a profiling/tracing tool attached | +| `/mad-tune ` | Manual | Measure-change-measure tuning loop | +| `/mad-add-model ` | Manual | Scaffold models.json + Dockerfile + run.sh from the nearest template | +| `/mad-report [csv]` | Auto | Summarize or compare benchmark results | +| `/mad-validate [tag]` | Auto | GPU-free lint of models.json and file references | + +Larger fan-out jobs use workflows: `mad-benchmark-sweep` (parallel matrix across tags) and `mad-tune-search` (profile-informed tuning with adversarial verification). + ## Contributing ### Adding New Models +> **Using Claude Code?** Run `/mad-add-model ` — it scaffolds all three artifacts (models.json entry, Dockerfile, run.sh) from the closest existing template and validates the result automatically. The manual steps below are for reference. + Follow these steps to add a new model to the MAD repository: #### Step 1: Create Workload Name