rubric

Interactively build a text rubric over your dataaset of text.

rubric is a tool where you interact with a curator agent, that gathers a dataset of your texts and scores on those texts, and then tries to collaborate with you to create a rubric of interpretable features that explain the scores.

rubric uses ramure as the underlying agent library.

Requirements

You need the following setup:

• the pi agent harness
• anthropic API key, you can put it in your .env
• cloning the repo and installing its packages

Quickstart

We use uv. Start the rubric program:

uv run python -m rubric

It spawns the curator with access to the right tools, you should let it run there.

────────────────────────────────────────────────────────────────
  rubric session ready (id ab12cd34)
  empty state
  judge model: default (claude-opus-4-6)
  autosave:    /home/you/some/workdir/state.json

  attach in another terminal:
    uv run ramure connect curator --id ab12cd34

  Ctrl+C here ends the session and saves state.
────────────────────────────────────────────────────────────────

In a second terminal, copy the ramure connect line from the banner, and start talking with the curator.

Describe what you want to make a rubric for, and what data you already have to help with rubric generation (eg files on your system, inputting your own scores, etc...)

The curator will save all these datapoints. Then, when you want, it can start proposing features to explain the data, and then evaluating those features and getting statistical reports on which ones are useful.

When you've seen enough, say "we're done." The curator will call finish(summary), which writes the final state to your --save file (or a timestamped fallback) and ends the host process. You can always resume off of your state file.

To pick up where you left off, pass the same file as both --state (load from) and --save (autosave to):

uv run --project ~/code/rubric python -m rubric \
    --state state.json \
    --save  state.json

CLI flags

flag	meaning
-d, --datapoints FILE	preload a JSONL of {"text":..., "score":...} rows
-s, --save FILE	autosave to this path on every mutation, and on finish / Ctrl+C
--state FILE	resume from a previously-saved session JSON
-m, --model MODEL	override the default judge model (claude-opus-4-6)

curator tools

tool	effect
add_datapoint(text, score)	record a labeled example
add_feature(name, description, judge_prompt)	register a rubric feature; judge_prompt is the 0–10 scoring rubric the LLM judge will see
disable_feature(name) / enable_feature(name)	toggle a feature without deleting its data; disabled features are excluded from new judging and from every section of the report
list_state()	show datapoints, features (active + disabled), judged-cell counts
ask_user_for_grade(text)	shows text in chat; you reply with a score and the curator records it
judge(feature="", text="", model="")	run the judge in parallel; with no args, scores every active feature against every datapoint, skipping cached cells
report(top_subsets=5)	five sections: per-feature Pearson r/p; feature-feature correlation matrix; full-model OLS with R², adj-R², LOO-R², AIC, BIC; top-N subsets ranked by BIC; Lasso with leave-one-out CV
save(path="")	persist state to JSON
finish(summary)	end the session; always writes a final JSON to disk