Repository navigation
Conversation
…edictions New `commonlid.analysis` module and `commonlid difficulty` CLI command that line up the per-sample predictions of the top-k models on a dataset (from the commonlid-results HF dataset or a local results dir) and report: - per sample: how many models get it wrong, the consensus prediction and a `label_suspect` flag when most models agree on a different, non-und label (candidate annotation error); optionally joins the text back in - per gold language: mean error rate, all-wrong / all-correct shares, most common confusion and per-model recall `--collapse-macrolanguages` treats individual languages and their macrolanguage (lvs/lav, cmn/zho) as a match. Functions are plain pandas so the leaderboard can reuse them later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QSqH6rXS2CuL5zySbJsGU
…asiest-first order The output covers easy rows as well as hard ones, so `difficulty` was misleading. Rename the CLI command to `commonlid breakdown`, the module to `commonlid.analysis.breakdown` and the functions to `sample_breakdown` / `language_breakdown`. Add `--order hardest|easiest` and `--max-error-rate` so the easy end of the table is as reachable as the hard end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QSqH6rXS2CuL5zySbJsGU
"hardest/easiest" did not say which measure it ranked by. Replace `--order hardest|easiest` with `--sort-by COLUMN` (default `error_rate`) and `--order desc|asc`. Rename the sample-level `consensus_share` column to `agreement` and add a mean `agreement` column at language level so rows can be ranked by high/low error or high/low model agreement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QSqH6rXS2CuL5zySbJsGU
Key findings so far (macrolanguage mismatches, label_suspect candidates on commonlid and commonlid_nano, hardest / lowest-agreement languages) and the next steps, so the work can be picked up elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QSqH6rXS2CuL5zySbJsGU
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QSqH6rXS2CuL5zySbJsGU
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.