You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PassAtK returns 1.0 for all-wrong samples when k > n #1410
PassAtK.pass_at_k returns 1.0 whenever n - c < k (metrics_sample.py:1321). With k > n that holds for every input, so a sample with zero correct answers scores a perfect pass@k. GPassAtK in the same file returns 0.0 for k > n (_compute_g_pass_at_k, metrics_sample.py:1416).
Repro on main (b634749), Python 3.12.13, fresh venv, pip install -e .:
k and n come from sample_params (the same call the built-in tasks use, e.g. math_500.py:69), and nothing checks k <= n. Since only the corpus mean is reported, a config like {"k": 16, "n": 8} shows up as a pass@16 of 1.0 with no warning.
Ways to fix it:
Raise ValueError in pass_at_k when k > n. This can't go in __init__, because Metric.__call__ sets k/n with setattr afterwards.
Return 0.0 for k > n, matching GPassAtK.
I'd go with 1, since pass@k is undefined there and a silent 0.0 is also misleading. Happy to send a PR with a test for whichever you prefer.
The behaviour here is clearly a bug: with k > n, a sample with zero correct answers scores a perfect pass@k, and since only the corpus mean is reported it surfaces as a silent 1.0 with no warning. Raising seems better than silently returning 0.0 as well — 0.0 would imply the samples were actually measured.
I'd like to pick this up if that's alright, @caiotheodoro — I see you offered to send the PR yourself, so I'm happy to stand down if you'd rather write it.
My plan (option 1 from your report):
raise ValueError in pass_at_k when k > n
regression test covering k > n, plus assertions that k < n and k == n are unchanged
check the built-in tasks that pass sample_params (e.g. math_500.py) so no shipped configuration starts failing loudly
Happy to switch to option 2, or to a warning instead of raising, if the maintainers prefer that.
Activity
woodforestsen commentedon Oct 1, 2026
The behaviour here is clearly a bug: with
k > n, a sample with zero correct answers scores a perfect pass@k, and since only the corpus mean is reported it surfaces as a silent1.0with no warning. Raising seems better than silently returning0.0as well —0.0would imply the samples were actually measured.I'd like to pick this up if that's alright, @caiotheodoro — I see you offered to send the PR yourself, so I'm happy to stand down if you'd rather write it.
My plan (option 1 from your report):
ValueErrorinpass_at_kwhenk > nk > n, plus assertions thatk < nandk == nare unchangedsample_params(e.g.math_500.py) so no shipped configuration starts failing loudlyHappy to switch to option 2, or to a warning instead of raising, if the maintainers prefer that.