Skip to content

PassAtK returns 1.0 for all-wrong samples when k > n #1410

Open

Description

@caiotheodoro

PassAtK.pass_at_k returns 1.0 whenever n - c < k (metrics_sample.py:1321). With k > n that holds for every input, so a sample with zero correct answers scores a perfect pass@k. GPassAtK in the same file returns 0.0 for k > n (_compute_g_pass_at_k, metrics_sample.py:1416).

Repro on main (b634749), Python 3.12.13, fresh venv, pip install -e .:

from lighteval.metrics.metrics import Metrics
from lighteval.metrics.metrics_sample import GPassAtK
from lighteval.tasks.requests import Doc
from lighteval.models.model_output import ModelResponse

doc = Doc(query="2+2?", choices=["4"], gold_index=0)
resp = ModelResponse(text=["5", "6", "7", "8"])  # 4 samples, all wrong

for k in (2, 4, 5, 10):
    m = Metrics.pass_at_k(sample_params={"k": k, "n": 4})
    print(f"pass@k k={k} n=4:", m.sample_level_fn.compute(doc, resp))

g = GPassAtK(k=10, n=4, strip_strings=True)
print("g-pass@10 n=4:", g.compute(doc, resp)["g-pass@10_0.0"])
pass@k k=2 n=4: 0.0
pass@k k=4 n=4: 0.0
pass@k k=5 n=4: 1.0
pass@k k=10 n=4: 1.0
g-pass@10 n=4: 0.0

k and n come from sample_params (the same call the built-in tasks use, e.g. math_500.py:69), and nothing checks k <= n. Since only the corpus mean is reported, a config like {"k": 16, "n": 8} shows up as a pass@16 of 1.0 with no warning.

Ways to fix it:

  1. Raise ValueError in pass_at_k when k > n. This can't go in __init__, because Metric.__call__ sets k/n with setattr afterwards.
  2. Return 0.0 for k > n, matching GPassAtK.

I'd go with 1, since pass@k is undefined there and a silent 0.0 is also misleading. Happy to send a PR with a test for whichever you prefer.

Activity

  1. woodforestsen commented on Oct 1, 2026

    @woodforestsen

    The behaviour here is clearly a bug: with k > n, a sample with zero correct answers scores a perfect pass@k, and since only the corpus mean is reported it surfaces as a silent 1.0 with no warning. Raising seems better than silently returning 0.0 as well — 0.0 would imply the samples were actually measured.

    I'd like to pick this up if that's alright, @caiotheodoro — I see you offered to send the PR yourself, so I'm happy to stand down if you'd rather write it.

    My plan (option 1 from your report):

    • raise ValueError in pass_at_k when k > n
    • regression test covering k > n, plus assertions that k < n and k == n are unchanged
    • check the built-in tasks that pass sample_params (e.g. math_500.py) so no shipped configuration starts failing loudly

    Happy to switch to option 2, or to a warning instead of raising, if the maintainers prefer that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions