Issues

• PassAtK returns 1.0 for all-wrong samples when k > n

  huggingface/lighteval#1410 · Filter by author caiotheodoro opened on Sep 30, 2026
  1

• [BUG] 187 multilingual/QA tasks can't load on datasets>=4 (script-based datasets); ifeval-fr/gpqa-fr/bac-fr repos are gone

  huggingface/lighteval#1406 · Filter by author David-Wu1119 opened on Sep 28, 2026
  1

• [BUG] LiveCodeBench scores every generation as -2 when multiprocessing uses spawn or forkserver

  huggingface/lighteval#1400 · Filter by author David-Wu1119 opened on Sep 27, 2026

• [FT] Add a format-neutral post-run export hook

  huggingface/lighteval#1397 · Filter by author halvrenofviryel opened on Sep 23, 2026

• Metrics.drop aggregates em/f1 across documents with max, so one correct sample reports 1.0

  huggingface/lighteval#1395 · Filter by author shaurya416 opened on Sep 23, 2026

• avg_at_n silently ignores strip_strings/normalize (preprocess not applied), deflating scores

  huggingface/lighteval#1384 · Filter by author Abelo9996 opened on Sep 10, 2026

• Normalized multiple-choice probability underflows for long sequence log-probabilities

  huggingface/lighteval#1375 · Filter by author YusefSyed opened on Sep 2, 2026

• SamplingMetric string normalize argument always raises: getmembers() list treated as a dict

  huggingface/lighteval#1365 · Filter by author Abelo9996 opened on Aug 29, 2026

• Prediction cache identity is the model config, not the model that ran: weights moving under the same revision name silently reuse the old responses

  huggingface/lighteval#1364 · Filter by author AUTHENSOR opened on Aug 29, 2026

• Sample cache: doc.id collides across evaluation splits, crashing runs and silently serving one split's answers for another split's questions

  huggingface/lighteval#1363 · Filter by author AUTHENSOR opened on Aug 29, 2026
  1

• OpenAI-compat via endpoint litellm: openai/ prefix and /v1 base_url

  huggingface/lighteval#1362 · Filter by author cursor opened on Aug 28, 2026

• F1_score collapses nltk's undefined result onto zero, including when the gold reference is empty

  huggingface/lighteval#1353 · Filter by author buzzedaldrinx86-creator opened on Aug 23, 2026
  1

• [BUG] Why isn't apply_chat_template called with add_generation_prompt=True in LLM as a judge?

  bug
  huggingface/lighteval#1346 · Filter by author Jeronymous opened on Aug 18, 2026

• [BUG] Incompatibility with latest version of xxhash

  bug
  huggingface/lighteval#1330 · Filter by author Giuseppe5 opened on Aug 13, 2026
  1

• [BUG] dyck_language scores with inspect_ai's exact(), which ignored word order in every release up to 0.3.252

  bug
  huggingface/lighteval#1329 · Filter by author LippInc opened on Aug 13, 2026

• lighteval eval crashes on fresh install: GenerateConfig rejects frequence_penalty / log_probs / response_format with current inspect-ai

  huggingface/lighteval#1327 · Filter by author sk8ordie84 opened on Aug 11, 2026

• [Proposal] OpenEval Import/Export Support

  huggingface/lighteval#1318 · Filter by author adhabnr-ux opened on Jul 29, 2026
  3

• Lighteval TriviaQA evaluation not giving results as per smolLM-360M reported numbers

  bug
  huggingface/lighteval#1316 · Filter by author Kun-AI-san opened on Jul 27, 2026
  1

• Balanced few-shot sampling truncates on a falsy class label and uses the global random module

  huggingface/lighteval#1309 · Filter by author ErenAta16 opened on Jul 20, 2026
  1

• Sequential few-shot selection mutates the task's shared few-shot pool in place, corrupting later variance seeds

  huggingface/lighteval#1307 · Filter by author ErenAta16 opened on Jul 20, 2026
  1

• [EVAL] Add ArmBench-LLM Armenian evaluation suite

  huggingface/lighteval#1306 · Filter by author bugdaryan opened on Jul 20, 2026

• avg_at_k / avg_at_k_math / maj_at_k metric regression fixtures silently no-op (metric_class name doesn't match registered Metrics enum members)

  huggingface/lighteval#1304 · Filter by author ErenAta16 opened on Jul 19, 2026

• [BUG] Task : wikitext:103:document_level does not work

  bug
  huggingface/lighteval#1299 · Filter by author debasisdwivedy opened on Jul 15, 2026
  3

• [BUG] LLM judge silently scores API error strings: litellm backend sends max_tokens as a list, provider returns 400

  bug
  huggingface/lighteval#1296 · Filter by author AbhilakshSinghReen opened on Jul 14, 2026

• hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset

  huggingface/lighteval#1293 · Filter by author ErenAta16 opened on Jul 11, 2026