Add LiveCodeBench: contamination-resistant competitive programming - #17
Add LiveCodeBench: contamination-resistant competitive programming#17DogukanUrker wants to merge 2 commits into
Conversation
|
Warning Review limit reached
Next review available in: 20 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (11)
📝 WalkthroughWalkthroughBenchKit adds LiveCodeBench with date filtering, dataset preparation and caching, sandboxed evaluation, CLI support, and difficulty-based reporting. Engine and report formats now support grouped task results and benchmark notes. ChangesLiveCodeBench integration
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant CLI
participant LiveCodeBench
participant DatasetCache
participant Sandbox
CLI->>LiveCodeBench: prepare or load tasks
LiveCodeBench->>DatasetCache: download, normalize, and read cached data
CLI->>LiveCodeBench: evaluate response
LiveCodeBench->>Sandbox: execute generated code against tests
Sandbox-->>LiveCodeBench: return pass or fail results
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/benchkit/cli.py (1)
283-295: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAvoid suppressing non-dataset failures in the headless count fallback.
The
execute 1path catches everytask_countexception and defaultstotal = 0, so a malformed registry entry or import error prints a warning and skips argument validation instead of failing fast. Limit the fallback to the known “uncached dataset” failure and raise or exit for other count errors.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/benchkit/cli.py` around lines 283 - 295, Restrict the exception fallback around task_count in the execute 1 flow to the known uncached-dataset failure only, preserving total = 0 for that case so slice syntax can still be checked. For malformed registry entries, import errors, and other task_count exceptions, propagate the error or exit instead of printing the warning and continuing; update the handling near task_count, SliceError, and specs.append.
🧹 Nitpick comments (5)
src/benchkit/benchmarks/livecodebench.py (1)
370-381: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSilence the Ruff S324 hash warning explicitly.
Ruff reports
S324at line 380. The digest here disambiguates file names only. It is not a security control. Passusedforsecurity=Falseso the intent is explicit and the lint gate stays green.♻️ Proposed change
- digest = hashlib.sha1(question_id.encode("utf-8")).hexdigest()[:8] + digest = hashlib.sha1( + question_id.encode("utf-8"), usedforsecurity=False + ).hexdigest()[:8]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/benchkit/benchmarks/livecodebench.py` around lines 370 - 381, Update the hashlib.sha1 call in _safe_name to pass usedforsecurity=False, preserving the existing UTF-8 input and eight-character digest used for filename disambiguation.Source: Linters/SAST tools
tests/test_livecodebench.py (2)
222-224: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winShorten the timeout for the non-terminating test.
This test runs the real sandbox until the budget expires.
stdin-1has two test cases, soevaluatesetsbudget = min(90, 6 * 2) = 12and callsexecutewithtimeout=17. The single test therefore blocks for about 17 seconds on every run. PatchPER_TEST_TIMEOUT_Sdown for this case to keep the suite fast while still proving the timeout path.♻️ Proposed change
+ def test_code_that_never_terminates_fails(self) -> None: + response = "```python\nwhile True:\n pass\n```" + with unittest.mock.patch( + "benchkit.benchmarks.livecodebench.PER_TEST_TIMEOUT_S", 1 + ): + self.assertFalse(self.bench.evaluate(self._task("stdin-1"), response)) - def test_code_that_never_terminates_fails(self) -> None: - response = "```python\nwhile True:\n pass\n```" - self.assertFalse(self.bench.evaluate(self._task("stdin-1"), response))Add
import unittest.mockat the top of the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_livecodebench.py` around lines 222 - 224, Update test_code_that_never_terminates_fails to patch benchkit.benchmarks.livecodebench.PER_TEST_TIMEOUT_S to 1 around the evaluate call, preserving the existing non-termination assertion. Add the unittest.mock import required for the patch.
60-64: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRestore the previous environment instead of deleting it.
setUpandtearDownremoveBENCHKIT_LCB_AFTERandBENCHKIT_LCB_BEFOREfrom the process environment permanently. A developer who exports either variable loses it for every test that runs after this class.DatasetTestsalready snapshots and restoresos.environ. Use the same pattern here.♻️ Proposed change
class WindowTests(unittest.TestCase): def setUp(self) -> None: + self._environ = dict(os.environ) for name in ("BENCHKIT_LCB_AFTER", "BENCHKIT_LCB_BEFORE"): os.environ.pop(name, None) - tearDown = setUp + def tearDown(self) -> None: + os.environ.clear() + os.environ.update(self._environ)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_livecodebench.py` around lines 60 - 64, Update DatasetTests’ environment handling around setUp and tearDown to snapshot the original BENCHKIT_LCB_AFTER and BENCHKIT_LCB_BEFORE values before each test and restore them afterward, including removing only variables that were originally absent. Do not permanently delete values inherited from the developer’s environment..env.example (1)
34-38: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument the remaining LiveCodeBench variables.
src/benchkit/benchmarks/livecodebench.pyalso readsBENCHKIT_LCB_CACHE(line 95) andBENCHKIT_LCB_MAX_TESTS(line 103). The cache location and the per-problem test cap both change disk usage and run time. Add commented entries so users can find them here.♻️ Proposed addition
# Reuse an already-downloaded release instead of fetching it # BENCHKIT_LCB_DATASET=~/datasets/code_generation_lite +# Where the prepared release is cached (default: ~/.cache/benchkit/livecodebench) +# BENCHKIT_LCB_CACHE= +# Maximum test cases kept per problem (default: 25) +# BENCHKIT_LCB_MAX_TESTS=25🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.env.example around lines 34 - 38, Update the LiveCodeBench environment-variable section in .env.example to document BENCHKIT_LCB_CACHE and BENCHKIT_LCB_MAX_TESTS as commented entries, with concise descriptions covering the cache location and per-problem test limit. Keep the existing variables and formatting unchanged.src/benchkit/cli.py (1)
606-609: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winValidate the LiveCodeBench window only when a command uses it.
_apply_livecodebench_windowruns before every dispatch and callswindow(), which parsesBENCHKIT_LCB_AFTERfrom the environment and from.env. A malformed value therefore makesbenchkit perf,benchkit --list, and the TUI exit 1, even though none of them touch LiveCodeBench. Set the environment variables here, and validate the window in_lcb_prepareand in the run paths that need it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/benchkit/cli.py` around lines 606 - 609, Remove the unconditional _apply_livecodebench_window call from main so unrelated commands do not parse LiveCodeBench configuration; set the relevant environment variables there instead. Move window validation into _lcb_prepare and each run path that uses LiveCodeBench, preserving existing behavior for commands that require the window.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 293-295: Update the fenced example in README.md around the
livecodebench entry to include an appropriate language identifier on the opening
fence, such as text, while preserving the example content.
- Line 255: Update the LiveCodeBench documentation at README.md lines 255-255,
272-277, 285-286, and 341-342 to describe contamination protection using the
operator-provided after/before date window, not the selected model’s training
cutoff. Clarify that the lower bound must be later than the model cutoff for the
guarantee, while the all option removes the lower bound and scores the whole
release.
In `@src/benchkit/benchmarks/livecodebench.py`:
- Around line 613-626: Update JobSpec.planned_total() in the job construction
flow to handle DatasetUnavailable from LiveCodeBench.task_count without
propagating the exception. Treat the count as unknown or zero so job
construction continues, or ensure LiveCodeBench is prepared before building the
job; preserve existing count behavior when the dataset is cached.
In `@src/benchkit/cli.py`:
- Around line 581-603: Separate the `prepare()` and
`LiveCodeBench().load_tasks()` handling in the CLI flow so `DatasetUnavailable`
does not use the preparation-failure path. Import `DatasetUnavailable` from
`benchkit.benchmarks.livecodebench`, preserve the successful `manifest` output
when loading tasks reports an empty window, and then report the window as empty
without exiting as a preparation failure; retain the existing error handling for
`RuntimeError`, `ValueError`, and `OSError`.
---
Outside diff comments:
In `@src/benchkit/cli.py`:
- Around line 283-295: Restrict the exception fallback around task_count in the
execute 1 flow to the known uncached-dataset failure only, preserving total = 0
for that case so slice syntax can still be checked. For malformed registry
entries, import errors, and other task_count exceptions, propagate the error or
exit instead of printing the warning and continuing; update the handling near
task_count, SliceError, and specs.append.
---
Nitpick comments:
In @.env.example:
- Around line 34-38: Update the LiveCodeBench environment-variable section in
.env.example to document BENCHKIT_LCB_CACHE and BENCHKIT_LCB_MAX_TESTS as
commented entries, with concise descriptions covering the cache location and
per-problem test limit. Keep the existing variables and formatting unchanged.
In `@src/benchkit/benchmarks/livecodebench.py`:
- Around line 370-381: Update the hashlib.sha1 call in _safe_name to pass
usedforsecurity=False, preserving the existing UTF-8 input and eight-character
digest used for filename disambiguation.
In `@src/benchkit/cli.py`:
- Around line 606-609: Remove the unconditional _apply_livecodebench_window call
from main so unrelated commands do not parse LiveCodeBench configuration; set
the relevant environment variables there instead. Move window validation into
_lcb_prepare and each run path that uses LiveCodeBench, preserving existing
behavior for commands that require the window.
In `@tests/test_livecodebench.py`:
- Around line 222-224: Update test_code_that_never_terminates_fails to patch
benchkit.benchmarks.livecodebench.PER_TEST_TIMEOUT_S to 1 around the evaluate
call, preserving the existing non-termination assertion. Add the unittest.mock
import required for the patch.
- Around line 60-64: Update DatasetTests’ environment handling around setUp and
tearDown to snapshot the original BENCHKIT_LCB_AFTER and BENCHKIT_LCB_BEFORE
values before each test and restore them afterward, including removing only
variables that were originally absent. Do not permanently delete values
inherited from the developer’s environment.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 19b83fe0-5c06-4b67-bd5c-d348f2366532
📒 Files selected for processing (11)
.env.exampleREADME.mdsrc/benchkit/benchmarks/__init__.pysrc/benchkit/benchmarks/livecodebench.pysrc/benchkit/cli.pysrc/benchkit/datasets/README.mdsrc/benchkit/engine.pysrc/benchkit/report.pysrc/benchkit/templates/report.htmlsrc/benchkit/tui/screens/setup.pytests/test_livecodebench.py
| try: | ||
| manifest = prepare() | ||
| counts = Counter( | ||
| str(task.metadata.get("difficulty", "unknown")) | ||
| for task in LiveCodeBench().load_tasks() | ||
| ) | ||
| except (RuntimeError, ValueError, OSError) as exc: | ||
| console.print(f"[red]LiveCodeBench preparation failed:[/red] {exc}") | ||
| sys.exit(1) | ||
|
|
||
| console.print( | ||
| f"[dim]Cached:[/dim] {manifest['problems']:,} problems " | ||
| f"({manifest['first_contest_date']} to {manifest['last_contest_date']}) " | ||
| f"· up to {manifest['max_tests']} tests each" | ||
| ) | ||
| in_window = sum(counts.values()) | ||
| breakdown = " · ".join( | ||
| f"{name} {counts[name]}" for name in DIFFICULTIES if counts.get(name) | ||
| ) | ||
| console.print( | ||
| f"[white]{in_window:,}[/white] problems in {active.label}" | ||
| + (f" [dim]({breakdown})[/dim]" if breakdown else "") | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Separate an empty window from a preparation failure.
prepare() and LiveCodeBench().load_tasks() share one try block. load_tasks raises DatasetUnavailable when no problem falls inside the window (src/benchkit/benchmarks/livecodebench.py lines 604-607). In that case the download and normalization already succeeded, but the command prints "LiveCodeBench preparation failed" and exits 1. Report the cached release, then report the empty window separately.
🐛 Proposed fix
try:
manifest = prepare()
- counts = Counter(
- str(task.metadata.get("difficulty", "unknown"))
- for task in LiveCodeBench().load_tasks()
- )
except (RuntimeError, ValueError, OSError) as exc:
console.print(f"[red]LiveCodeBench preparation failed:[/red] {exc}")
sys.exit(1)
console.print(
f"[dim]Cached:[/dim] {manifest['problems']:,} problems "
f"({manifest['first_contest_date']} to {manifest['last_contest_date']}) "
f"· up to {manifest['max_tests']} tests each"
)
+ try:
+ counts = Counter(
+ str(task.metadata.get("difficulty", "unknown"))
+ for task in LiveCodeBench().load_tasks()
+ )
+ except DatasetUnavailable as exc:
+ console.print(f"[yellow]{exc}[/yellow]")
+ return
in_window = sum(counts.values())Import DatasetUnavailable from benchkit.benchmarks.livecodebench.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| try: | |
| manifest = prepare() | |
| counts = Counter( | |
| str(task.metadata.get("difficulty", "unknown")) | |
| for task in LiveCodeBench().load_tasks() | |
| ) | |
| except (RuntimeError, ValueError, OSError) as exc: | |
| console.print(f"[red]LiveCodeBench preparation failed:[/red] {exc}") | |
| sys.exit(1) | |
| console.print( | |
| f"[dim]Cached:[/dim] {manifest['problems']:,} problems " | |
| f"({manifest['first_contest_date']} to {manifest['last_contest_date']}) " | |
| f"· up to {manifest['max_tests']} tests each" | |
| ) | |
| in_window = sum(counts.values()) | |
| breakdown = " · ".join( | |
| f"{name} {counts[name]}" for name in DIFFICULTIES if counts.get(name) | |
| ) | |
| console.print( | |
| f"[white]{in_window:,}[/white] problems in {active.label}" | |
| + (f" [dim]({breakdown})[/dim]" if breakdown else "") | |
| ) | |
| try: | |
| manifest = prepare() | |
| except (RuntimeError, ValueError, OSError) as exc: | |
| console.print(f"[red]LiveCodeBench preparation failed:[/red] {exc}") | |
| sys.exit(1) | |
| console.print( | |
| f"[dim]Cached:[/dim] {manifest['problems']:,} problems " | |
| f"({manifest['first_contest_date']} to {manifest['last_contest_date']}) " | |
| f"· up to {manifest['max_tests']} tests each" | |
| ) | |
| try: | |
| counts = Counter( | |
| str(task.metadata.get("difficulty", "unknown")) | |
| for task in LiveCodeBench().load_tasks() | |
| ) | |
| except DatasetUnavailable as exc: | |
| console.print(f"[yellow]{exc}[/yellow]") | |
| return | |
| in_window = sum(counts.values()) | |
| breakdown = " · ".join( | |
| f"{name} {counts[name]}" for name in DIFFICULTIES if counts.get(name) | |
| ) | |
| console.print( | |
| f"[white]{in_window:,}[/white] problems in {active.label}" | |
| (f" [dim]({breakdown})[/dim]" if breakdown else "") | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/benchkit/cli.py` around lines 581 - 603, Separate the `prepare()` and
`LiveCodeBench().load_tasks()` handling in the CLI flow so `DatasetUnavailable`
does not use the preparation-failure path. Import `DatasetUnavailable` from
`benchkit.benchmarks.livecodebench`, preserve the successful `manifest` output
when loading tasks reports an empty window, and then report the window as empty
without exiting as a preparation failure; retain the existing error handling for
`RuntimeError`, `ValueError`, and `OSError`.
|
@copilot resolve the merge conflicts in this pull request |
LiveCodeBench timestamps its problems, so a run can be restricted to problems published after a model's training cutoff. That window is the point of the suite, so it is always applied: it defaults to 2024-08 onwards, moves with --lcb-after / --lcb-before, and is printed with the results so a score documents itself. The code_generation_lite release is fetched on demand and cached rather than bundled, the way the EvalPlus suites get their data; `benchkit lcb-prepare` does it ahead of a run. Hidden tests run in the existing sandbox and cover both call-a-function and stdin/stdout problems. Scores are reported per difficulty as well as in aggregate, through a generic hook: a task metadata `group` gives any benchmark a breakdown in the CLI summary, results.json, results.csv, results.md and results.html, and a `report_note` documents how the suite was configured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015DBjLyFRpnqpDtWWU1edH7
The picker showed a bare "unavailable" for any suite whose tasks could not be counted, which is exactly what LiveCodeBench does before its dataset is cached - correct, since browsing benchmarks must never start a several-hundred-megabyte download, but unhelpful. The count worker now carries the reason onto the row, and `--list` shows LiveCodeBench's `list_note` pointing at `benchkit lcb-prepare`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015DBjLyFRpnqpDtWWU1edH7
5eeea68 to
1b0516b
Compare
2 participants
Adds LiveCodeBench as a new benchmark suite, bringing contamination-resistant code generation evaluation to BenchKit. LiveCodeBench timestamps its problems, allowing runs to be restricted to problems published after a model's training cutoff—making it the first suite in BenchKit that can definitively separate memorization from capability.
Key Changes
New benchmark module (
src/benchkit/benchmarks/livecodebench.py): Complete implementation of LiveCodeBench integration including:code_generation_literelease on demand)BENCHKIT_LCB_AFTERandBENCHKIT_LCB_BEFOREenvironment variablesCLI enhancements (
src/benchkit/cli.py):lcb-preparecommand to download and cache the dataset--lcb-afterand--lcb-beforeflags for contest-date windowingEngine updates (
src/benchkit/engine.py):group_breakdown()function to compute per-group pass rates for labeled benchmarksTaskRecord.groupfield to track task grouping (e.g., difficulty level)Report generation (
src/benchkit/report.py):group_summary()function for readable per-group score formattingDocumentation and configuration:
.env.examplewith LiveCodeBench configuration optionsFixes #2
Notable Implementation Details
_NoGlobalsUnpickler) prevents code execution when decoding test cases from the downloaded dataset2024-08onwards) targets most 4-35B models;--lcb-after allenables full-release scoring for transparencyBENCHKIT_LCB_DATASET, custom base URL viaBENCHKIT_LCB_BASE_URL, or automatic download from HuggingFaceThe implementation treats LiveCodeBench as a first-class benchmark with full integration into BenchKit's evaluation pipeline, reporting infrastructure, and TUI.
https://claude.ai/code/session_015DBjLyFRpnqpDtWWU1edH7
Summary by CodeRabbit