Benchmark
SWE-Bench-Pro Hard
Each task is a bug fix in an established repository. It starts from the commit right before a real issue was fixed, so the agent has to read the surrounding code before it can repair anything, and the fix has to leave existing behavior intact. Many of them touch several files.
The format comes from Scale AI's SWE-Bench Pro, which covers longer repository work. We drew the tasks from pull requests in public community repositories and kept them private, and each one holds back tests for the requested change and for the behavior that was already there. The reference patches change a median of 173.5 lines across 5 files. There are 20 tasks in the set.
Abbreviated example task
Fix overly strict allOf property merging in openapi-python-client. Derived schemas cannot override inherited properties, and overlapping definitions replace validation rules instead of merging them. The reference fix changes 549 lines in 14 files across the generator core. Twenty-one fail-to-pass tests check the fix, while 186 pass-to-pass tests check for regressions.Task mix
The domain and empirical difficulty of the 20 tasks in this set.
Domain spread
tasks per domain
Difficulty
Challenging: 0% pass rate. Hard: above 0% to 33%. Medium: above 33% to 66%. Easy: above 66% to below 100%. Trivial: 100%.
tasks per difficulty
See SWE-Bench-Pro Hard results →
Talk to us
Revelo works with model labs on:
- Training datasets for code generation
- Evaluations of new and unreleased models
- Leaderboard analysis, including failure modes and agent trajectories