Our method

Introduction

Code Index ranks the most recent state-of-the-art Large Language Models based on their coding capabilities. The final index consolidates scores from currently 4 high-quality benchmarks with 20 tasks each. As a live benchmark, more task sets will soon be added and then contribute to the final Index.

Coding benchmarks usually contain tasks narrowed to specific use cases or knowledge domains, which can vary daily in real-world scenarios for a single developer. Our tasks, therefore, follow the rubrics of multiple benchmarks, so that we can provide a single general index for a valuable and practical comparison. Because the tasks are kept private, they carry less contamination risk than public benchmark tasks.

Benchmarks

Each of the current 4 benchmarks contains new, private tasks that follow an established task format and rubric.

• SWE-Bench-Pro Hard: These tasks bring complex real-world pull-request scenarios to really challenge SOTA models. To learn more: SWE-Bench Pro.
• SWE-Lancer: The tasks derived from OpenAI's benchmark contribute to our index with market-relevant demands. Job posts from Freelancer, Upwork, RedditHire, and 99Lancers. To learn more: SWE-Lancer - published February 18, 2025.
• Terminal Bench 2.1: One of the most relevant benchmarks on the market is used to increment diversity of domains. To learn more: Terminal-Bench 2.1 verified leaderboard.
• Frontier-Bench: Follows the format of Frontier-Bench, by Stanford and the Laude Institute.

Task domains and difficulty

The benchmark pages report task categories from the task metadata. Difficulty is empirical, not assigned by the author. A task is challenging when no included run solves it, hard when its pass rate across the included runs is above 0% and at most 33%, medium when it is above 33% and at most 66%, easy when it is above 66% but below 100%, and trivial when every run solves it.

Task generation

All 80 tasks were assembled for this index and kept private before evaluation, which reduces direct lookup and training contamination. Ambiguous instructions and weak tests can still distort a private benchmark, so each task goes through author validation, human review, and automated checks before it enters the set.

Written and reviewed by real engineers

Senior developers from Revelo's network author the tasks. Authors and reviewers have at least five years of professional experience, work with Python, JavaScript, or TypeScript, and receive training in data annotation for model evaluation.

After completing the task, the author must run a reference solution to show that the task is solvable. Then, two reviewers check each task before it enters the benchmark. Human review is paired with automated gates and an LLM rubric validation. The process iterates until the task is fully approved.

Our review breakdown:

• Underspecification: The instructions must contain enough information to solve what the tests require. Reviewers look for missing context and mismatches between the request and the verifier.
• Overspecification: The verifiers must validate every requirement from the instructions. Reviewers evaluate whether the instructions do not contain extra information that may influence the agent's behavior without verifying it.
• Reward hacking: Reviewers inspect whether an agent passed by solving the task or by bypassing the grader, for example by hardcoding test values or disabling a check.
• Broken verifiers: Tests must cover the required behavior and reject incomplete solutions. A passing test suite is useful only if it checks the work described in the task.

Our metrics

Success criteria

The current 4 benchmarks use the same aggregation rules. Their verifiers differ:

• SWE-Bench-Pro Hard : Fail-to-pass tests verify the requested fix. Pass-to-pass tests check that existing repository behavior still works. Both held-out test sets must pass.
• SWE-Lancer : Each task runs its own held-out executable tests against the finished application. The task passes only when all required checks succeed.
• Terminal Bench 2.1 : Tests assert the final state of the container rather than the commands used to reach it, so any working route to the result counts.
• Frontier-Bench : The final verifiers evaluate not only against the test suite, but also against a set of rubrics to ensure the task is fair and solvable.

Index score - Pass@1→

We first calculate a pass rate for each benchmark, then take the unweighted mean of those rates. Each benchmark therefore contributes equally to a complete index score.

The scorer uses exactly three trials for each model-task combination. We average those three binary outcomes to get a task score, then average the task scores to get the benchmark pass rate.

The scoring pipeline treats verifier failures, execution errors, and runs without a positive reward as failures. However, our Harbor jobs allow a single infrastructure retry for specified timeout and platform errors.

Expected execution time and token cost→

Time is the agent execution time in minutes, excluding verifier time. The cost comes from the agent's reported USD cost. For both metrics, the pipeline averages all runs (per task and for all tasks), then gives each available benchmark equal weight.

Score-versus-steps→

The score-versus-steps chart is a P95-truncated empirical cumulative curve. The observed runs are re-scored with only the succeeding tasks whose final step count fits the budget, using an unweighted aggregation that iterates for all simmulated budgets. The curves end, then, at a ceiling of the 95th percentile of successful their trajectory lengths.