Our method

Introduction

Code Index ranks the most recent state-of-the-art Large Language Models based on their coding capabilities. The final index consolidates scores from currently 3 high-quality benchmarks with 20 tasks each. As a live benchmark, more task sets will soon be added and then contribute to the final Index.

Coding benchmarks usually contain tasks narrowed to specific use cases or knowledge domains, which can vary daily in real-world scenarios for a single developer. Our tasks, therefore, follow the rubrics of multiple benchmarks, so that we can provide a single general index for a valuable and practical comparison. Because the tasks are kept private, they carry less contamination risk than public benchmark tasks.

Benchmarks

Each of the current 3 benchmarks contains new, private tasks that follow an established task format and rubric.

• SWE-Lancer: The tasks derived from OpenAI's benchmark contribute to our index with market-relevant demands. Job posts from Freelancer, Upwork, RedditHire, and 99Lancers. To learn more: SWE-Lancer - published February 18, 2025.
• Terminal Bench 2.1: One of the most relevant benchmarks on the market is used to increment diversity of domains. To learn more: Terminal-Bench 2.1 verified leaderboard.
• Terminal Bench 3.0: Follows the format of Terminal Bench 3.0, by Stanford and the Laude Institute.

Not in the index for now: SWE-Bench-Pro Hard. Its results stay on the site for reference, but they do not count toward the index. Its page explains why.

Task domains and difficulty

The benchmark pages report task categories from the task metadata. Difficulty is empirical, not assigned by the author. A task is challenging when no included run solves it, hard when its pass rate across the included runs is above 0% and at most 33%, medium when it is above 33% and at most 66%, easy when it is above 66% but below 100%, and trivial when every run solves it.

Task generation

All 60 tasks were assembled for this index and kept private before evaluation, which reduces direct lookup and training contamination. Ambiguous instructions and weak tests can still distort a private benchmark, so each task goes through author validation, human review, and automated checks before it enters the set.

Written and reviewed by real engineers

Senior developers from Revelo's network author the tasks. Authors and reviewers have at least five years of professional experience, work with Python, JavaScript, or TypeScript, and receive training in data annotation for model evaluation.

After completing the task, the author must run a reference solution to show that the task is solvable. Then, two reviewers check each task before it enters the benchmark. Human review is paired with automated gates and an LLM rubric validation. The process iterates until the task is fully approved.

Our review breakdown:

• Underspecification: The instructions must contain enough information to solve what the tests require. Reviewers look for missing context and mismatches between the request and the verifier.
• Overspecification: The verifiers must validate every requirement from the instructions. Reviewers evaluate whether the instructions do not contain extra information that may influence the agent's behavior without verifying it.
• Reward hacking: Reviewers inspect whether an agent passed by solving the task or by bypassing the grader, for example by hardcoding test values or disabling a check.
• Broken verifiers: Tests must cover the required behavior and reject incomplete solutions. A passing test suite is useful only if it checks the work described in the task.

Our metrics

Success criteria

The current 3 benchmarks use the same aggregation rules. Their verifiers differ:

• SWE-Lancer : Each task runs its own held-out executable tests against the finished application. The task passes only when all required checks succeed.
• Terminal Bench 2.1 : Tests assert the final state of the container rather than the commands used to reach it, so any working route to the result counts.
• Terminal Bench 3.0 : The final verifiers evaluate not only against the test suite, but also against a set of rubrics to ensure the task is fair and solvable.

Index score - Pass@1→

We first calculate a pass rate for each benchmark in the index, then take the unweighted mean of those rates. Each benchmark in the index therefore contributes equally to a complete index score.

The scorer uses exactly three trials for each model-task combination. We average those three binary outcomes to get a task score, then average the task scores to get the benchmark pass rate.

The scoring pipeline counts a trial as a pass when its verifier reward is at least 1, also when the run ended on an execution error; verifier failures and runs without a reward count as failures. Our Harbor jobs retried some error types automatically, such as sandbox failures, API errors and non-zero agent exit codes. The retry limit was between zero and three, depending on the job, and some earlier jobs were also set to retry agent timeouts.

For the three models added in September 2026, we re-ran a trial only when it ended on an error from our infrastructure or the model provider, whatever its verifier result, with the same model, agent, reasoning effort and time limit. Failures caused by the model were kept, and an agent timeout counted as a failure of the model, unless the agent never reached the model. This covered 32 DeepSeek V4.1 Flash trials that ended on provider errors, 10 of them provider content-filter blocks; 9 GPT 6 Astra trials on Terminal Bench 2.1 that timed out because the agent could not reach the model (three task images had no CA certificate bundle; we fixed this by installing one at setup); and 1 Gemini 3.8 Flash trial that hit a sandbox command-length limit.

Agent time limit

Each task sets its own agent time limit in its task configuration, and each run multiplies that limit by a factor (1× is the task's own limit). The factor was not the same for every run, so on the same task some models had more time than others. The table shows the factor per benchmark for each model on the leaderboard. When a cell shows two factors, the trials of that benchmark ran with different factors, and the numbers in parentheses count those trials.

Expected execution time and token cost→

Time is the agent execution time in minutes, excluding verifier time. The cost comes from the agent's reported USD cost. For both metrics, the pipeline averages all runs (per task and for all tasks), then gives each benchmark in the index equal weight.

Score-versus-steps→

The score-versus-steps chart is a P95-truncated empirical cumulative curve. The observed runs are re-scored with only the succeeding tasks whose final step count fits the budget, using an unweighted aggregation that iterates for all simmulated budgets. The curves end, then, at a ceiling of the 95th percentile of successful their trajectory lengths.