Results

Detailed results

For deeper analysis and transparency into the results, this page covers the metrics for each dataset and the results per trial for all 3 benchmarks in the index. It also keeps the results of SWE-Bench-Pro Hard for reference; they do not count toward the index.

Code Index per benchmark

Here we show not only the Index score per benchmark, but also the consumed tokens and the total cost of each model.

1
gemini-3.8-flash
terminus-2
275.1M 4.9M
$64.3
51.7
2
claude-fable-5
claude-code
48.2M 1.8M
$164.0
51.7
3
grok-4.5
terminus-2
20.2M 1.4M
$19.3
46.7
4
claude-opus-4.8
claude-code
94.5M 1.7M
$105.7
45.0
5
claude-opus-5
claude-code
217.3M 3.9M
$237.1
45.0
6
kimi-k3
terminus-2
12.1M 1.8M
$35.4
43.3
7
gpt-5.6-sol
codex
31.1M 1.0M
$57.7
43.3
8
gemini-3.6-flash
terminus-2
113.6M 3.1M
$78.5
41.7
9
gpt-5.6-terra
codex
16.9M 766.2K
$16.4
40.0
10
glm-5.2
terminus-2
18.7M 4.0M
$31.5
40.0
11
gpt-6-astra
codex
39.6M 1.8M
$186.3
40.0
12
gpt-5.6-luna
codex
32.8M 1.1M
$2.5
38.3
13
deepseek-v4.1-flash
deepseek-harness-minimal
299.8M 5.7M
$14.3
35.0
14
deepseek-v4-flash-0731
terminus-2
24.8M 4.7M
$2.6
35.0
15
qwen3.8-max
claude-code
74.5M 2.3M
$123.8
28.3
16
deepseek-v4-pro
terminus-2
49.1M 2.6M
$20.0
26.7
17
minimax-m3
terminus-2
208.0M 3.0M
$17.5
21.7
18
hy3
terminus-2
35.3M 4.9M
$4.5
16.7
19
mimo-v2.5-pro
terminus-2
29.9M 1.7M
$5.5
6.7
20
nemotron-3-ultra-550b-a55b
terminus-2
22.3M 1.2M
$10.5
1.7

Trial outcomes

Below you can find the outcome of each trial. Each model ran three trials per task, which resulted in either PASS, FAIL, or ERROR. Each is described below, along with the error description, task domain, and task empirical difficulty, which can be seen by hovering each result dot.

Talk to us

Revelo works with model labs on:

  • Training datasets for code generation
  • Evaluations of new and unreleased models
  • Leaderboard analysis, including failure modes and agent trajectories

Talk to us

Tell us whether you need a dataset, an evaluation, or help analyzing model results.

Loading form…