Results
Detailed results
For deeper analysis and transparency into the results, this page covers the metrics for each dataset and the results per trial for all 4 benchmarks.
Code Index per benchmark
Here we show not only the Index score per benchmark, but also the consumed tokens and the total cost of each model.
1
gpt-5.6-sol
codex
176.0M
985.5K
$142.9
2
claude-opus-5
claude-code
308.2M
2.3M
$259.3
3
claude-fable-5
claude-code
188.9M
1.9M
$347.9
4
grok-4.5
terminus-2
115.4M
2.2M
$59.2
5
qwen3.8-max
claude-code
347.8M
2.9M
$562.1
6
claude-opus-4.8
claude-code
269.1M
1.9M
$214.2
7
gpt-5.6-terra
codex
104.8M
901.6K
$42.1
8
gpt-5.6-luna
codex
180.0M
1.2M
$6.3
9
kimi-k3
terminus-2
52.2M
1.4M
$45.0
10
deepseek-v4-flash-0731
terminus-2
132.7M
7.1M
$8.5
11
glm-5.2
terminus-2
122.3M
2.5M
$70.2
12
gemini-3.6-flash
terminus-2
712.6M
4.4M
$236.3
13
hy3
terminus-2
193.3M
4.0M
$10.4
14
nemotron-3-ultra-550b-a55b
terminus-2
186.8M
2.7M
$58.2
15
mimo-v2.5-pro
terminus-2
141.2M
2.9M
$17.7
16
minimax-m3
terminus-2
472.3M
2.6M
$38.8
17
deepseek-v4-pro
terminus-2
178.9M
4.4M
$47.1
1
claude-fable-5
claude-code
48.2M
1.8M
$164.0
2
grok-4.5
terminus-2
20.2M
1.4M
$19.3
3
claude-opus-4.8
claude-code
94.5M
1.7M
$105.7
4
claude-opus-5
claude-code
217.3M
3.9M
$237.1
5
kimi-k3
terminus-2
12.1M
1.8M
$35.4
6
gpt-5.6-sol
codex
31.1M
1.0M
$57.7
7
gemini-3.6-flash
terminus-2
113.6M
3.1M
$78.5
8
glm-5.2
terminus-2
18.7M
4.0M
$31.5
9
gpt-5.6-terra
codex
16.9M
766.2K
$16.4
10
gpt-5.6-luna
codex
32.8M
1.1M
$2.5
11
deepseek-v4-flash-0731
terminus-2
24.8M
4.7M
$2.6
12
qwen3.8-max
claude-code
74.5M
2.3M
$123.8
13
deepseek-v4-pro
terminus-2
49.1M
2.6M
$20.0
14
minimax-m3
terminus-2
208.0M
3.0M
$17.5
15
hy3
terminus-2
35.3M
4.9M
$4.5
16
mimo-v2.5-pro
terminus-2
29.9M
1.7M
$5.5
17
nemotron-3-ultra-550b-a55b
terminus-2
22.3M
1.2M
$10.5
1
claude-opus-5
claude-code
198.6M
3.4M
$219.2
2
gpt-5.6-sol
codex
44.6M
1.3M
$74.8
3
claude-fable-5
claude-code
99.7M
2.8M
$294.5
4
kimi-k3
terminus-2
43.2M
2.1M
$52.2
5
claude-opus-4.8
claude-code
174.5M
3.1M
$191.7
6
qwen3.8-max
claude-code
109.6M
2.7M
$178.6
7
gpt-5.6-terra
codex
44.0M
1.1M
$27.7
8
gpt-5.6-luna
codex
70.7M
1.6M
$4.1
9
glm-5.2
terminus-2
48.8M
5.4M
$49.5
10
deepseek-v4-flash-0731
terminus-2
48.7M
6.6M
$5.0
11
grok-4.5
terminus-2
95.3M
2.6M
$54.5
12
gemini-3.6-flash
terminus-2
151.6M
3.3M
$85.8
13
hy3
terminus-2
90.2M
5.3M
$7.0
14
deepseek-v4-pro
terminus-2
147.2M
6.2M
$49.3
15
minimax-m3
terminus-2
462.1M
3.8M
$38.6
16
mimo-v2.5-pro
terminus-2
87.8M
3.9M
$18.8
17
nemotron-3-ultra-550b-a55b
terminus-2
109.9M
3.1M
$43.2
1
claude-opus-5
claude-code
624.2M
12.7M
$809.0
2
claude-fable-5
claude-code
433.8M
9.2M
$1.1k
3
qwen3.8-max
claude-code
655.3M
9.9M
$1.2k
4
gpt-5.6-sol
codex
217.4M
2.8M
$217.4
5
kimi-k3
terminus-2
175.7M
4.2M
$136.3
6
gpt-5.6-luna
codex
179.4M
2.4M
$7.7
7
claude-opus-4.8
claude-code
449.4M
6.6M
$455.5
8
gpt-5.6-terra
codex
82.9M
1.6M
$43.4
9
gemini-3.6-flash
terminus-2
2.75B
6.5M
$586.9
10
hy3
terminus-2
516.7M
11.3M
$38.7
11
deepseek-v4-pro
terminus-2
1.76B
19.8M
$248.8
12
deepseek-v4-flash-0731
terminus-2
343.7M
19.4M
$16.8
13
glm-5.2
terminus-2
538.0M
12.1M
$294.8
14
mimo-v2.5-pro
terminus-2
383.8M
8.6M
$75.7
15
minimax-m3
terminus-2
1.19B
7.2M
$87.8
16
grok-4.5
terminus-2
1.41B
8.3M
$767.7
17
nemotron-3-ultra-550b-a55b
terminus-2
730.1M
6.5M
$219.2
Trial outcomes
Below you can find the outcome of each trial. Each model ran three trials per task, which resulted in either PASS, FAIL, or ERROR. Each is described below, along with the error description, task domain, and task empirical difficulty, which can be seen by hovering each result dot.
Talk to us
Revelo works with model labs on:
- Training datasets for code generation
- Evaluations of new and unreleased models
- Leaderboard analysis, including failure modes and agent trajectories