Key findings
- The test runs inside Aider. The model edits stub files through Aider's edit format, and the exercise's unit tests run. After a failure, the model sees the first 50 lines of test errors and gets one more attempt.
- The top score is 88.0% for GPT-5 with high reasoning effort, run on August 23, 2025 with a development build of Aider 0.86.2, at a benchmark cost of US$29.08.
- Cost per run varies far more than score: DeepSeek-V3.2-Exp scored 74.2% for US$1.30, and o3-pro scored 84.9% for US$146.32.
- The leaderboard holds 69 results and was last updated on November 20, 2025, with no model released after October 2025.
- An open GitHub issue reports that a model's own files can set the test command's exit code, which the benchmark uses to mark an exercise solved.
Aider's polyglot leaderboard is the best public record of how 2025 models edit code inside Aider, and its newest run is dated October 3, 2025. Use it to compare 2025 models and edit formats, and test any newer model on your own tasks. Each model works through Aider on 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust, with one retry after failing tests.
What the benchmark measures
Aider introduced the polyglot benchmark on December 21, 2024 to replace its Python benchmark, which top models had saturated with scores above 80% on 133 exercises. The new set takes the 225 hardest of 697 Exercism problems in six languages, chosen so that the top models of late 2024 spread between about 5% and 50%.
Each result records the model, the edit format, the Aider version, the run date, the cost at that day's prices, and the share of responses that used the edit format correctly. Percent correct is the share of exercises whose tests pass after the second attempt.
The top results
Benchmark cost is what one full run cost at the time, and Aider's notes say providers change prices. Read it as a relative measure between models run in the same period.
| Model | Run date | Benchmark cost | Edit format | Percent correct |
|---|---|---|---|---|
| GPT-5 (high) | Aug 23, 2025 | US$29.08 | diff | |
| GPT-5 (medium) | Aug 25, 2025 | US$17.69 | diff | |
| o3-pro (high) | Jun 28, 2025 | US$146.32 | diff | |
| Gemini 2.5 Pro 06-05 (32k thinking) | Jun 6, 2025 | US$49.88 | diff-fenced | |
| o3 (high) | Jun 25, 2025 | US$21.23 | diff | |
| Grok 4 (high) | Jul 11, 2025 | US$59.62 | diff | |
| DeepSeek-V3.2-Exp (Reasoner) | Oct 3, 2025 | US$1.30 | diff | |
| Claude Opus 4 (32k thinking) | May 25, 2025 | US$65.75 | diff | |
| Claude Sonnet 4 (32k thinking) | May 24, 2025 | US$26.58 | diff | |
| Kimi K2 | Jul 17, 2025 | US$1.24 | diff | |
| gpt-oss-120b (high) | Aug 6, 2025 | US$0.74 | diff |
Aider LLM Leaderboards, polyglot benchmark, 225 exercises per run; page last updated November 20, 2025; checked October 12, 2026
Why it no longer settles a 2026 model choice
The newest results are DeepSeek-V3.2-Exp runs dated October 3, 2025. Claude appears up to Opus 4 and Sonnet 4 from May 2025, and Gemini up to 2.5, so the models most teams run in 2026 are absent. Aider's main branch added settings for GPT-5.5 and Claude Opus 4.7 in April 2026 without new benchmark runs.
Agent builders now run the suite as their own test. One team reports 100% on the 148 exercises it had run in four languages with Claude Opus 4.8. Its agent reads files and runs tests, which is a different loop from the leaderboard's. A practitioner running Aider on a local 32B Qwen model reports a 20% pass rate on multi-step tasks.
- Use the leaderboard to compare 2025 models and edit formats inside Aider.
- Run your own tasks for any model released after October 2025.
- Check the share of well-formed edits beside the score. A model that breaks the edit format spends tokens on retries.
What to watch
A GitHub issue opened on September 5, 2026 reports that the benchmark scores an exercise from the exit code of a test command. That command runs in a directory the model can write to. A planted test-runner config file can therefore make failing tests count as solved. Aider restores the test files before each attempt, and the report says it leaves runner config files in place. Until the issue is fixed, treat polyglot scores that third parties publish for themselves like any unaudited benchmark.
Methodology
GILD read the Aider LLM Leaderboards page, its benchmark notes, and the December 21, 2024 polyglot announcement on October 12, 2026. It also read the July 2, 2023 post that describes the two-attempt test loop that day. Every figure is the leaderboard's own, as published: run dates, costs, edit formats, and versions are its fields. The GitHub issues about the test loop and about third-party submissions are their authors' reports, linked in Sources.
Products in this piece
Sources
- Official documentation
- 6
- Public source
- 3
- Aider LLM Leaderboards (last updated Nov 20, 2025)
- Aider LLM Leaderboards: Benchmark notes
- Aider blog: o1 tops aider's new polyglot leaderboard (Dec 21, 2024)
- Aider blog: GPT code editing benchmarks (Jul 2, 2023)
- Aider docs: Edit formats
- Aider release history
- GitHub issue #5681: the benchmark scores an exercise from a test exit code the model's files control (Sep 5, 2026; not a Signals source)
- GitHub issue #5321: an agent team's polyglot submission question (Jun 24, 2026; not a Signals source)
- Hacker News, ebercruzdev: a 20% pass rate for Aider on a local Qwen 32B (Mar 16, 2026)