Key findings
- Factory reported Droid at 58.8% on Terminal-Bench 1.0 in September 2025 with Claude Opus 4.1, against 43.2% for Claude Code on the same model and 42.8% for Codex CLI with GPT-5. Droid placed first that month and fifth by August 2026.
- The 63.1% on Factory's docs is Droid with Claude Opus 4.5 on Terminal-Bench 2.0. The page sets it beside Claude Code at 40.1% on the older Claude Sonnet 4.5, so the gap mixes agent and model.
- Factory reports that Factory Router cut aggregate production cost by 63% against frontier-model rates and reached 99% of Claude Opus 4.7's pass rate on Terminal-Bench 2. Both outside write-ups on the Router say they did not reproduce these figures.
- Outside tests split by model. In the OctoBench study, Droid on Claude Opus 4.5 followed every instruction in more tasks than Claude Code, and on MiniMax-M2.1 it followed fewer. One YouTube reviewer saw three of four projects fail on Sonnet 4.5.
Treat every Droid benchmark score as Factory's own until your repositories confirm it. Factory submitted each Droid entry on the public Terminal-Bench leaderboards itself, and no Terminal-Bench team member reran any of them. Run Droid against your current agent on 10 to 20 of your own closed tickets before you budget on a figure below.
What Factory reported on Terminal-Bench 1.0
Factory published Droid's Terminal-Bench 1.0 results on September 25, 2025 and submitted five runs per model to the public leaderboard. Its best configuration used Claude Opus 4.1 without extended thinking. Factory argues that agent design moved scores more than model choice, because Droid beat the labs' own agents on the same models.
Factory ran these with its own prompts and timeouts, and says it tuned Droid for speed because many tasks have aggressive timeouts. The leaderboard marks a run verified when "A Terminal-Bench team member ran the evaluation and verified the results." No Droid run on the board carries that mark.
| Agent and model | Task success |
|---|---|
| Droid with Claude Opus 4.1 | |
| Droid with GPT-5 (medium reasoning) | |
| Droid with Claude Sonnet 4 | |
| Claude Code with Claude Opus 4.1 | |
| Codex CLI with GPT-5 |
Factory, Terminal-Bench 1.0 results published Sep 25, 2025, five runs per model, vendor-reported
The docs page figure and Factory's model rankings
Factory's Terminal Bench docs page, last updated December 2025, lists Droid at 63.1%, Codex CLI at 60.4%, and Claude Code at 40.1%, with no model or benchmark version named. Each figure matches a Terminal-Bench 2.0 leaderboard entry. Droid ran Claude Opus 4.5, Codex CLI ran GPT-5.1-Codex-Max, and Claude Code ran Claude Sonnet 4.5. A Terminal-Bench team member verified the Codex CLI and Claude Code runs. Factory submitted the Droid run itself.
Factory's other benchmarks rank models inside Droid. Legacy-Bench, published September 24, 2026, tests debugging, implementation, and migration in COBOL, Java 7, BASIC, C89, Fortran, and Assembly. The code review benchmark, published April 29, 2026, scored 13 models on 50 pull requests from five open source projects against a curated set of known bugs. Use both to pick a model within Droid. Neither compares Droid with another agent.
Factory Router savings and the Comarch figures
For the week ending September 13, 2026, Factory reports 63% lower aggregate session cost on Factory Router than the same work would cost at published frontier-model rates. The median session saving was 72.5%. Your baseline is what your team spends today, so compare the saving with your own Analytics data before you budget on it. Developers Digest in June 2026 and Kingy AI in October 2026 both wrote up the Router claims, and both state that they did not reproduce Factory's figures.
Factory's Comarch case study reports a 40% efficiency gain among Comarch engineers and an XSLT modernization that Comarch estimated at 10,000 man-days, finished in 45 days. Those figures appear only on Factory's pages. Comarch's own English press pages describe its AI-first program without naming Factory, and its public GitHub ships a plugin for Droid, which shows Comarch engineers use it. Ask Factory for a reference call with Comarch before you plan around the figures.
What outside testers found
OctoBench, a January 2026 arXiv study by Deming Ding and co-authors, ran Droid v0.42.2, Claude Code, and Kilo on the same models. It scored whether each followed every instruction in a coding task. With Claude Opus 4.5, Droid did so in 40.17% of tasks and Claude Code in 28.39%. With MiniMax-M2.1, Droid trailed Claude Code, 15.38% to 18.60%. OctoBench scores instruction following, which Terminal-Bench does not.
AICodeKing ran four projects with Droid on Sonnet 4.5 in October 2025. Three failed, and the fourth used a library without installing it. Switching to GLM-4.6 through a custom provider improved three of the four, and AICodeKing concluded that Factory's claims did not match those tests.
Other builders report the opposite: a migration Claude Code kept failing that Droid finished in one pass on the same model, and Spec Mode plans they rate above other agents' plan modes.
How to test the claims yourself
Run Droid and your current agent from the same repository state, with the same prompts and the same model where both support it. Compare the final diff, test results, elapsed time, reviewer interventions, and total cost. Then repeat with the cheaper models Factory Router would choose.
- Pick 10 to 20 closed tickets with tests that prove the fix.
- Fix the model per run so the agent is the only variable, then add one routed run.
- Record reviewer time per change as well as pass or fail.
- Price each run from the Analytics API or the session's recorded usage.
Methodology
GILD read Factory's benchmark and product posts and its docs on October 12, 2026. On October 25, 2026 it matched Factory's figures against the Terminal-Bench 1.0 and 2.0 leaderboards and read the OctoBench paper. It also checked Comarch's own press pages and GitHub, and read outside write-ups of Factory Router. Hands-on accounts come from the public posts and videos cited on GILD's Factory page. Every benchmark figure here is its publisher's own, linked in Sources, and every figure attributed to Factory appears as Factory published it.
Products in this piece
Sources
- Official documentation
- 8
- Public source
- 10
- Factory news: Droid, the #1 Software Development Agent on Terminal-Bench (Sep 25, 2025)
- Factory docs: Terminal Bench results
- Factory news: Which model is best for legacy code? (Sep 24, 2026)
- Factory news: Which Model Reviews Code Best? (Apr 29, 2026)
- Factory news: Factory Router cuts inference costs by 63% (Sep 23, 2026)
- Factory docs: Factory Router
- Factory customer story: How Comarch Is Building Its Own Software Factory (Jul 29, 2026)
- Factory docs: Available Models
- AICodeKing (YouTube): Droid CLI + GLM-4.6 + FREE 4.5 Sonnet, GPT-5: Is it really the NEXT CLAUDE CODE? (Oct 3, 2025)
- Every: Vibe Check: I Canceled Two AI Max Plans for Factory's Coding Agent Droid (Oct 30, 2025)
- Sid Bharath: Factory Droid Tutorial: Missions, Skills, and Agents (Sep 30, 2025)
- Terminal-Bench: Terminal-Bench 1.0 leaderboard (Laude Institute, Oct 25, 2026)
- Terminal-Bench: Terminal-Bench 2.0 leaderboard (Laude Institute, Oct 25, 2026)
- arXiv: OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding (Deming Ding et al., Jan 2026)
- Developers Digest: Factory Router, Explained: How Automatic Model Routing Cuts Coding-Agent Spend 20-25% (Jun 17, 2026)
- Kingy AI: Factory Analytics Adds Agent Cost Tracking: How to Read the Savings Claims (Oct 4, 2026)
- Comarch: Comarch at the Data Science Summit AI Edition: How We Are Driving the AI-First Transformation (Jul 3, 2026)
- Comarch on GitHub: git-byline README (viewed Oct 25, 2026)