Key findings
- On 45 SWE-bench-verified-mini cases run ten times each, Vibe on Devstral 2 passed 37.6% of runs and Claude Code 39.8%, a gap inside the 95% confidence intervals. Vibe averaged 296 seconds per run against 357.
- With the same GLM-4.7 model on Terminal-Bench 2.0, Vibe scored 0.35, Claude Code 0.29, Gemini CLI 0.23, OpenCode 0.21, and Codex 0.15. The harness alone therefore moved the score by more than 2x.
- SecMate found that Vibe 2.25.0 approved shell commands from a reduced reading of the command and then executed the full string. Mistral released 2.25.4 on Sep 12, 2026 after reports from two research teams.
What do independent measurements say about Mistral Vibe? Two public benchmarks and one security review tested Vibe directly. They put Vibe within statistical error of Claude Code on one bug-fix set and show its harness scoring highest when every agent ran the same model. The security review documents a permission check that once let read-only commands execute code.
What the bug-fix benchmark settles
The study ran Claude Code and Vibe on the same 45 Django and Sphinx issues, ten runs per case per assistant, 900 runs in all, with hidden tests deciding each pass. Claude Code ran on automatic model selection, usually Opus; Vibe ran Devstral 2, its model at the time.
The two means sit inside each other's confidence intervals, and each agent passed cases the other never did. About four cases in ten flipped between pass and fail across reruns for both agents, so one run on one task says little about either. Mistral has since replaced Devstral 2 in Vibe with Mistral Medium 3.5. The independent Vibe studies here use the older models; validate the current model in your own pilot.
| Agent and model | Pass rate | Pass rate |
|---|---|---|
| Claude Code, automatic model selection | ||
| Mistral Vibe, Devstral 2 |
Kvit blog, Dec 17, 2025
What the same-model benchmark settles
One engineer forked Codex, Gemini CLI, and Vibe to run GLM-4.7 and ran OpenCode and Claude Code on the same model, averaging five runs over 89 Terminal-Bench 2.0 tasks. Vibe placed first. The author credits forgiving edit tools and clean provider adapters, and reports the ranking held when he repeated the runs on GLM-5.
The result separates the harness from the model. Users who judge Vibe on Mistral's hosted models judge both at once, and on this test Vibe's harness scored highest.
| Agent | Score | Share of tasks passed |
|---|---|---|
| Mistral Vibe | ||
| Claude Code | ||
| Gemini CLI | ||
| OpenCode | ||
| Codex |
Charles Azam, Feb 10, 2026
What the security review settles
SecMate reviewed Vibe 2.25.0 and reported two flaws on Sep 5, 2026. Shell syntax the permission parser ignored could run arbitrary code. Redirects and expansions could read or write outside the workspace. A command that looked like an allowlisted read ran without an approval prompt. Mistral shipped 2.25.4 on Sep 12.
For a buyer, the point is where approval logic sits. Mistral's own documentation calls smart-approve a classifier with no sandbox and keeps auto-approve for disposable environments. Run Vibe on untrusted repositories only inside a container or VM, whatever approval mode is set.
What to validate in your pilot
GILD checked Mistral's model release and the SWE-bench and Terminal-Bench results on Oct 26, 2026; the independent Vibe runs found use older models. Test Mistral Medium 3.5, Mistral Large 4, and the GLM models Mistral now hosts on your own code. Count the pull requests reviewers accept; the published runs do not measure that outcome for your team. Run your pilot on your own repositories and count the changes reviewers accept.
Methodology
GILD read each study's published method and figures on Oct 25, 2026 and reports them as their authors state them; GILD did not rerun them. The bug-fix study and the same-model study each come from a single author, and the security review comes from the firm that found the flaws. Mistral's model and release facts come from its own pages, checked the same day.
Products in this piece
Sources
- Official documentation
- 5
- Public source
- 3
- Mistral: Introducing Devstral 2 and Mistral Vibe CLI (Dec 9, 2025)
- Mistral: Remote agents in Vibe, powered by Mistral Medium 3.5 (May 22, 2026)
- Mistral docs: Configuration
- Mistral docs: Safety, approvals, and permissions
- Mistral Vibe releases on GitHub
- Kvit blog: Run-to-run variance in coding assistants, Claude Code and Vibe on 45 bug fixes (Dec 17, 2025)
- Charles Azam: I forked 4 CLI coding agents to run the same model (Feb 10, 2026)
- SecMate blog: Mistral Vibe shell permission bypass leading to arbitrary code execution (Sep 23, 2026)