Cost
$0.00per task
Cost per task. One Intelligence Index task at list price, every token counted. Run at reasoning on.
Model family by Google
Gemma
Google's open-weight Gemma models, built from the technology behind Gemini.
Where it sits in Google's lineup
- Gemini ProGoogle's Gemini family for complex reasoning and multimodal work.Gemini 4 Argon
- Gemini FlashGoogle's Gemini family balancing capability, speed, and cost.Gemini 3.8 Flash
- GemmaGoogle's open-weight Gemma models, built from the technology behind Gemini.This model
- Made by
- Releases
- 1 tracked
- Current release
- Gemma 4 31B
Strengths and weaknesses
Compare with other models Up to 3 beside Gemma 4 31B
- Gemma 4 31B
Suggested comparisons
The models a buyer weighs against Gemma 4 31B, picked on list price and on Artificial Analysis's Intelligence Index of overall quality. Add one to draw it beside Gemma 4 31B.
Gemma 4 31B
Among the 40 current releases on this chart, Gemma 4 31B ranks 6th of 25 on Answers in full. Science is Artificial Analysis's run on GPQA Diamond, Factuality is Epoch AI's run on SimpleQA Verified and Instruction following is Artificial Analysis's run on IFBench, marked on the chart. It ranks lowest on Tool use, 37th of 38 and Long context, 36th of 38. Agentic coding, Math and Computer use: not yet published by Google or any independent evaluator, checked Nov 2, 2026.
Each axis is one public benchmark at one setting, on its own fixed 0 to 100 scale, so adding a model never moves another. A dotted bridge crosses a missing result, which is never a zero. A hollow mark is a result from another source: the maker's own figure or another lab's run, where the benchmark's usual source has not tested the model yet. The readout names it.
On the chart
- Gemma 4 31BThis pageResults on 9 of 12 axes$0.14 / $0.40On Amazon Bedrock; the maker sells no API access36 tokens/sFirst token in 1.1 s
Prices and speeds as of Oct 28, 2026.
Gemma 4 31B on each axis9 of 12 with a result
The share of Terminal-Bench 4.0's long terminal tasks the model finishes on its own: services, kernels, proofs and data pipelines, graded on the final artifact with no partial credit.
Terminal-Bench 4.0, pass@1 averaged over 3 runs, mini-swe-agent harness
- Gemma 4 31BNot yet published by Google or any independent evaluator, checked Nov 2, 2026
The share of SciCode's scientific programming subproblems the model's code solves on the first attempt.
SciCode, subproblem accuracy, pass@1, highest effort run
- Gemma 4 31B45.5%
The share of FrontierMath's held-out research-grade problems (tiers 1 to 3) the model answers correctly.
FrontierMath Tiers 1-3, accuracy, pass@1, highest effort run
- Gemma 4 31BNot yet published by Google or any independent evaluator, checked Nov 2, 2026
The share of GPQA Diamond's graduate-level biology, chemistry and physics questions the model answers correctly.
GPQA Diamond, accuracy, pass@1, highest effort run. Where it has no run of a model, the maker's own figure or another lab's run stands in, marked with its benchmark.
- Gemma 4 31B85.7%Artificial Analysis's runBenchmark: GPQA Diamond, accuracy, pass@1, Artificial Analysis run, where the axis reads highest effort run. From another source, so it is drawn and marked but not ranked with the others.Artificial Analysis's run of the reasoning version. Epoch AI, the axis's source, ran it only at minimal effort, which is not drawn.
The share of Humanity's Last Exam's 2,158 text-only questions, expert-written at the edge of human knowledge, the model answers correctly without tools.
Humanity's Last Exam, accuracy, text-only questions, no tools
- Gemma 4 31B23.6%
The share of each AutomationBench task's objectives the model completes by calling REST APIs across simulated business apps (Gmail, Sheets, Slack, Salesforce, Zendesk, Jira, HubSpot), averaged over tasks. A task that breaks one of its policy guardrails scores zero.
AutomationBench-AA, score, Artificial Analysis run
- Gemma 4 31B4.9%Unusual next to its other results. Artificial Analysis publishes 4.9 for the reasoning version of Gemma 4 31B on this test, far below its other results. Artificial Analysis run of Oct 28, 2026.
How much of each OSWorld 2.1 workflow the model completes by operating a real Ubuntu desktop through screenshots, mouse and keyboard: all 108 long tasks across apps, files, mail and browsers, on the September 2026 v2.1 release of the task files. A point is the mean share of each task's weighted checks the model passes (the partial score, as a percentage). The OSWorld team's own runs come first. Where the team has not run a release, the point is another lab's run or the maker's own published figure, marked with who ran it and the benchmark it ran: OSWorld 2.1 in the maker's harness, OSWorld 2.0's offline set or the earlier OSWorld-Verified. Those points are drawn so the release has a figure, and they are not ranked against the team's runs.
OSWorld 2.1, partial score, full 108-task set, v2.1 release
- Gemma 4 31BNot yet published by Google or any independent evaluator, checked Nov 2, 2026
The share of AA-LCR's questions the model answers correctly by reasoning across long sets of real documents, about 100K tokens each.
AA-LCR v1.1, accuracy, Artificial Analysis run
- Gemma 4 31B69.7%
The share of MMMU-Pro's college-level questions over charts, diagrams and photos the model answers correctly.
MMMU-Pro, overall accuracy, no tools
- Gemma 4 31B73.4%
The share of AA-Omniscience's 6,000 knowledge questions across 42 economically relevant topics the model answers correctly from memory, without search.
AA-Omniscience Accuracy, accuracy, Artificial Analysis run. Where it has no run of a model, the maker's own figure or another lab's run stands in, marked with its benchmark.
- Gemma 4 31B10.4%Epoch AI's runBenchmark: SimpleQA Verified, mean score, Epoch AI run. From another source, so it is drawn and marked but not ranked with the others.Epoch AI ran SimpleQA Verified on the instruction-tuned release. SimpleQA Verified prints no hallucination rate, so none is shown beside it.
The share of SpeechMap's 2,120 sensitive and controversial but legal requests the model answers in full: arguments, essays and explanations on contested themes in politics, religion, race, sex, history and similar topics, four wordings per theme across 530 themes. SpeechMap's judge rates each response complete, evasive (hedged or redirected), a denial, or an error (the provider's API blocked the request). Only complete answers count, so the point is the share answered in full: 100 minus the share hedged, refused or blocked by the provider. Higher means more lawful requests answered in full. The share blocked at the provider is printed apart in the table, because an API error is not the model declining. Where SpeechMap runs a release both with reasoning off and with reasoning on (a -reasoning or -thinking entry), the reasoning-on entry is used; a release with one entry uses that entry.
SpeechMap, % Complete over 2,120 contested requests, reasoning-on entry
- Gemma 4 31B82.3%SpeechMap counted 16.8% of requests denied and 0.9% evaded.
The share of MultiChallenge's multi-turn conversations in which the model keeps every instruction it was given earlier in the conversation.
MultiChallenge, accuracy, Scale AI run. Where it has no run of a model, the maker's own figure or another lab's run stands in, marked with its benchmark.
- Gemma 4 31B75.6%Artificial Analysis's runBenchmark: IFBench, accuracy, Artificial Analysis run. From another source, so it is drawn and marked but not ranked with the others.Artificial Analysis's run of the reasoning version on IFBench, a test of precise output constraints. Scale AI has not run it on MultiChallenge.
Cost, speed, capacity, quality and accuracy for Gemma 4 31B, with where each figure sits among the 40 models GILD tracks. These measures stand beside the chart and never feed it.
Speed
35tokens a second
Output speed. How fast it writes once the answer starts. Run at reasoning on.
- First token
- 1.1 s
- Whole answer
- 64 s
Capacity
256Ktokens
Context window. The most one request can hold, as the maker states it.
- Max output
- 16K
- Reads
- Text, Images, Video
Quality
14.7of 100
Intelligence Index. One composite score across independent evaluations. Run at reasoning on.
- GDPval knowledge work
- 627 Elo
- Terminal-Bench 4.0
- 0.0%
Hallucination
85.0%wrong when it should say “I don’t know”
Hallucination rate. Of the questions it can't get right, how often it answers anyway. Lower is better. Run at reasoning on.
- Knowledge accuracy
- 20.0%
Where you can get it
3of 19 routes
Routes offering it. Of the clouds, gateways and coding products GILD tracks, those offering it today.
- Open weights
- Yes
- Licence
- Apache 2.0
Figures from Artificial Analysis, OpenRouter, GILD's route matrix and Google AI for Developers, read Oct 27 to 29, 2026. Each measure's definition and source open under . Not yet published for Gemma 4 31B: Cached input and Epoch Capabilities Index.
Every value with its benchmark, run and source
| Axis | Benchmark and setting | Gemma 4 31B | Run | Source | Date |
|---|---|---|---|---|---|
| Agentic coding | Terminal-Bench 4.0Pass@1 averaged over 3 runs, mini-swe-agent harness | Not yet published by Google or any independent evaluator, checked Nov 2, 2026 | |||
| Code | SciCodeSubproblem accuracy, pass@1, highest effort run | 45.5% | Gemma 4 31B (Reasoning)Effort: reasoning on. Harness: none. | Artificial Analysis | Oct 28, 2026 |
| Math | FrontierMath Tiers 1-3Accuracy, pass@1, highest effort run | Not yet published by Google or any independent evaluator, checked Nov 2, 2026 | |||
| Science | GPQA DiamondAccuracy, pass@1, Artificial Analysis run. Artificial Analysis's run of the reasoning version. Epoch AI, the axis's source, ran it only at minimal effort, which is not drawn.Stands in for GPQA Diamond, accuracy, pass@1, highest effort run, which has no run of Gemma 4 31B yet. | 85.7% | Gemma 4 31B (Reasoning)Effort: reasoning on. Harness: none.Not drawn: Google 84.3% in its maker's own harness, Apr 2, 2026. Source | Artificial AnalysisRun by Artificial Analysis | Oct 28, 2026 |
| Hard reasoning | Humanity's Last ExamAccuracy, text-only questions, no tools | 23.6% | Gemma 4 31B (Reasoning)Effort: reasoning on. Harness: none. | Artificial Analysis | Oct 28, 2026 |
| Tool use | AutomationBench-AAScore, Artificial Analysis run. Artificial Analysis publishes 4.9 for the reasoning version of Gemma 4 31B on this test, far below its other results. | 4.9%Unusual next to its other results. Artificial Analysis publishes 4.9 for the reasoning version of Gemma 4 31B on this test, far below its other results. Artificial Analysis run of Oct 28, 2026. | Gemma 4 31B (Reasoning)Effort: reasoning on. Harness: Artificial Analysis AutomationBench harness. | Artificial Analysis | Oct 28, 2026 |
| Computer use | OSWorld 2.1Partial score, full 108-task set, v2.1 release | Not yet published by Google or any independent evaluator, checked Nov 2, 2026 | |||
| Long context | AA-LCR v1.1Accuracy, Artificial Analysis run | 69.7% | Gemma 4 31B (Reasoning)Effort: reasoning on. Harness: none. | Artificial Analysis | Oct 28, 2026 |
| Multimodal | MMMU-ProOverall accuracy, no tools | 73.4% | Gemma 4 31B (Reasoning)Effort: reasoning on. Harness: none. | Artificial Analysis | Oct 28, 2026 |
| Factuality | SimpleQA VerifiedMean score, Epoch AI run. Epoch AI ran SimpleQA Verified on the instruction-tuned release. SimpleQA Verified prints no hallucination rate, so none is shown beside it.Stands in for AA-Omniscience Accuracy, accuracy, Artificial Analysis run, which has no run of Gemma 4 31B yet. | 10.4% | Effort: not published. Harness: none. | Epoch AIRun by Epoch AI | Aug 27, 2026 |
| Answers in full | SpeechMap% Complete over 2,120 contested requests, reasoning-on entry. SpeechMap counted 16.8% of requests denied and 0.9% evaded. | 82.3% | Effort: reasoning on. Harness: none. | SpeechMap.AI | Oct 28, 2026 |
| Instruction following | IFBenchAccuracy, Artificial Analysis run. Artificial Analysis's run of the reasoning version on IFBench, a test of precise output constraints. Scale AI has not run it on MultiChallenge.Stands in for MultiChallenge, accuracy, Scale AI run, which has no run of Gemma 4 31B yet. | 75.6% | Gemma 4 31B (Reasoning)Effort: reasoning on. Harness: none. | Artificial AnalysisRun by Artificial Analysis | Oct 28, 2026 |
Price and speed
- Google sells no API access to Gemma 4 31B. Among Amazon Bedrock, Google Vertex AI and Microsoft Foundry, the lowest list price is on Amazon Bedrock: $0.14 per million input tokens and $0.40 per million output tokens (US East (N. Virginia), US East (Ohio) and US West (Oregon), on-demand). Every host's price is in Routes. Amazon Web Services pricing, Oct 29, 2026.
- Gemma 4 31B writes 35.5 output tokens per second on Google's API, and its first token arrives 1.1 s after the request. Artificial Analysis, Oct 28, 2026.
- Fastest on the chart is the shortest time to a 500-token answer: time to first token plus 500 tokens at the median output speed.
Every model with a result, axis by axis
57 models. A maker's own figure and another lab's run are marked with their benchmark; a missing result reads "Not yet published", never zero.
| Model | Agentic coding | Code | Math | Science | Hard reasoning | Tool use | Computer use | Long context | Multimodal | Factuality | Answers in full | Instructions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B | Not yet published | 45.5% | Not yet published | 85.7%Run by Artificial AnalysisGPQA Diamond | 23.6% | 4.9% | Not yet published | 69.7% | 73.4% | 10.4%Run by Epoch AISimpleQA Verified | 82.3% | 75.6%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Gemini 4 Argon | 57.6% | 61.8% | 86% | 96.4% | 57.1% | 77.5% | 69.2%Reported by GoogleOSWorld 2.0 | 79.7% | Not yet published | 48.3% | 87% | 78.7% |
| ||||||||||||
| Gemini 3.8 Flash | 19.7% | 56.6% | 68.4% | 95.4% | 47.8% | 59.9% | 59%Reported by GoogleOSWorld 2.0 | 81.3% | 85.6% | 50.5% | 86.1% | Not yet published |
| ||||||||||||
| Gemini 3.1 Pro Preview | 2.5% | 58.7% | 59.6% | 94.4% | 47% | 35.4% | Not yet published | 82% | 82.4% | 54.9% | 59.9% | 71.4% |
| ||||||||||||
| Gemini 3.6 Flash | 7.1%Run by Artificial AnalysisTerminal-Bench 4.0 | 53.4% | 58.9% | 94.1% | 40.8% | 53% | 83%Reported by GoogleOSWorld-Verified | 80% | 83.2% | 66.2%Run by Epoch AISimpleQA Verified | 87.2% | Not yet published |
| ||||||||||||
| Claude Haiku 5.5 | 36.4% | 55% | 75.1% | 89.6% | 44.4% | 35.4% | 72.4%Reported by AnthropicOSWorld 2.1 | 82.7% | Not yet published | 34.7% | 78.8% | 67.7% |
| ||||||||||||
| Claude Sonnet 5.5 | 64.1% | 61% | 88.8% | 95.6% | 55% | 71.8% | 80.1%Reported by Anthropic | 82.7% | Not yet published | 51.5% | 81.6% | Not yet published |
| ||||||||||||
| Claude Opus 5.5 | 65.2% | 66.9% | 91.2% | 90.6% | 61.4% | 69.5% | 81.8%Reported by Anthropic | 84.7% | 87.7% | 65.5% | 82.8% | Not yet published |
| ||||||||||||
| Claude Fable 5.1 | 58.1% | 63.1% | 90.2% | 93.4%Run by Vals.ai | 59.1% | 59.4% | 80.7%Reported by Anthropic | 85.3% | 86.3%Run by Moonshot AI | 68.4% | 72.4% | Not yet published |
| ||||||||||||
| Claude Opus 5 | 53.5% | 56.4% | 85.6% | 93.9% | 54.9% | 56.6% | 77.7% | 79.3% | 84.7% | 60.9% | 73.7% | Not yet published |
| ||||||||||||
| Claude Sonnet 5 | 10.6% | 54.3% | 65.6% | 80.3% | 41.3% | 36.5% | 57%Reported by Anthropic | 82% | 77.3% | 40.1% | Not yet published | Not yet published |
| ||||||||||||
| Claude Fable 5 | 41.4% | 61% | 87% | 85.9% | 55.5% | 54.1% | Not yet published | 82.3% | Not yet published | 65.4% | 64.8% | 63.5%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Claude Opus 4.8 | 23.2% | 54.4% | 80% | 91% | 48.7% | 45.6% | 54.8%Run by OSWorld teamOSWorld 2.0 | 77.7% | Not yet published | 48.8% | 67.4% | 62.2%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Claude Opus 4.7 | Not yet published | 54.5% | 70.2% | 86.4% | 42.3% | Not yet published | 48.9%Run by OSWorld teamOSWorld 2.0 | 78.7% | 78.8% | 48.9% | 70.2% | 58.6%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Claude Sonnet 4.6 | 3%Run by Artificial AnalysisTerminal-Bench 4.0 | 50.1% | 71.1%Run by Epoch AIOTIS Mock AIME 2024-2025 | 78.8% | 33.6% | 20.1% | 41.5%Run by OSWorld teamOSWorld 2.0 | 80% | 73.3% | 40.9% | 35.8% | 56.6%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Claude Opus 4.6 | Not yet published | Not yet published | 66% | 88.4% | 39.9% | Not yet published | Not yet published | 78% | 75.4% | 47% | 59.9% | 37.2% |
| ||||||||||||
| Claude Opus 4.5 | Not yet published | Not yet published | 34.4% | 86% | 30.1% | Not yet published | Not yet published | 77.3% | 74% | 46.6% | 57.5% | 59% |
| ||||||||||||
| Claude Haiku 4.5 | Not yet published | 42.2% | 66.7%Run by Epoch AIOTIS Mock AIME 2024-2025 | 71.2% | 10.4% | 3.2% | Not yet published | 74.3% | 58.6% | 18% | 24.7% | 50.5% |
| ||||||||||||
| Claude Sonnet 4.5 | Not yet published | 45.7% | 23.9% | 82.3% | 17.8% | 14.1% | Not yet published | 72.3% | 68.7% | 32.9% | 31.4% | 55.3% |
| ||||||||||||
| GPT-6.1 Terra | 49.5% | 55.1% | 90.9% | 94.3% | 46.2% | 60.2% | 74.2%Reported by OpenAIOSWorld 2.1 | 82.7% | 84.7% | 47.6% | Not yet published | 74.2% |
| ||||||||||||
| GPT-6.1 Sol | 55.1% | 54.2% | 93.7% | 95.4% | 52.9% | 64.9% | 78%Reported by OpenAIOSWorld 2.1 | 83% | 86% | 61.8% | 69.5% | 82% |
| ||||||||||||
| GPT-6 Luna | 13.6% | 54.6% | 78.9% | 90.5% | 38.5% | 53.2% | 48.9%Run by AnthropicOSWorld 2.1 | 83.3% | 79.7% | 39.6% | 63.4% | Not yet published |
| ||||||||||||
| GPT-6 Astra | 59.6% | 56.5% | 93.7% | 95.8% | 54.7% | 68.5% | 75.8%Run by Meta | 80.7% | 86.9% | 61.6% | 68.2% | Not yet published |
| ||||||||||||
| GPT-5.6 Sol | 37.9% | 57.1% | 89.1% | 93.5% | 49.5% | 60.1% | 62.7%Run by OSWorld teamOSWorld 2.0 | 84% | 83.4% | 69.7%Run by Epoch AISimpleQA Verified | Not yet published | 72.7%Run by Artificial AnalysisIFBench |
| ||||||||||||
| GPT-5.6 Terra | 22.7% | 55% | 86% | 93.3% | 42.9% | 59.6% | Not yet published | 83% | 80.7% | 42.3% | Not yet published | 71.2%Run by Artificial AnalysisIFBench |
| ||||||||||||
| GPT-5.6 Luna | 11.6% | 53.6% | 82.1% | 91.6% | 39.5% | 50.2% | Not yet published | 83.7% | 78.6% | 41%Run by Epoch AISimpleQA Verified | Not yet published | Not yet published |
| ||||||||||||
| GPT-5.5 | 14.6%Run by Artificial AnalysisTerminal-Bench 4.0 | 55.8% | 85.3% | 93.5%Run by Artificial Analysis | 45.8% | 47.3% | 49.5%Run by OSWorld teamOSWorld 2.0 | 84.3% | 79.9% | 63%Run by Epoch AISimpleQA Verified | Not yet published | 75.9%Run by Artificial AnalysisIFBench |
| ||||||||||||
| GPT-5.4 | Not yet published | Not yet published | 78.6% | 93.3% | 43.7% | Not yet published | Not yet published | 82% | 78.4% | Not yet published | Not yet published | 73.9%Run by Artificial AnalysisIFBench |
| ||||||||||||
| gpt-oss-120b | 14.2%Run by Terminal-Bench (via Epoch AI)Terminal-Bench 2.0 | 34% | 88.9%Run by Epoch AIOTIS Mock AIME 2024-2025 | 75.8% | 19.6% | 0.2% | Not yet published | 52% | Not yet published | 16.8%Reported by OpenAISimpleQA | 36.3% | 45.3% |
| ||||||||||||
| GPT-6 Sol | 44.4% | 57.6% | 89.8% | 94.3% | 47.9% | 61.6% | 64.4%Reported by OpenAIOSWorld 2.0 | 83.7% | 82.9% | 54.5% | 64% | Not yet published |
| ||||||||||||
| GPT-5.3-Codex | Not yet published | Not yet published | Not yet published | 91.5%Run by Artificial Analysis | 42.5% | Not yet published | Not yet published | 83.3% | 78.5% | Not yet published | Not yet published | 75.4%Run by Artificial AnalysisIFBench |
| ||||||||||||
| DeepSeek V4.1 Flash | 31.8% | 55.9% | 71.2% | 93.2% | 43.3% | 66.6% | 69.3%Run by Moonshot AIOSWorld 2.1 | 82.7% | 80.6% | 39.6% | Not yet published | 74.9% |
| ||||||||||||
| DeepSeek V4 Pro | 14.1% | 51% | 64.6% | 91.7% | 41% | 56.7% | Not yet published | 80.3% | Not yet published | 45.2% | 72% | Not yet published |
| ||||||||||||
| Kimi K3.1 Pro | 63.6% | 60.8% | Not yet published | 94.2%Reported by Moonshot AI | 52.5% | 73.3% | 79.6%Reported by Moonshot AIOSWorld 2.1 | 86.3% | 83.9% | 48.9% | Not yet published | Not yet published |
| ||||||||||||
| Kimi K3 | 17.2% | 59.5% | 72.2% | 93.1% | 46.9% | 58.3% | Not yet published | 88.7% | 80.5% | 50.6%Run by Epoch AISimpleQA Verified | 65.9% | Not yet published |
| ||||||||||||
| Kimi K2.7 Code | 1%Run by Artificial AnalysisTerminal-Bench 4.0 | 47.8% | 54% | 87.9% | 35% | 24.5% | Not yet published | 79.3% | Not yet published | 36.5%Run by Epoch AISimpleQA Verified | Not yet published | 63.1%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Mistral Large 4 | 24.7%Run by Artificial AnalysisTerminal-Bench 4.0 | Not yet published | Not yet published | Not yet published | Not yet published | 57.8% | Not yet published | Not yet published | Not yet published | 26% | Not yet published | Not yet published |
| ||||||||||||
| Mistral Medium 3.5 | 0%Run by Artificial AnalysisTerminal-Bench 4.0 | 40.2% | Not yet published | 74.8%Run by Artificial AnalysisGPQA Diamond | 13.8% | 6.3% | Not yet published | 69.3% | 64.9% | 23% | 70.5% | 68.8%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Grok 4.7 | 28.8% | 57.4% | 53% | 92.7% | 43.1% | 65.6% | Not yet published | 76.7% | Not yet published | 42.7% | 64.1% | Not yet published |
| ||||||||||||
| Grok 4.6 | 17.2% | 56.5% | 66% | 93.2% | 42.9% | 66.7% | Not yet published | 80.3% | Not yet published | 48.9%Run by Epoch AISimpleQA Verified | 83.6% | Not yet published |
| ||||||||||||
| Grok 4.5 | 8.6% | 55% | 57.2% | 93.4% | 42.7% | 57.9% | Not yet published | 79.3% | 80.4% | 48.3%Run by Epoch AISimpleQA Verified | 93.1% | Not yet published |
| ||||||||||||
| MiniMax M3.1 Flash | 25.3% | 53.4% | Not yet published | Not yet published | 48% | 72.5% | 61.7%Reported by MiniMaxOSWorld 2.0 | 79.7% | 79.1% | 45.1% | Not yet published | Not yet published |
| ||||||||||||
| Mercury 3 | 3% | Not yet published | Not yet published | Not yet published | Not yet published | Not yet published | Not yet published | Not yet published | Not yet published | Not yet published | Not yet published | Not yet published |
| GLM-5.3-Flash | 37.4% | 51.6% | 55.8% | 90.2% | 39.9% | 60.4% | 59.1%Reported by Z.aiOSWorld 2.0 | 80% | 86%Run by Vals.aiMMMU-Pro | 28.2% | 65.4% | Not yet published |
| ||||||||||||
| GLM-5.3 | 38.9% | 59% | 68.8% | 90.9% | 42.3% | 62.2% | Not yet published | 79.7% | Not yet published | 32.3% | 71.5% | Not yet published |
| ||||||||||||
| GLM-5.2 | 1%Run by Artificial AnalysisTerminal-Bench 4.0 | 51.2% | 59.2% | Not yet published | 41.1% | 28.4% | Not yet published | 78.3% | Not yet published | 34.2%Run by Epoch AISimpleQA Verified | Not yet published | 73.3%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Muse Spark Open | 6.6% | 58.4% | 69.8% | 88.6% | 43.9% | 49.7% | 80.4%Reported by MetaOSWorld-Verified | 78.3% | 80.9% | 42% | Not yet published | 74.2% |
| ||||||||||||
| Muse Wonder | 49.5% | 60.3% | 82.1% | Not yet published | 50.1% | 68.1% | 72.6%Reported by Meta | 80.7% | Not yet published | 38% | 42.7% | Not yet published |
| ||||||||||||
| Muse Spark 1.3 | 10.6% | 58.8% | 74% | 93.5%Run by Artificial AnalysisGPQA Diamond | 48.7% | 57.9% | 69.6% | 83% | 82% | 38% | 25.8% | Not yet published |
| ||||||||||||
| Llama 4 Scout | Not yet published | 21.3% | 7.8%Run by Epoch AIOTIS Mock AIME 2024-2025 | 51.8% | 3.8% | 0.2% | Not yet published | 27.7% | 52.9% | Not yet published | 70.1% | 39.5%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Llama 4 Maverick | Not yet published | 31.7% | 20.6%Run by Epoch AIOTIS Mock AIME 2024-2025 | 67% | 4.9% | 0.2% | Not yet published | 50% | 62.1% | Not yet published | 58.3% | 43%Run by Artificial AnalysisIFBench |
| ||||||||||||
| Qwen3.8 27B | 5.6%Run by Artificial AnalysisTerminal-Bench 4.0 | 46.6% | Not yet published | 90.5%Run by Artificial AnalysisGPQA Diamond | 33.9% | 48.2% | 84.3%Reported by AlibabaOSWorld-Verified | 82% | 76.3% | 17.3% | Not yet published | 79.5%Reported by AlibabaIFBench |
| ||||||||||||
| Qwen3.8 Max | 38.9% | 52.1% | 65.6% | 92.3% | 43.1% | 56.2% | Not yet published | 80.3% | 82.8% | 29.5% | 65.3% | Not yet published |
| ||||||||||||
| MiMo-V2.6-Pro | 31.3% | 60.9% | Not yet published | Not yet published | 49.4% | 58.6% | 82%Reported by XiaomiOSWorld-Verified | 86.3% | Not yet published | 34.8% | 66.5% | Not yet published |
| ||||||||||||
| MiMo-V2.6-Flash | 24.2% | 51.3% | Not yet published | Not yet published | 35.1% | 64.1% | 80.8%Reported by XiaomiOSWorld-Verified | 74.3% | 73.1% | 27.2% | 66.9% | Not yet published |
| ||||||||||||
| Ling 3.1 Flash | 32.8%Run by Artificial AnalysisTerminal-Bench 4.0 | 56.8% | Not yet published | Not yet published | 43.5% | 67.5% | Not yet published | 82.3% | Not yet published | 26.5% | Not yet published | Not yet published |
| ||||||||||||
| Step 5 Preview | 31.8% | 58.9% | Not yet published | 93.5%Reported by StepFunGPQA Diamond | 46.5% | 51% | Not yet published | 88.3% | 76.4% | 36.4% | 59.7% | Not yet published |
| ||||||||||||
Agentic codingTerminal-Bench 4.0
- Claude Opus 5.565.2%
- Claude Sonnet 5.564.1%
- Kimi K3.1 Pro63.6%
- GPT-6 Astra59.6%
- Claude Fable 5.158.1%
- Gemini 4 Argon57.6%
- GPT-6.1 Sol55.1%
- Claude Opus 553.5%
- GPT-6.1 Terra49.5%
- Muse Wonder49.5%
- GPT-6 Sol44.4%
- Claude Fable 541.4%
- GLM-5.338.9%
- Qwen3.8 Max38.9%
- GPT-5.6 Sol37.9%
- GLM-5.3-Flash37.4%
- Claude Haiku 5.536.4%
- DeepSeek V4.1 Flash31.8%
- Step 5 Preview31.8%
- MiMo-V2.6-Pro31.3%
- Grok 4.728.8%
- MiniMax M3.1 Flash25.3%
- MiMo-V2.6-Flash24.2%
- Claude Opus 4.823.2%
- GPT-5.6 Terra22.7%
- Gemini 3.8 Flash19.7%
- Kimi K317.2%
- Grok 4.617.2%
- DeepSeek V4 Pro14.1%
- GPT-6 Luna13.6%
- GPT-5.6 Luna11.6%
- Claude Sonnet 510.6%
- Muse Spark 1.310.6%
- Grok 4.58.6%
- Muse Spark Open6.6%
- Mercury 33%
- Gemini 3.1 Pro Preview2.5%
- Ling 3.1 Flash32.8%Artificial Analysis's run, Terminal-Bench 4.0
- Mistral Large 424.7%Artificial Analysis's run, Terminal-Bench 4.0
- GPT-5.514.6%Artificial Analysis's run, Terminal-Bench 4.0
- gpt-oss-120b14.2%Terminal-Bench's run, Terminal-Bench 2.0
- Gemini 3.6 Flash7.1%Artificial Analysis's run, Terminal-Bench 4.0
- Qwen3.8 27B5.6%Artificial Analysis's run, Terminal-Bench 4.0
- Claude Sonnet 4.63%Artificial Analysis's run, Terminal-Bench 4.0
- Kimi K2.7 Code1%Artificial Analysis's run, Terminal-Bench 4.0
- GLM-5.21%Artificial Analysis's run, Terminal-Bench 4.0
- Mistral Medium 3.50%Artificial Analysis's run, Terminal-Bench 4.0
The other models in the grid have no result on Terminal-Bench 4.0 yet.
CodeSciCode
- Claude Opus 5.566.9%
- Claude Fable 5.163.1%
- Gemini 4 Argon61.8%
- Claude Sonnet 5.561%
- Claude Fable 561%
- MiMo-V2.6-Pro60.9%
- Kimi K3.1 Pro60.8%
- Muse Wonder60.3%
- Kimi K359.5%
- GLM-5.359%
- Step 5 Preview58.9%
- Muse Spark 1.358.8%
- Gemini 3.1 Pro Preview58.7%
- Muse Spark Open58.4%
- GPT-6 Sol57.6%
- Grok 4.757.4%
- GPT-5.6 Sol57.1%
- Ling 3.1 Flash56.8%
- Gemini 3.8 Flash56.6%
- GPT-6 Astra56.5%
- Grok 4.656.5%
- Claude Opus 556.4%
- DeepSeek V4.1 Flash55.9%
- GPT-5.555.8%
- GPT-6.1 Terra55.1%
- Claude Haiku 5.555%
- GPT-5.6 Terra55%
- Grok 4.555%
- GPT-6 Luna54.6%
- Claude Opus 4.754.5%
- Claude Opus 4.854.4%
- Claude Sonnet 554.3%
- GPT-6.1 Sol54.2%
- GPT-5.6 Luna53.6%
- Gemini 3.6 Flash53.4%
- MiniMax M3.1 Flash53.4%
- Qwen3.8 Max52.1%
- GLM-5.3-Flash51.6%
- MiMo-V2.6-Flash51.3%
- GLM-5.251.2%
- DeepSeek V4 Pro51%
- Claude Sonnet 4.650.1%
- Kimi K2.7 Code47.8%
- Qwen3.8 27B46.6%
- Claude Sonnet 4.545.7%
- Gemma 4 31B45.5%
- Claude Haiku 4.542.2%
- Mistral Medium 3.540.2%
- gpt-oss-120b34%
- Llama 4 Maverick31.7%
- Llama 4 Scout21.3%
The other models in the grid have no result on SciCode yet.
MathFrontierMath Tiers 1-3
- GPT-6.1 Sol93.7%
- GPT-6 Astra93.7%
- Claude Opus 5.591.2%
- GPT-6.1 Terra90.9%
- Claude Fable 5.190.2%
- GPT-6 Sol89.8%
- GPT-5.6 Sol89.1%
- Claude Sonnet 5.588.8%
- Claude Fable 587%
- Gemini 4 Argon86%
- GPT-5.6 Terra86%
- Claude Opus 585.6%
- GPT-5.585.3%
- GPT-5.6 Luna82.1%
- Muse Wonder82.1%
- Claude Opus 4.880%
- GPT-6 Luna78.9%
- GPT-5.478.6%
- Claude Haiku 5.575.1%
- Muse Spark 1.374%
- Kimi K372.2%
- DeepSeek V4.1 Flash71.2%
- Claude Opus 4.770.2%
- Muse Spark Open69.8%
- GLM-5.368.8%
- Gemini 3.8 Flash68.4%
- Claude Opus 4.666%
- Grok 4.666%
- Claude Sonnet 565.6%
- Qwen3.8 Max65.6%
- DeepSeek V4 Pro64.6%
- Gemini 3.1 Pro Preview59.6%
- GLM-5.259.2%
- Gemini 3.6 Flash58.9%
- Grok 4.557.2%
- GLM-5.3-Flash55.8%
- Kimi K2.7 Code54%
- Grok 4.753%
- Claude Opus 4.534.4%
- Claude Sonnet 4.523.9%
- gpt-oss-120b88.9%Epoch AI's run, OTIS Mock AIME 2024-2025
- Claude Sonnet 4.671.1%Epoch AI's run, OTIS Mock AIME 2024-2025
- Claude Haiku 4.566.7%Epoch AI's run, OTIS Mock AIME 2024-2025
- Llama 4 Maverick20.6%Epoch AI's run, OTIS Mock AIME 2024-2025
- Llama 4 Scout7.8%Epoch AI's run, OTIS Mock AIME 2024-2025
The other models in the grid have no result on FrontierMath Tiers 1-3 yet.
ScienceGPQA Diamond
- Gemini 4 Argon96.4%
- GPT-6 Astra95.8%
- Claude Sonnet 5.595.6%
- Gemini 3.8 Flash95.4%
- GPT-6.1 Sol95.4%
- Gemini 3.1 Pro Preview94.4%
- GPT-6.1 Terra94.3%
- GPT-6 Sol94.3%
- Gemini 3.6 Flash94.1%
- Claude Opus 593.9%
- GPT-5.6 Sol93.5%
- Grok 4.593.4%
- GPT-5.6 Terra93.3%
- GPT-5.493.3%
- DeepSeek V4.1 Flash93.2%
- Grok 4.693.2%
- Kimi K393.1%
- Grok 4.792.7%
- Qwen3.8 Max92.3%
- DeepSeek V4 Pro91.7%
- GPT-5.6 Luna91.6%
- Claude Opus 4.891%
- GLM-5.390.9%
- Claude Opus 5.590.6%
- GPT-6 Luna90.5%
- GLM-5.3-Flash90.2%
- Claude Haiku 5.589.6%
- Muse Spark Open88.6%
- Claude Opus 4.688.4%
- Kimi K2.7 Code87.9%
- Claude Opus 4.786.4%
- Claude Opus 4.586%
- Claude Fable 585.9%
- Claude Sonnet 4.582.3%
- Claude Sonnet 580.3%
- Claude Sonnet 4.678.8%
- gpt-oss-120b75.8%
- Claude Haiku 4.571.2%
- Llama 4 Maverick67%
- Llama 4 Scout51.8%
- Kimi K3.1 Pro94.2%Moonshot AI's own figure
- GPT-5.593.5%Artificial Analysis's run
- Muse Spark 1.393.5%Artificial Analysis's run, GPQA Diamond
- Step 5 Preview93.5%StepFun's own figure, GPQA Diamond
- Claude Fable 5.193.4%Vals.ai's run
- GPT-5.3-Codex91.5%Artificial Analysis's run
- Qwen3.8 27B90.5%Artificial Analysis's run, GPQA Diamond
- Gemma 4 31B85.7%Artificial Analysis's run, GPQA Diamond
- Mistral Medium 3.574.8%Artificial Analysis's run, GPQA Diamond
The other models in the grid have no result on GPQA Diamond yet.
Hard reasoningHumanity's Last Exam
- Claude Opus 5.561.4%
- Claude Fable 5.159.1%
- Gemini 4 Argon57.1%
- Claude Fable 555.5%
- Claude Sonnet 5.555%
- Claude Opus 554.9%
- GPT-6 Astra54.7%
- GPT-6.1 Sol52.9%
- Kimi K3.1 Pro52.5%
- Muse Wonder50.1%
- GPT-5.6 Sol49.5%
- MiMo-V2.6-Pro49.4%
- Claude Opus 4.848.7%
- Muse Spark 1.348.7%
- MiniMax M3.1 Flash48%
- GPT-6 Sol47.9%
- Gemini 3.8 Flash47.8%
- Gemini 3.1 Pro Preview47%
- Kimi K346.9%
- Step 5 Preview46.5%
- GPT-6.1 Terra46.2%
- GPT-5.545.8%
- Claude Haiku 5.544.4%
- Muse Spark Open43.9%
- GPT-5.443.7%
- Ling 3.1 Flash43.5%
- DeepSeek V4.1 Flash43.3%
- Grok 4.743.1%
- Qwen3.8 Max43.1%
- GPT-5.6 Terra42.9%
- Grok 4.642.9%
- Grok 4.542.7%
- GPT-5.3-Codex42.5%
- Claude Opus 4.742.3%
- GLM-5.342.3%
- Claude Sonnet 541.3%
- GLM-5.241.1%
- DeepSeek V4 Pro41%
- Gemini 3.6 Flash40.8%
- Claude Opus 4.639.9%
- GLM-5.3-Flash39.9%
- GPT-5.6 Luna39.5%
- GPT-6 Luna38.5%
- MiMo-V2.6-Flash35.1%
- Kimi K2.7 Code35%
- Qwen3.8 27B33.9%
- Claude Sonnet 4.633.6%
- Claude Opus 4.530.1%
- Gemma 4 31B23.6%
- gpt-oss-120b19.6%
- Claude Sonnet 4.517.8%
- Mistral Medium 3.513.8%
- Claude Haiku 4.510.4%
- Llama 4 Maverick4.9%
- Llama 4 Scout3.8%
The other models in the grid have no result on Humanity's Last Exam yet.
Tool useAutomationBench-AA
- Gemini 4 Argon77.5%
- Kimi K3.1 Pro73.3%
- MiniMax M3.1 Flash72.5%
- Claude Sonnet 5.571.8%
- Claude Opus 5.569.5%
- GPT-6 Astra68.5%
- Muse Wonder68.1%
- Ling 3.1 Flash67.5%
- Grok 4.666.7%
- DeepSeek V4.1 Flash66.6%
- Grok 4.765.6%
- GPT-6.1 Sol64.9%
- MiMo-V2.6-Flash64.1%
- GLM-5.362.2%
- GPT-6 Sol61.6%
- GLM-5.3-Flash60.4%
- GPT-6.1 Terra60.2%
- GPT-5.6 Sol60.1%
- Gemini 3.8 Flash59.9%
- GPT-5.6 Terra59.6%
- Claude Fable 5.159.4%
- MiMo-V2.6-Pro58.6%
- Kimi K358.3%
- Grok 4.557.9%
- Muse Spark 1.357.9%
- Mistral Large 457.8%
- DeepSeek V4 Pro56.7%
- Claude Opus 556.6%
- Qwen3.8 Max56.2%
- Claude Fable 554.1%
- GPT-6 Luna53.2%
- Gemini 3.6 Flash53%
- Step 5 Preview51%
- GPT-5.6 Luna50.2%
- Muse Spark Open49.7%
- Qwen3.8 27B48.2%
- GPT-5.547.3%
- Claude Opus 4.845.6%
- Claude Sonnet 536.5%
- Gemini 3.1 Pro Preview35.4%
- Claude Haiku 5.535.4%
- GLM-5.228.4%
- Kimi K2.7 Code24.5%
- Claude Sonnet 4.620.1%
- Claude Sonnet 4.514.1%
- Mistral Medium 3.56.3%
- Gemma 4 31B4.9%
- Claude Haiku 4.53.2%
- gpt-oss-120b0.2%
- Llama 4 Scout0.2%
- Llama 4 Maverick0.2%
The other models in the grid have no result on AutomationBench-AA yet.
Computer useOSWorld 2.1
- Claude Opus 577.7%
- Muse Spark 1.369.6%
- Qwen3.8 27B84.3%Alibaba's own figure, OSWorld-Verified
- Gemini 3.6 Flash83%Google's own figure, OSWorld-Verified
- MiMo-V2.6-Pro82%Xiaomi's own figure, OSWorld-Verified
- Claude Opus 5.581.8%Anthropic's own figure
- MiMo-V2.6-Flash80.8%Xiaomi's own figure, OSWorld-Verified
- Claude Fable 5.180.7%Anthropic's own figure
- Muse Spark Open80.4%Meta's own figure, OSWorld-Verified
- Claude Sonnet 5.580.1%Anthropic's own figure
- Kimi K3.1 Pro79.6%Moonshot AI's own figure, OSWorld 2.1
- GPT-6.1 Sol78%OpenAI's own figure, OSWorld 2.1
- GPT-6 Astra75.8%Meta's run
- GPT-6.1 Terra74.2%OpenAI's own figure, OSWorld 2.1
- Muse Wonder72.6%Meta's own figure
- Claude Haiku 5.572.4%Anthropic's own figure, OSWorld 2.1
- DeepSeek V4.1 Flash69.3%Moonshot AI's run, OSWorld 2.1
- Gemini 4 Argon69.2%Google's own figure, OSWorld 2.0
- GPT-6 Sol64.4%OpenAI's own figure, OSWorld 2.0
- GPT-5.6 Sol62.7%OSWorld team's run, OSWorld 2.0
- MiniMax M3.1 Flash61.7%MiniMax's own figure, OSWorld 2.0
- GLM-5.3-Flash59.1%Z.ai's own figure, OSWorld 2.0
- Gemini 3.8 Flash59%Google's own figure, OSWorld 2.0
- Claude Sonnet 557%Anthropic's own figure
- Claude Opus 4.854.8%OSWorld team's run, OSWorld 2.0
- GPT-5.549.5%OSWorld team's run, OSWorld 2.0
- Claude Opus 4.748.9%OSWorld team's run, OSWorld 2.0
- GPT-6 Luna48.9%Anthropic's run, OSWorld 2.1
- Claude Sonnet 4.641.5%OSWorld team's run, OSWorld 2.0
The other models in the grid have no result on OSWorld 2.1 yet.
Long contextAA-LCR v1.1
- Kimi K388.7%
- Step 5 Preview88.3%
- Kimi K3.1 Pro86.3%
- MiMo-V2.6-Pro86.3%
- Claude Fable 5.185.3%
- Claude Opus 5.584.7%
- GPT-5.584.3%
- GPT-5.6 Sol84%
- GPT-5.6 Luna83.7%
- GPT-6 Sol83.7%
- GPT-6 Luna83.3%
- GPT-5.3-Codex83.3%
- GPT-6.1 Sol83%
- GPT-5.6 Terra83%
- Muse Spark 1.383%
- Claude Haiku 5.582.7%
- Claude Sonnet 5.582.7%
- GPT-6.1 Terra82.7%
- DeepSeek V4.1 Flash82.7%
- Claude Fable 582.3%
- Ling 3.1 Flash82.3%
- Gemini 3.1 Pro Preview82%
- Claude Sonnet 582%
- GPT-5.482%
- Qwen3.8 27B82%
- Gemini 3.8 Flash81.3%
- GPT-6 Astra80.7%
- Muse Wonder80.7%
- DeepSeek V4 Pro80.3%
- Grok 4.680.3%
- Qwen3.8 Max80.3%
- Gemini 3.6 Flash80%
- Claude Sonnet 4.680%
- GLM-5.3-Flash80%
- Gemini 4 Argon79.7%
- MiniMax M3.1 Flash79.7%
- GLM-5.379.7%
- Claude Opus 579.3%
- Kimi K2.7 Code79.3%
- Grok 4.579.3%
- Claude Opus 4.778.7%
- GLM-5.278.3%
- Muse Spark Open78.3%
- Claude Opus 4.678%
- Claude Opus 4.877.7%
- Claude Opus 4.577.3%
- Grok 4.776.7%
- Claude Haiku 4.574.3%
- MiMo-V2.6-Flash74.3%
- Claude Sonnet 4.572.3%
- Gemma 4 31B69.7%
- Mistral Medium 3.569.3%
- gpt-oss-120b52%
- Llama 4 Maverick50%
- Llama 4 Scout27.7%
The other models in the grid have no result on AA-LCR v1.1 yet.
MultimodalMMMU-Pro
- Claude Opus 5.587.7%
- GPT-6 Astra86.9%
- GPT-6.1 Sol86%
- Gemini 3.8 Flash85.6%
- Claude Opus 584.7%
- GPT-6.1 Terra84.7%
- Kimi K3.1 Pro83.9%
- GPT-5.6 Sol83.4%
- Gemini 3.6 Flash83.2%
- GPT-6 Sol82.9%
- Qwen3.8 Max82.8%
- Gemini 3.1 Pro Preview82.4%
- Muse Spark 1.382%
- Muse Spark Open80.9%
- GPT-5.6 Terra80.7%
- DeepSeek V4.1 Flash80.6%
- Kimi K380.5%
- Grok 4.580.4%
- GPT-5.579.9%
- GPT-6 Luna79.7%
- MiniMax M3.1 Flash79.1%
- Claude Opus 4.778.8%
- GPT-5.6 Luna78.6%
- GPT-5.3-Codex78.5%
- GPT-5.478.4%
- Claude Sonnet 577.3%
- Step 5 Preview76.4%
- Qwen3.8 27B76.3%
- Claude Opus 4.675.4%
- Claude Opus 4.574%
- Gemma 4 31B73.4%
- Claude Sonnet 4.673.3%
- MiMo-V2.6-Flash73.1%
- Claude Sonnet 4.568.7%
- Mistral Medium 3.564.9%
- Llama 4 Maverick62.1%
- Claude Haiku 4.558.6%
- Llama 4 Scout52.9%
- Claude Fable 5.186.3%Moonshot AI's run
- GLM-5.3-Flash86%Vals.ai's run, MMMU-Pro
The other models in the grid have no result on MMMU-Pro yet.
FactualityAA-Omniscience Accuracy
- Claude Fable 5.168.4%
- Claude Opus 5.565.5%
- Claude Fable 565.4%
- GPT-6.1 Sol61.8%
- GPT-6 Astra61.6%
- Claude Opus 560.9%
- Gemini 3.1 Pro Preview54.9%
- GPT-6 Sol54.5%
- Claude Sonnet 5.551.5%
- Gemini 3.8 Flash50.5%
- Claude Opus 4.748.9%
- Kimi K3.1 Pro48.9%
- Claude Opus 4.848.8%
- Gemini 4 Argon48.3%
- GPT-6.1 Terra47.6%
- Claude Opus 4.647%
- Claude Opus 4.546.6%
- DeepSeek V4 Pro45.2%
- MiniMax M3.1 Flash45.1%
- Grok 4.742.7%
- GPT-5.6 Terra42.3%
- Muse Spark Open42%
- Claude Sonnet 4.640.9%
- Claude Sonnet 540.1%
- GPT-6 Luna39.6%
- DeepSeek V4.1 Flash39.6%
- Muse Wonder38%
- Muse Spark 1.338%
- Step 5 Preview36.4%
- MiMo-V2.6-Pro34.8%
- Claude Haiku 5.534.7%
- Claude Sonnet 4.532.9%
- GLM-5.332.3%
- Qwen3.8 Max29.5%
- GLM-5.3-Flash28.2%
- MiMo-V2.6-Flash27.2%
- Ling 3.1 Flash26.5%
- Mistral Large 426%
- Mistral Medium 3.523%
- Claude Haiku 4.518%
- Qwen3.8 27B17.3%
- GPT-5.6 Sol69.7%Epoch AI's run, SimpleQA Verified
- Gemini 3.6 Flash66.2%Epoch AI's run, SimpleQA Verified
- GPT-5.563%Epoch AI's run, SimpleQA Verified
- Kimi K350.6%Epoch AI's run, SimpleQA Verified
- Grok 4.648.9%Epoch AI's run, SimpleQA Verified
- Grok 4.548.3%Epoch AI's run, SimpleQA Verified
- GPT-5.6 Luna41%Epoch AI's run, SimpleQA Verified
- Kimi K2.7 Code36.5%Epoch AI's run, SimpleQA Verified
- GLM-5.234.2%Epoch AI's run, SimpleQA Verified
- gpt-oss-120b16.8%OpenAI's own figure, SimpleQA
- Gemma 4 31B10.4%Epoch AI's run, SimpleQA Verified
The other models in the grid have no result on AA-Omniscience Accuracy yet.
Answers in fullSpeechMap
- Grok 4.593.1%
- Gemini 3.6 Flash87.2%
- Gemini 4 Argon87%
- Gemini 3.8 Flash86.1%
- Grok 4.683.6%
- Claude Opus 5.582.8%
- Gemma 4 31B82.3%
- Claude Sonnet 5.581.6%
- Claude Haiku 5.578.8%
- Claude Opus 573.7%
- Claude Fable 5.172.4%
- DeepSeek V4 Pro72%
- GLM-5.371.5%
- Mistral Medium 3.570.5%
- Claude Opus 4.770.2%
- Llama 4 Scout70.1%
- GPT-6.1 Sol69.5%
- GPT-6 Astra68.2%
- Claude Opus 4.867.4%
- MiMo-V2.6-Flash66.9%
- MiMo-V2.6-Pro66.5%
- Kimi K365.9%
- GLM-5.3-Flash65.4%
- Qwen3.8 Max65.3%
- Claude Fable 564.8%
- Grok 4.764.1%
- GPT-6 Sol64%
- GPT-6 Luna63.4%
- Gemini 3.1 Pro Preview59.9%
- Claude Opus 4.659.9%
- Step 5 Preview59.7%
- Llama 4 Maverick58.3%
- Claude Opus 4.557.5%
- Muse Wonder42.7%
- gpt-oss-120b36.3%
- Claude Sonnet 4.635.8%
- Claude Sonnet 4.531.4%
- Muse Spark 1.325.8%
- Claude Haiku 4.524.7%
The other models in the grid have no result on SpeechMap yet.
Instruction followingMultiChallenge
- GPT-6.1 Sol82%
- Gemini 4 Argon78.7%
- DeepSeek V4.1 Flash74.9%
- GPT-6.1 Terra74.2%
- Muse Spark Open74.2%
- Gemini 3.1 Pro Preview71.4%
- Claude Haiku 5.567.7%
- Claude Opus 4.559%
- Claude Sonnet 4.555.3%
- Claude Haiku 4.550.5%
- gpt-oss-120b45.3%
- Claude Opus 4.637.2%
- Qwen3.8 27B79.5%Alibaba's own figure, IFBench
- GPT-5.575.9%Artificial Analysis's run, IFBench
- Gemma 4 31B75.6%Artificial Analysis's run, IFBench
- GPT-5.3-Codex75.4%Artificial Analysis's run, IFBench
- GPT-5.473.9%Artificial Analysis's run, IFBench
- GLM-5.273.3%Artificial Analysis's run, IFBench
- GPT-5.6 Sol72.7%Artificial Analysis's run, IFBench
- GPT-5.6 Terra71.2%Artificial Analysis's run, IFBench
- Mistral Medium 3.568.8%Artificial Analysis's run, IFBench
- Claude Fable 563.5%Artificial Analysis's run, IFBench
- Kimi K2.7 Code63.1%Artificial Analysis's run, IFBench
- Claude Opus 4.862.2%Artificial Analysis's run, IFBench
- Claude Opus 4.758.6%Artificial Analysis's run, IFBench
- Claude Sonnet 4.656.6%Artificial Analysis's run, IFBench
- Llama 4 Maverick43%Artificial Analysis's run, IFBench
- Llama 4 Scout39.5%Artificial Analysis's run, IFBench
The other models in the grid have no result on MultiChallenge yet.
Releases
Every Gemma release we track, newest first.
What practitioners report
Strongest on Quality and reasoning, weakest on Long context.
Each figure is the share of accounts naming Gemma 4 31B that speak well of it on that axis. An axis is scored once praise and concerns are both on record from at least five accounts by three authors. An account from the maker, a company that sells or serves the model, or the maker's expert program is listed and counts toward no figure.
| Axis | Favorable | Accounts |
|---|---|---|
| Quality and reasoningWhether its answers and code are right on the author's own task: reasoning, accuracy, instruction following, and consistency. | 80% | 24 favorable, 6 not, 28 authors |
| Tool use and structured outputWhether it calls tools correctly, returns valid structured output, and keeps agent loops on track. | 67% | 4 favorable, 2 not, 6 authors |
| Long contextHow it holds up over long documents, large codebases, long sessions, and many images or PDFs. | 20% | 1 favorable, 4 not, 5 authors |
| Developer experienceHow it is to build with: prompting, steering, effort settings, and support across SDKs and providers. | Only concerns on record | 0 favorable, 7 not, 7 authors |
| CostWhat the work costs: price per task, token use, and what caching and batch save. | Only praise on record | 7 favorable, 0 not, 7 authors |
| SpeedResponse time, time to first token, and throughput. | 53% | 10 favorable, 9 not, 17 authors |
| ReliabilityWhether requests keep succeeding: overloaded errors, outages, rate limits, and changes that broke what worked. | Only concerns on record | 0 favorable, 5 not, 5 authors |
Every model on these seven axes
| Axis | Gemma 4 31B | Gemini 4 Argon | Gemini 3.8 Flash | Gemini 3.1 Pro Preview | Gemini 3.6 Flash | Claude Haiku 5.5 | Claude Sonnet 5.5 | Claude Opus 5.5 | Claude Fable 5.1 | Claude Opus 5 | Claude Sonnet 5 | Claude Fable 5 | Claude Opus 4.8 | Claude Opus 4.7 | Claude Sonnet 4.6 | Claude Opus 4.6 | Claude Opus 4.5 | Claude Haiku 4.5 | Claude Sonnet 4.5 | GPT-6.1 Terra | GPT-6.1 Sol | GPT-6 Luna | GPT-6 Astra | GPT-5.6 Sol | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.5 | GPT-5.4 | gpt-oss-120b | GPT-6 Sol | GPT-5.3-Codex | DeepSeek V4.1 Flash | DeepSeek V4 Pro | Kimi K3.1 Pro | Kimi K3 | Kimi K2.7 Code | Mistral Large 4 | Mistral Medium 3.5 | Grok 4.7 | Grok 4.6 | Grok 4.5 | MiniMax M3.1 Flash | Mercury 3 | GLM-5.3-Flash | GLM-5.3 | GLM-5.2 | Muse Spark Open | Muse Wonder | Muse Spark 1.3 | Llama 4 Scout | Llama 4 Maverick | Qwen3.8 27B | Qwen3.8 Max | MiMo-V2.6-Pro | MiMo-V2.6-Flash | Ling 3.1 Flash | Step 5 Preview |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Quality and reasoning | 80%24 favorable, 6 not, 28 authors | 63%22 favorable, 13 not, 35 authors | 56%27 favorable, 21 not, 45 authors | 56%20 favorable, 16 not, 36 authors | Needs 5 accounts from 3 authors1 favorable, 3 not, 4 authors | 73%19 favorable, 7 not, 26 authors | 64%14 favorable, 8 not, 22 authors | 68%17 favorable, 8 not, 25 authors | 54%15 favorable, 13 not, 28 authors | 50%6 favorable, 6 not, 12 authors | 56%9 favorable, 7 not, 16 authors | 68%13 favorable, 6 not, 18 authors | 68%17 favorable, 8 not, 25 authors | 57%12 favorable, 9 not, 21 authors | 67%12 favorable, 6 not, 18 authors | 70%19 favorable, 8 not, 27 authors | 62%13 favorable, 8 not, 20 authors | 58%14 favorable, 10 not, 24 authors | 69%11 favorable, 5 not, 16 authors | 33%3 favorable, 6 not, 8 authors | 55%12 favorable, 10 not, 22 authors | 55%16 favorable, 13 not, 29 authors | 44%12 favorable, 15 not, 26 authors | 33%3 favorable, 6 not, 9 authors | 38%3 favorable, 5 not, 8 authors | Needs 5 accounts from 3 authors3 favorable, 1 not, 4 authors | 33%4 favorable, 8 not, 12 authors | 33%2 favorable, 4 not, 6 authors | 58%22 favorable, 16 not, 38 authors | 31%10 favorable, 22 not, 32 authors | Needs 5 accounts from 3 authors1 favorable, 3 not, 4 authors | 61%25 favorable, 16 not, 40 authors | Only concerns on record0 favorable, 4 not, 4 authors | 59%20 favorable, 14 not, 32 authors | Only praise on record3 favorable, 0 not, 3 authors | 50%3 favorable, 3 not, 6 authors | 58%11 favorable, 8 not, 18 authors | 36%5 favorable, 9 not, 14 authors | 43%16 favorable, 21 not, 36 authors | 86%6 favorable, 1 not, 7 authors | 60%3 favorable, 2 not, 5 authors | 62%13 favorable, 8 not, 21 authors | 86%18 favorable, 3 not, 21 authors | 67%14 favorable, 7 not, 21 authors | 76%47 favorable, 15 not, 61 authors | 71%5 favorable, 2 not, 7 authors | Needs 5 accounts from 3 authors2 favorable, 1 not, 3 authors | 20%1 favorable, 4 not, 5 authors | Only praise on record2 favorable, 0 not, 2 authors | 53%8 favorable, 7 not, 15 authors | 44%8 favorable, 10 not, 17 authors | 71%24 favorable, 10 not, 34 authors | 57%4 favorable, 3 not, 7 authors | 22%2 favorable, 7 not, 9 authors | 40%4 favorable, 6 not, 10 authors | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors2 favorable, 1 not, 3 authors |
| Tool use and structured output | 67%4 favorable, 2 not, 6 authors | 25%2 favorable, 6 not, 8 authors | 47%7 favorable, 8 not, 15 authors | 54%7 favorable, 6 not, 12 authors | Only concerns on record0 favorable, 1 not, 1 author | 67%6 favorable, 3 not, 9 authors | 40%4 favorable, 6 not, 10 authors | 46%6 favorable, 7 not, 13 authors | 30%3 favorable, 7 not, 10 authors | Needs 5 accounts from 3 authors3 favorable, 1 not, 4 authors | 70%7 favorable, 3 not, 10 authors | 57%4 favorable, 3 not, 7 authors | 50%4 favorable, 4 not, 8 authors | 50%4 favorable, 4 not, 8 authors | 29%2 favorable, 5 not, 7 authors | 33%4 favorable, 8 not, 12 authors | 70%7 favorable, 3 not, 10 authors | 43%3 favorable, 4 not, 6 authors | 50%3 favorable, 3 not, 6 authors | 40%2 favorable, 3 not, 5 authors | 40%4 favorable, 6 not, 10 authors | 33%3 favorable, 6 not, 9 authors | 62%8 favorable, 5 not, 13 authors | No accounts yet | Only concerns on record0 favorable, 2 not, 2 authors | Only praise on record1 favorable, 0 not, 1 author | Only praise on record1 favorable, 0 not, 1 author | No accounts yet | 27%4 favorable, 11 not, 15 authors | 56%5 favorable, 4 not, 9 authors | No accounts yet | 59%10 favorable, 7 not, 17 authors | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors3 favorable, 1 not, 4 authors | Only concerns on record0 favorable, 1 not, 1 author | No accounts yet | 38%5 favorable, 8 not, 13 authors | Only concerns on record0 favorable, 5 not, 5 authors | 38%6 favorable, 10 not, 16 authors | Only praise on record2 favorable, 0 not, 2 authors | Only concerns on record0 favorable, 1 not, 1 author | 67%4 favorable, 2 not, 6 authors | 80%8 favorable, 2 not, 10 authors | 33%2 favorable, 4 not, 6 authors | 57%4 favorable, 3 not, 7 authors | No accounts yet | No accounts yet | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | Needs 5 accounts from 3 authors1 favorable, 2 not, 3 authors | Only concerns on record0 favorable, 4 not, 4 authors | 46%6 favorable, 7 not, 13 authors | Only praise on record1 favorable, 0 not, 1 author | Only concerns on record0 favorable, 5 not, 5 authors | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author |
| Long context | 20%1 favorable, 4 not, 5 authors | 25%2 favorable, 6 not, 8 authors | 38%5 favorable, 8 not, 13 authors | 44%7 favorable, 9 not, 16 authors | Only praise on record1 favorable, 0 not, 1 author | 29%2 favorable, 5 not, 7 authors | 60%3 favorable, 2 not, 5 authors | 80%8 favorable, 2 not, 10 authors | 63%5 favorable, 3 not, 8 authors | 67%4 favorable, 2 not, 6 authors | 50%3 favorable, 3 not, 6 authors | 33%2 favorable, 4 not, 6 authors | 60%6 favorable, 4 not, 10 authors | 50%4 favorable, 4 not, 7 authors | 71%5 favorable, 2 not, 7 authors | 64%7 favorable, 4 not, 11 authors | 40%4 favorable, 6 not, 10 authors | 40%2 favorable, 3 not, 5 authors | 50%4 favorable, 4 not, 8 authors | Only concerns on record0 favorable, 1 not, 1 author | 38%3 favorable, 5 not, 8 authors | 20%1 favorable, 4 not, 5 authors | 27%3 favorable, 8 not, 11 authors | No accounts yet | No accounts yet | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors2 favorable, 1 not, 3 authors | 25%2 favorable, 6 not, 8 authors | Only concerns on record0 favorable, 3 not, 3 authors | No accounts yet | 58%7 favorable, 5 not, 12 authors | No accounts yet | Needs 5 accounts from 3 authors3 favorable, 1 not, 4 authors | Only praise on record1 favorable, 0 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors1 favorable, 3 not, 4 authors | No accounts yet | 42%5 favorable, 7 not, 12 authors | No accounts yet | No accounts yet | 40%2 favorable, 3 not, 5 authors | 60%3 favorable, 2 not, 5 authors | No accounts yet | 40%2 favorable, 3 not, 5 authors | No accounts yet | No accounts yet | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 3 not, 3 authors | Needs 5 accounts from 3 authors2 favorable, 1 not, 3 authors | 75%6 favorable, 2 not, 8 authors | Only praise on record1 favorable, 0 not, 1 author | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | No accounts yet | No accounts yet |
| Developer experience | Only concerns on record0 favorable, 7 not, 7 authors | 33%3 favorable, 6 not, 9 authors | 39%7 favorable, 11 not, 17 authors | 40%8 favorable, 12 not, 20 authors | No accounts yet | 25%2 favorable, 6 not, 8 authors | 38%5 favorable, 8 not, 13 authors | 38%6 favorable, 10 not, 16 authors | 50%7 favorable, 7 not, 14 authors | 25%3 favorable, 9 not, 12 authors | 29%2 favorable, 5 not, 7 authors | 25%3 favorable, 9 not, 12 authors | 53%10 favorable, 9 not, 19 authors | 29%4 favorable, 10 not, 14 authors | 57%4 favorable, 3 not, 7 authors | 45%5 favorable, 6 not, 11 authors | 67%8 favorable, 4 not, 12 authors | Needs 5 accounts from 3 authors2 favorable, 2 not, 4 authors | 50%3 favorable, 3 not, 6 authors | Only concerns on record0 favorable, 2 not, 2 authors | 27%3 favorable, 8 not, 11 authors | 25%2 favorable, 6 not, 8 authors | 41%7 favorable, 10 not, 17 authors | No accounts yet | Only concerns on record0 favorable, 1 not, 1 author | No accounts yet | No accounts yet | No accounts yet | 14%1 favorable, 6 not, 7 authors | 25%3 favorable, 9 not, 12 authors | No accounts yet | 35%8 favorable, 15 not, 23 authors | No accounts yet | 9%1 favorable, 10 not, 11 authors | No accounts yet | Only concerns on record0 favorable, 1 not, 1 author | 13%2 favorable, 14 not, 16 authors | Only concerns on record0 favorable, 9 not, 9 authors | 25%5 favorable, 15 not, 20 authors | No accounts yet | No accounts yet | Only concerns on record0 favorable, 7 not, 7 authors | 11%1 favorable, 8 not, 9 authors | Only concerns on record0 favorable, 3 not, 3 authors | 33%5 favorable, 10 not, 15 authors | No accounts yet | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | No accounts yet | Only concerns on record0 favorable, 6 not, 6 authors | Only concerns on record0 favorable, 1 not, 1 author | 17%1 favorable, 5 not, 6 authors | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | No accounts yet | No accounts yet |
| Cost | Only praise on record7 favorable, 0 not, 7 authors | 35%7 favorable, 13 not, 20 authors | 58%14 favorable, 10 not, 23 authors | 50%8 favorable, 8 not, 16 authors | Only concerns on record0 favorable, 1 not, 1 author | 69%20 favorable, 9 not, 29 authors | 45%10 favorable, 12 not, 22 authors | 56%15 favorable, 12 not, 27 authors | 20%4 favorable, 16 not, 20 authors | 43%3 favorable, 4 not, 7 authors | 39%7 favorable, 11 not, 17 authors | 8%1 favorable, 11 not, 12 authors | 40%6 favorable, 9 not, 14 authors | 50%7 favorable, 7 not, 13 authors | 50%6 favorable, 6 not, 12 authors | 33%7 favorable, 14 not, 20 authors | 53%8 favorable, 7 not, 15 authors | 29%4 favorable, 10 not, 12 authors | 38%3 favorable, 5 not, 8 authors | 80%8 favorable, 2 not, 9 authors | 74%14 favorable, 5 not, 19 authors | 79%22 favorable, 6 not, 26 authors | 29%7 favorable, 17 not, 23 authors | 57%4 favorable, 3 not, 7 authors | Only concerns on record0 favorable, 1 not, 1 author | 50%3 favorable, 3 not, 6 authors | Needs 5 accounts from 3 authors1 favorable, 3 not, 4 authors | 63%5 favorable, 3 not, 8 authors | 60%3 favorable, 2 not, 5 authors | 55%12 favorable, 10 not, 21 authors | Needs 5 accounts from 3 authors2 favorable, 1 not, 3 authors | 66%25 favorable, 13 not, 37 authors | Only praise on record2 favorable, 0 not, 2 authors | 40%10 favorable, 15 not, 25 authors | 71%5 favorable, 2 not, 7 authors | Needs 5 accounts from 3 authors1 favorable, 3 not, 4 authors | 27%3 favorable, 8 not, 11 authors | Only concerns on record0 favorable, 8 not, 8 authors | 19%5 favorable, 21 not, 26 authors | 20%1 favorable, 4 not, 5 authors | Only praise on record4 favorable, 0 not, 4 authors | 80%12 favorable, 3 not, 15 authors | 50%6 favorable, 6 not, 12 authors | 76%16 favorable, 5 not, 21 authors | 35%8 favorable, 15 not, 23 authors | 71%5 favorable, 2 not, 7 authors | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 9 not, 9 authors | Needs 5 accounts from 3 authors2 favorable, 1 not, 3 authors | Only praise on record1 favorable, 0 not, 1 author | Only praise on record1 favorable, 0 not, 1 author | Needs 5 accounts from 3 authors2 favorable, 2 not, 4 authors | Only concerns on record0 favorable, 2 not, 2 authors | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | Only praise on record3 favorable, 0 not, 3 authors | Only praise on record1 favorable, 0 not, 1 author | Only praise on record1 favorable, 0 not, 1 author |
| Speed | 53%10 favorable, 9 not, 17 authors | 45%5 favorable, 6 not, 11 authors | 61%19 favorable, 12 not, 31 authors | 35%6 favorable, 11 not, 16 authors | Needs 5 accounts from 3 authors2 favorable, 1 not, 3 authors | 74%14 favorable, 5 not, 19 authors | 88%14 favorable, 2 not, 16 authors | 46%6 favorable, 7 not, 13 authors | 44%4 favorable, 5 not, 9 authors | 60%3 favorable, 2 not, 5 authors | 44%4 favorable, 5 not, 9 authors | 57%4 favorable, 3 not, 7 authors | 31%4 favorable, 9 not, 13 authors | 63%5 favorable, 3 not, 8 authors | 17%1 favorable, 5 not, 6 authors | 36%4 favorable, 7 not, 11 authors | 73%8 favorable, 3 not, 11 authors | 78%7 favorable, 2 not, 9 authors | 75%6 favorable, 2 not, 8 authors | 83%5 favorable, 1 not, 5 authors | 37%7 favorable, 12 not, 19 authors | 45%5 favorable, 6 not, 11 authors | 41%7 favorable, 10 not, 17 authors | Only praise on record1 favorable, 0 not, 1 author | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | 80%4 favorable, 1 not, 5 authors | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | No accounts yet | 85%23 favorable, 4 not, 27 authors | 57%4 favorable, 3 not, 7 authors | Only praise on record1 favorable, 0 not, 1 author | 68%21 favorable, 10 not, 30 authors | Only concerns on record0 favorable, 1 not, 1 author | 21%4 favorable, 15 not, 19 authors | Only concerns on record0 favorable, 2 not, 2 authors | No accounts yet | 21%3 favorable, 11 not, 14 authors | Only concerns on record0 favorable, 2 not, 2 authors | 29%5 favorable, 12 not, 17 authors | Needs 5 accounts from 3 authors1 favorable, 2 not, 3 authors | Only praise on record2 favorable, 0 not, 2 authors | 44%4 favorable, 5 not, 9 authors | 65%13 favorable, 7 not, 20 authors | 33%3 favorable, 6 not, 9 authors | 20%3 favorable, 12 not, 15 authors | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | Needs 5 accounts from 3 authors3 favorable, 1 not, 4 authors | No accounts yet | 33%7 favorable, 14 not, 21 authors | Needs 5 accounts from 3 authors1 favorable, 3 not, 4 authors | Only concerns on record0 favorable, 4 not, 4 authors | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 2 not, 2 authors | Only concerns on record0 favorable, 1 not, 1 author |
| Reliability | Only concerns on record0 favorable, 5 not, 5 authors | 5%1 favorable, 18 not, 19 authors | 15%3 favorable, 17 not, 20 authors | 14%3 favorable, 19 not, 22 authors | Only concerns on record0 favorable, 1 not, 1 author | 20%1 favorable, 4 not, 5 authors | 18%2 favorable, 9 not, 11 authors | 13%1 favorable, 7 not, 8 authors | 9%1 favorable, 10 not, 11 authors | 13%1 favorable, 7 not, 8 authors | 63%5 favorable, 3 not, 7 authors | 22%2 favorable, 7 not, 9 authors | 17%2 favorable, 10 not, 12 authors | 18%2 favorable, 9 not, 11 authors | 17%1 favorable, 5 not, 6 authors | 20%3 favorable, 12 not, 15 authors | 38%6 favorable, 10 not, 16 authors | 29%2 favorable, 5 not, 7 authors | 40%4 favorable, 6 not, 10 authors | Only concerns on record0 favorable, 1 not, 1 author | 13%1 favorable, 7 not, 8 authors | Only concerns on record0 favorable, 5 not, 5 authors | 14%2 favorable, 12 not, 14 authors | Only concerns on record0 favorable, 4 not, 4 authors | Only concerns on record0 favorable, 1 not, 1 author | No accounts yet | Only concerns on record0 favorable, 2 not, 2 authors | Only concerns on record0 favorable, 4 not, 4 authors | Only concerns on record0 favorable, 4 not, 4 authors | 17%1 favorable, 5 not, 6 authors | Only concerns on record0 favorable, 3 not, 3 authors | 42%5 favorable, 7 not, 12 authors | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | 8%1 favorable, 12 not, 13 authors | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors1 favorable, 1 not, 2 authors | 38%3 favorable, 5 not, 8 authors | Only concerns on record0 favorable, 4 not, 4 authors | 22%2 favorable, 7 not, 9 authors | Only concerns on record0 favorable, 1 not, 1 author | No accounts yet | 60%3 favorable, 2 not, 5 authors | 33%2 favorable, 4 not, 5 authors | Only concerns on record0 favorable, 7 not, 7 authors | Only concerns on record0 favorable, 4 not, 4 authors | Only concerns on record0 favorable, 1 not, 1 author | No accounts yet | Only concerns on record0 favorable, 2 not, 2 authors | Only concerns on record0 favorable, 1 not, 1 author | Needs 5 accounts from 3 authors1 favorable, 3 not, 4 authors | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 9 not, 9 authors | Only concerns on record0 favorable, 1 not, 1 author | Only concerns on record0 favorable, 2 not, 2 authors | No accounts yet | No accounts yet | Only concerns on record0 favorable, 1 not, 1 author |
The 66 accounts behind Gemma 4 31B's figures
GILD searched Hacker News and GitHub for accounts that name Gemma 4 31B.
Quality and reasoning 80%
- Favorable
This local LLM marks the first version that 1) doesn't hallucinate words - at least, for the largest model
Google releases Gemma 4 open models. Retro_Dev, Hacker News commenter, self-described, Apr 3, 2026. Gemma 4 31B, the largest Gemma 4 model, run locally through Ollama on a laptop for toki pona translation.
- Favorable
Gemma 4 is surprisingly good at multilingual tasks, especially given its size. This is mostly true for the dense 31B model.
Google releases Gemma 4 open models. curioussquirrel, Hacker News commenter, self-described, Apr 4, 2026. Gemma 4 31B dense in the author's multilingual testing.
- Favorable
gemma-4-26b-a4b-it and gemma-4-31b-it produced accurate results in a few of my tests.
Google releases Gemma 4 open models. aggregator-ios, Hacker News commenter, self-described, Apr 3, 2026. gemma-4-31b-it (Gemma 4 31B) generating jq queries from natural language for jsonquery.app.
- Favorable
From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results.
My local model setup on an M4 Pro Mac Mini. ActorNightly, Hacker News commenter, self-described, Sep 2, 2026. Gemma 4 31B (Gemma4:31b) in the author's local agentic loops, and on an M4 Mac.
- Favorable
Gemma4:31B is 6gb larger and does a much better job across all areas.
New iMac with M6. ActorNightly, Hacker News commenter, self-described, Oct 28, 2026. Gemma 4 31B (Gemma4:31B) against Qwen 3.8 27B, on GPUs and on an M1 Mac.
- Favorable
Simple tasks like data extraction or trivial refactors (rename this variable etc) I often push down to Terra or Luna, or even to self-hosted Gemma4:31B.
Why isn't the industry freaking out about DeepSeek 4.1 Flash?. aftbit, Hacker News commenter, self-described, Oct 9, 2026. self-hosted Gemma 4 31B (Gemma4:31B) for simple tasks alongside GPT 5.6 tiers.
- Unfavorable
I find Qwen 3.8 better at coding but I also use Gemma 4 31B.
Qwen 3.8 Omni Flash. verst, Hacker News commenter, self-described, Sep 18, 2026. Gemma 4 31B in Ollama, toggled against Qwen 3.8.
- Favorable
In our business benchmarks their Mistral Medium 3.5 with reasoning is worse than Gemma 4 31B and Glimmer 30B.
Mistral raises €3B. nik736, Hacker News commenter, self-described, Sep 8, 2026. Gemma 4 31B in the author's company business benchmarks against Mistral Medium 3.5.
- Unfavorable
But, then I noticed that it was missing birds that were at the back of the pool and I needed to use higher resolution and better models
I turned my security cameras into an automatic bird identification system. sahn44, Hacker News commenter, self-described, Sep 1, 2026. gemma-4-31B-it (Gemma 4 31B) classifying bird images from security cameras every 2 minutes.
- Favorable
The gemma4 26/31b models have been deeply impressive on weird prose analysis tasks that I am working on.
What Happens When the Cost of Intelligence Drops 100x. newAccount2025, Hacker News commenter, self-described, Aug 21, 2026. Gemma 4 31B and 26B on prose analysis tasks, run on Bedrock.
- Unfavorable
and better performance in most tasks I've tried on both models.
Qwen 3.8 27B. plasticchris, Hacker News commenter, self-described, Aug 15, 2026. Gemma 4 31B in vLLM against Qwen 3.6 and 3.8 27B.
- Favorable
They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model).
Nvidia Nemotron 3.5 Lightning and NeMo Switchyard. kentonv, Hacker News commenter, self-described, Aug 12, 2026. Gemma 4-31B (Gemma 4 31B) building a collaborative whiteboard app in Cloudflare OS.
- Favorable
I am using it for finding security bugs, though, as the 31b variant is outrageously good at it for its size
I love LLMs, I hate hype. SwellJoe, Hacker News commenter, self-described, Jul 13, 2026. Gemma 4 31B (the 31b variant) finding security bugs.
- Unfavorable
it blows away Gemma 4 31B and Qwen 27B in my tests. It's not even close.
Qwen 3.6 27B is the sweet spot for local development. SwellJoe, Hacker News commenter, self-described, Jul 1, 2026. Gemma 4 31B against Gemma 4 12B on the author's vision tests.
- Favorable
It's one of the better models out there for consumer hardware. Allows me to run it on a 7900XTX with 64k context.
Qwen 3.8. akazantsev, Hacker News commenter, self-described, Jul 19, 2026. Gemma 4 31B QAT on an AMD 7900XTX with 64k context.
- Favorable
seems to follow instructions quite well as a research assistant
Kagi Changelog (July 2): Heads, tails, and an AI toggle. quinncom, Hacker News commenter, self-described, Jul 4, 2026. Gemma 4 31B hosted at Cerebras in Kagi Assistant as a research assistant.
- Favorable
All I can say is locally, quantized in a 24GB scenario, Gemma 4 31B is better in my tests which are mostly reasoning or C programming related.
Qwen 3.6 27B is the sweet spot for local development. CMay, Hacker News commenter, self-described, Jun 29, 2026. Gemma 4 31B QAT quantized locally in 24GB on reasoning and C programming tests.
- Favorable
I tested Gemma 4 31b for OCR and it's very good at it.
Gemma 4 12B: A unified, encoder-free multimodal model. staticman2, Hacker News commenter, self-described, Jun 4, 2026. Gemma 4 31B for OCR.
- Favorable
Gemma 4 31B is enormously impressive.
Gemma 4 12B: A unified, encoder-free multimodal model. qingcharles, Hacker News commenter, self-described, Jun 4, 2026. Gemma 4 31B on the free tiers of Google's API and OpenRouter.
- Unfavorable
For coding Qwen seems to be pretty far ahead
Gemma 4 12B: A unified, encoder-free multimodal model. embedding-shape, Hacker News commenter, self-described, Jun 3, 2026. Gemma 4 31B against Qwen 3.6 27B for programming.
- Favorable
Gemma 4 31b is better for coding than Gemini in my limited testing on a small C project single source file project
Google's Antigravity bait and switch. fluffyspork, Hacker News commenter, self-described, May 21, 2026. Gemma 4 31B on a small single-file C project at temperature 0.
- Unfavorable
Even the 9B model beats Gemma 4 31B in my use case.
Qwen 3.7 Preview. tredre3, Hacker News commenter, self-described, May 18, 2026. Gemma 4 31B at bf16 and Q8 describing images.
- Favorable
Gemma 31b just solved a physics problem Opus 4.7 gave up on.
Running local models on an M4 with 24GB memory. thot_experiment, Hacker News commenter, self-described, May 11, 2026. Gemma 4 31B (Gemma 31b) against Opus 4.7 on a physics problem.
- Favorable
I consider Gemma 4 31B (dense / no MoE), the new baseline for local models.
Running local models on an M4 with 24GB memory. soganess, Hacker News commenter, self-described, May 11, 2026. Gemma 4 31B dense on an M5 Max with 128 GB and the full 256K context window.
- Favorable
I have been using Gemma 4 31B a lot and have found it very competent.
ZAYA1-8B matches DeepSeek-R1 on math with less than 1B active parameters. simjnd, Hacker News commenter, self-described, May 7, 2026. Gemma 4 31B used heavily with deliberate prompting.
- Favorable
Gemma 4 31B is fantastic.
Accelerating Gemma 4: faster inference with multi-token prediction drafters. msp26, Hacker News commenter, self-described, May 5, 2026. Gemma 4 31B on a 24GB 4090 with vision.
- Favorable
And I've been using this commonly as a test when changing various parameters, so I've run it several times, these models get it consistently right.
Claude Opus 4.7. lambda, Hacker News commenter, self-described, Apr 16, 2026. Gemma 4 31B run locally on a Strix Halo laptop GPU with 128 GiB.
- Favorable
It's a great personal assistant.
Ask HN: Who is using OpenClaw?. mv4, Hacker News commenter, self-described, Apr 15, 2026. google/gemma-4-31b-it (Gemma 4 31B) as the primary model of a personal assistant agent with Telegram, email and Google Docs.
- Favorable
I'd say it is smarter than GPT-4o
Claude may require identity verification in some cases. fy20, Hacker News commenter, self-described, Apr 15, 2026. Gemma 4 31B Q6 on a MacBook Pro M5 with 48GB.
- Favorable
Former is at around 20 tok/s, the latter is at around 90tok/s but distinctly worse in my benchmarks.
X post by @mishafrenkman. Misha Frenkman (mishafrenkman), founder of reado.app per his X bio, self-described, Oct 27, 2026. Gemma 4 31B against Gemma 4 26B-A4B in the author's own benchmarks for chat, text generation and file management.
Tool use and structured output 67%
- Favorable
Gemma4 31B is not coding model but it is excellent at intent understanding and task execution used in agentic software.
Qwen 3.8 27B available on Cerebras at 1500 tokens/s. dshat, Hacker News commenter, self-described, Sep 3, 2026. Gemma 4 31B in agentic business software on the Cerebras PayGo plan.
- Favorable
I've been comparing Gemma 4 (31B at 8-bits, also very good with tools and reasoning for its size)
Granite 4.1: IBM's 8B Model Matching 32B MoE. SwellJoe, Hacker News commenter, self-described, Apr 30, 2026. Gemma 4 31B at 8-bit, compared with Qwen 3.6 27B, Claude Opus and Gemini Pro.
- Favorable
Gemma 4 31b outperformed Gemini 3.1 Flash-Lite in our app benchmarks (agentic tool use via api in our application as a part of various workflows).
Gemma 4 12B: A unified, encoder-free multimodal model. mchusma, Hacker News commenter, self-described, Jun 3, 2026. Gemma 4 31B in the author's app benchmarks for agentic tool use over an API.
- Favorable
it just went at it, writing more and more specific diagnostic logging/probing tools over the course of like 8 turns.
Running local models on an M4 with 24GB memory. thot_experiment, Hacker News commenter, self-described, May 11, 2026. Gemma 4 31B running agentically on the author's GPUs to reverse engineer a Bluetooth protocol.
- Unfavorable
It can be a bit unstable and start spiraling or end up in infinite loops that you need to reset
ZAYA1-8B matches DeepSeek-R1 on math with less than 1B active parameters. simjnd, Hacker News commenter, self-described, May 7, 2026. Gemma 4 31B used heavily with deliberate prompting.
- Unfavorable
confuses Gemma-4-31B on every new session, into thinking every tool must be used with
"Code Mode" instructions are confusing Gemma-4-31B. cmoncure, GitHub issue author, self-described, Oct 6, 2026. Gemma-4-31B (Gemma 4 31B) in OpenCode v2.0.24 Code Mode.
Long context 20%
- Unfavorable
My experience using both in vllm supports, with considerably more overhead in context size on these models than Gemma 4 31b
Qwen 3.8 27B. plasticchris, Hacker News commenter, self-described, Aug 15, 2026. Gemma 4 31B in vLLM against Qwen 3.6 and 3.8 27B.
- Favorable
I personally found in my work (long non-programming business-related context, 150k+) that only Gemma 4 31B full precision and full context works
Kimi K2.7 Code is generally available in GitHub Copilot. pbgcp2026, Hacker News commenter, self-described, Jul 4, 2026. Gemma 4 31B at full precision and full context on long business documents of 150k+ tokens.
- Unfavorable
with gemma 4 31b IQ4_XS quant I have to use Q8_0 cache to fit 262k context on the vram, which is a little annoying
Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model. RandyOrion, Hacker News commenter, self-described, Apr 23, 2026. Gemma 4 31B IQ4_XS in llama.cpp on 32GB of VRAM at 262k context.
- Unfavorable
When at higher contexts (say 50k+) MTP on VLLM will be very slow.
[Bug]: Gemma 4 31B MTP - Slower T/S at high context. frenzybiscuit, GitHub issue author, self-described, Aug 12, 2026. Gemma 4 31B with MTP on vLLM on four RTX 3090 cards.
- Unfavorable
At 128K, llama.cpp holds 21.9 tok/s while MLX drops to 12.6: MLX keeps only 31% of its 4K speed on that model.
X post by @sf_tristanb. Tristan (sf_tristanb), CTO @ Wamiz (Nestlé) per his X bio, self-described, Oct 27, 2026. Gemma 4 31B on llama.cpp and MLX at 4K and 128K context.
Developer experience Only concerns on record
- Unfavorable
Spec decode (DFlash, n-gram) didn't give a good boost which is annoying
So you want to use OpenRouter?. msp26, Hacker News commenter, self-described, Sep 11, 2026. Gemma 4 31B NVFP4 on vLLM 0.29 on a single B200 for a batch vision workload.
- Unfavorable
now that they no longer offer it, i find myself reaching for a model a lot less
Qwen 3.8 27B available on Cerebras at 1500 tokens/s. laweijfmvo, Hacker News commenter, self-described, Sep 4, 2026. Gemma 4 31B served on Cerebras through Kagi.
- Unfavorable
I'm saddened that Gemma4 is replaced by Qwen 3.8 on PayGo plan.
Qwen 3.8 27B available on Cerebras at 1500 tokens/s. dshat, Hacker News commenter, self-described, Sep 3, 2026. Gemma 4 31B in agentic business software on the Cerebras PayGo plan.
- Unfavorable
But google won't let you pay to use Gemma models, you have to go elsewhere
Gemma 4 12B: A unified, encoder-free multimodal model. mchusma, Hacker News commenter, self-described, Jun 3, 2026. Gemma 4 31B in the author's app benchmarks for agentic tool use over an API.
- Unfavorable
but it was consuming tons of memory on SWA checkpoints, I had to turn them way down
Granite 4.1: IBM's 8B Model Matching 32B MoE. lambda, Hacker News commenter, self-described, Apr 30, 2026. Gemma 4 31B in llama.cpp on a Strix Halo.
- Unfavorable
just tried the google/gemma-4-31B-it through HuggingFace (inference provider seems to be Novita) and function/tool calling was not enabled
Google releases Gemma 4 open models. james2doyle, Hacker News commenter, self-described, Apr 2, 2026. google/gemma-4-31B-it (Gemma 4 31B) through Hugging Face with Novita as the inference provider.
- Unfavorable
The current template scales quadratically with message count.
Feature Request: gemma4 31B chat template quadratic scan cost. hasanyazarr, GitHub issue author, self-described, Oct 3, 2026. the Gemma 4 31B chat template in llama.cpp on a Mac M4 Pro.
Cost Only praise on record
- Favorable
are affordable for infrequent tasks
LLMs: Intelligence vs. Cost. quinncom, Hacker News commenter, self-described, Sep 2, 2026. Gemma 4 31B (xhigh) running on Cerebras for a few tasks a day.
- Favorable
I had been using gemma-4-31B-it llm for cost/accuracy and it was super cheap, about $0.65/month.
I turned my security cameras into an automatic bird identification system. sahn44, Hacker News commenter, self-described, Sep 1, 2026. gemma-4-31B-it (Gemma 4 31B) classifying bird images from security cameras every 2 minutes.
- Favorable
You get 1000 requests/day for free on Google's API and another 1000/day off OpenRouter.
Gemma 4 12B: A unified, encoder-free multimodal model. qingcharles, Hacker News commenter, self-described, Jun 4, 2026. Gemma 4 31B on the free tiers of Google's API and OpenRouter.
- Favorable
Gemma 4 31B is $0.12/M input, $0.37/M output vs $1/M input, $5/M output for Haiku.
Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap). nl, Hacker News commenter, self-described, May 22, 2026. Gemma 4 31B on OpenRouter as a cheaper option to Claude Haiku.
- Favorable
Or use the paid version from Vertex AI which is around 3x cheaper than Gemini 3 Flash.
Accelerating Gemma 4: faster inference with multi-token prediction drafters. fomoz, Hacker News commenter, self-described, May 6, 2026. Gemma 4 31B in the author's app with 5 agents at 1.5k requests a day each.
- Favorable
FWIW in my recent testing I couldn't find a better model than Gemma 4 31B for the price (openrouter only).
Qwen3.6-Max-Preview: Smarter, Sharper, Still Evolving. chrisss395, Hacker News commenter, self-described, Apr 21, 2026. Gemma 4 31B on OpenRouter identifying business ideas from discussions.
- Favorable
Gemma 4 31B is impressive, but it costs practically nothing via Openrouter.
Codex for almost everything. adamsmark, Hacker News commenter, self-described, Apr 17, 2026. Gemma 4 31B through OpenRouter.
Speed 53%
- Favorable
On the largest model (gemma4:31b), I seem to be getting about 5 tokens per second. This is amazing to me
Google releases Gemma 4 open models. mikewarot, Hacker News commenter, self-described, Apr 3, 2026. Gemma 4 31B (gemma4:31b) in Ollama on a $100 computer with no GPU and a 200 GB swap file.
- Unfavorable
It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow
New iMac with M6. ActorNightly, Hacker News commenter, self-described, Oct 27, 2026. Gemma 4 31B (Gemma4:31b) on a refurbished RTX 3090 in an eGPU dock.
- Unfavorable
On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things.
My local model setup on an M4 Pro Mac Mini. ActorNightly, Hacker News commenter, self-described, Sep 2, 2026. Gemma 4 31B (Gemma4:31b) in the author's local agentic loops, and on an M4 Mac.
- Unfavorable
Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.
New iMac with M6. ActorNightly, Hacker News commenter, self-described, Oct 28, 2026. Gemma 4 31B (Gemma4:31B) against Qwen 3.8 27B, on GPUs and on an M1 Mac.
- Favorable
Kagi used to serve gemma4 31b on cerebras, and i got so used to the speed that i basically stopped using other models.
Qwen 3.8 27B available on Cerebras at 1500 tokens/s. laweijfmvo, Hacker News commenter, self-described, Sep 4, 2026. Gemma 4 31B served on Cerebras through Kagi.
- Favorable
I have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy.
Qwen 3.8 27B available on Cerebras at 1500 tokens/s. freehorse, Hacker News commenter, self-described, Sep 3, 2026. Gemma 4 31B served through Kagi.
- Favorable
On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative decoding methods).
Qwen3.8 27B scores 52 on Artificial Analysis. trouve_search, Hacker News commenter, self-described, Aug 17, 2026. Gemma 4 31B on vLLM and llama.cpp with speculative decoding on dual RTX 5090s.
- Unfavorable
Gemma 4 31B is much slower on pre-Blackwell GPUs as a result, which is a bit of a shame for local model experimentation.
Qwen 3.8 27B is excellent, but it defaults to overthinking things. dofm, Hacker News commenter, self-described, Aug 17, 2026. Gemma 4 31B on pre-Blackwell GPUs for local experimentation.
- Favorable
You can get easily over 100 tok/s on the Gemma 4 31B QAT if you enable MTP.
Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution. NorwegianDude, Hacker News commenter, self-described, Jul 29, 2026. Gemma 4 31B QAT with MTP on a single RTX 5090.
- Unfavorable
Because it is dog slow at 8-bit quantization, but shockingly smart and effective for such a tiny model.
My first impressions on ROCm and Strix Halo. SwellJoe, Hacker News commenter, self-described, Apr 19, 2026. Gemma 4 31B with reasoning at 8-bit quantization on a Strix Halo.
- Unfavorable
Yes it's between this and Gemma 4 31B which is much slower
Qwen 3.8. ernsheong, Hacker News commenter, self-described, Jul 19, 2026. Gemma 4 31B against an MoE model the author runs.
- Favorable
I thought something was short-circuiting due to the speed of response, I'd never seen a model operate that fast.
Kagi Changelog (July 2): Heads, tails, and an AI toggle. mroche, Hacker News commenter, self-described, Jul 4, 2026. Gemma 4 31B on Cerebras, after it was added.
- Favorable
Gemma 4 31B as hosted by Cerebras is almost too fast!
Kagi Changelog (July 2): Heads, tails, and an AI toggle. aryonoco, Hacker News commenter, self-described, Jul 4, 2026. Gemma 4 31B as hosted by Cerebras through Kagi.
- Favorable
Kagi Assistant now offers Gemma 4 31B hosted at Cerebras. It's insanely fast
Kagi Changelog (July 2): Heads, tails, and an AI toggle. quinncom, Hacker News commenter, self-described, Jul 4, 2026. Gemma 4 31B hosted at Cerebras in Kagi Assistant as a research assistant.
- Favorable
I've seen 95-100 tokens per second on M5 Max 128GB running Gemma 4 31B.
Apple Silicon costs more than OpenRouter. avidphantasm, Hacker News commenter, self-described, May 17, 2026. Gemma 4 31B on an M5 Max with 128GB.
- Favorable
In my testing the Gemma 4 31b model had the biggest speed boost in Ollama w/ the MLX runner for coding tasks (at about 2x).
Accelerating Gemma 4: faster inference with multi-token prediction drafters. Patrick_Devine, Hacker News commenter, self-described, May 5, 2026. Gemma 4 31B with MTP in Ollama's MLX runner on coding tasks.
- Unfavorable
as a 31b dense model is fairly slow on a Strix Halo
Granite 4.1: IBM's 8B Model Matching 32B MoE. lambda, Hacker News commenter, self-described, Apr 30, 2026. Gemma 4 31B in llama.cpp on a Strix Halo.
- Unfavorable
Gemma 4 31B Q6: 9tok/s, I'd say it is smarter than GPT-4o, but yeah it's slow.
Claude may require identity verification in some cases. fy20, Hacker News commenter, self-described, Apr 15, 2026. Gemma 4 31B Q6 on a MacBook Pro M5 with 48GB.
- Unfavorable
decode at 0.8–1.6 tokens/s on an M2 Ultra (192 GB)
MLX runner: bf16 Gemma 4 decodes at ~1 tok/s on M2 Ultra. dev-001-jpres, GitHub issue author, self-described, Oct 6, 2026. Gemma 4 31B (gemma4:31b-mlx-bf16) in Ollama's MLX runner on an M2 Ultra.
Reliability Only concerns on record
- Unfavorable
Only problem is you get 503 like crazy.
Gemma 4 12B: A unified, encoder-free multimodal model. qingcharles, Hacker News commenter, self-described, Jun 4, 2026. Gemma 4 31B on the free tiers of Google's API and OpenRouter.
- Unfavorable
llama-server will crash on the 2nd prompt in the session.
Eval bug: Gemma 4 crashes with MTP. gordan-bobic, GitHub issue author, self-described, Jul 10, 2026. Gemma 4 31B with MTP (Unsloth QAT Q4_K_XL) in llama-server on two 2080Ti cards.
- Unfavorable
reliably enters an infinite generation loop
Eval bug: Gemma 4 31B enters <unused49> infinite generation loop after long idle period. sagitario-jpn, GitHub issue author, self-described, Jul 24, 2026. Gemma 4 31B GGUF quantizations from two sources in llama-server on two RTX 5060 Ti cards.
- Unfavorable
The model was working fine previously, but after updating llama.cpp to the latest master, llama-server fails to start inference and crashes with Metal Out-Of-Memory (OOM) errors.
[Bug] macOS Metal OOM & Compute error (-3) on Gemma 4 31B after update. pyroxenites, GitHub issue author, self-described, Sep 27, 2026. unsloth gemma-4-31B-it GGUF (Gemma 4 31B) UD-Q8_K_XL in llama-server on an M5 Max with 128 GB.
- Unfavorable
My vLLM server crashed when the feature is enabled.
[Bug]: Gemma 4 KVCache CPU offloading broken. dannawang0221, GitHub issue author, self-described, May 11, 2026. google/gemma-4-31B-it (Gemma 4 31B) with vLLM native CPU KV cache offloading.