Google's Gemini 3.1 Pro Dominates 12 Benchmarks But Can't Code
Google claims its new Gemini 3.1 Pro is the most powerful language model ever built. The benchmarks say they're right. Reality? It's complicated.
Released as a preview on February 19th, Gemini 3.1 Pro swept 12 benchmark tests, leaving Claude Opus 4.6, GPT-5.2, and even GPT-5.3-Codex in the dust. The standout performance? A jaw-dropping 77.1% on ARC-AGI-2—more than double its predecessor's score on tests designed to measure genuine reasoning ability.
But here's where it gets interesting.
The Real Story
While Google's marketing team celebrates their benchmark victories, the model face-plants on actual programming tasks. SWE-Bench Pro and SWE-Bench Verified—tests that measure real engineering capabilities—revealed Gemini 3.1 Pro's Achilles heel. It can solve abstract logic puzzles but struggles with the bread-and-butter work most developers actually do.
The performance gap tells a familiar story about AI development priorities. Companies optimize for benchmark headlines, not practical utility.
<> "Gemini 3.1 Pro is now at the top of the APEX-Agents leaderboard," gushed Brendan Foody, CEO of AI startup Mercor. "The results demonstrate how quickly agents are improving at real knowledge work."/>
Real knowledge work? Let's examine that claim.
Gemini 3.1 Pro processes 1 million tokens and generates output at 105.8 tokens per second. Impressive specs. But it's also ridiculously verbose—spitting out 57 million tokens during evaluation compared to 12 million for comparable models. That's not efficiency; it's digital diarrhea.
Worse still: 29.16 seconds to first token. In an era where users expect instant responses, half a minute feels like digital death.
Pricing Reality Check
Google's pricing strategy reveals their true confidence level:
- $2.00 per 1 million input tokens
- $12.00 per 1 million output tokens
That's premium pricing for a preview model. Either Google believes they've built something genuinely revolutionary, or they're testing how much enterprises will pay for bragging rights.
The tiered rollout—Pro and Ultra subscribers get priority access through Gemini app and NotebookLM—screams "manufactured scarcity." Classic Silicon Valley playbook: create artificial demand through exclusivity.
What Actually Works
Credit where due: Gemini 3.1 Pro excels at complex reasoning tasks. The Human Final Exam, GPQA Diamond, and ARC-AGI-2 victories aren't flukes. Google's focus on extended thinking and chain-of-thought reasoning produces measurable results.
The model can generate code-based animations and build live dashboards from text prompts. One demo created an International Space Station orbit visualization—genuinely impressive for a language model.
Google's safety evaluations also passed without drama. No CBRN risks, no harmful manipulation concerns, no cyber safety issues. Boring? Maybe. But responsible AI development beats reckless innovation.
The Hype Cycle Continues
We've seen this movie before. Revolutionary AI model launches. Benchmark records fall. Media coverage explodes. Six months later, everyone's moved on to the next "breakthrough."
Gemini 3.1 Pro represents genuine progress in reasoning capabilities. But progress isn't perfection. Until these models can reliably handle basic programming tasks—the actual work that pays developer salaries—the revolution remains incomplete.
Google built a Ferrari for the Autobahn but forgot to teach it city driving. Impressive? Absolutely. Practical for most developers? That's the $12-per-million-token question.
