llms.blog
AI

DeepSeek V4 Flash tops charts but fails half its real agent tasks

DeepSeek's V4 Flash has become the most-used AI model on OpenRouter and one of the highest-rated open-weight models available. In real agent tests, though, it finished barely more than half of the jobs it was given. The gap between leaderboard and workplace is the story. What Composio found The integration company Composio ran V4 Flash through eight agent harnesses, including Claude Code, Codex, and OpenCode, on 30 deliberately hard multi-step tasks. The tasks used live tools: Gmail, GitHub,

axc
axc
2 min
DeepSeek V4 Flash tops charts but fails half its real agent tasks

DeepSeek's V4 Flash has become the most-used AI model on OpenRouter and one of the highest-rated open-weight models available. In real agent tests, though, it finished barely more than half of the jobs it was given. The gap between leaderboard and workplace is the story.

What Composio found

The integration company Composio ran V4 Flash through eight agent harnesses, including Claude Code, Codex, and OpenCode, on 30 deliberately hard multi-step tasks. The tasks used live tools: Gmail, GitHub, Slack, and Google Sheets.

A conceptual illustration of an AI model solving a few tasks while leaving many unsolved

Across 240 runs, 129 passed, a completion rate of 53.8 percent. Only six of the 30 workflows were finished successfully by every harness tested.

Same model, different results

The same model produced very different scores depending on the harness, configuration, caching, and retries. That points to a shift in what matters for enterprise use. Raw model quality still sets the ceiling, but orchestration, the scaffolding that wires a model to tools, often decides whether a task actually gets done.

Prices are moving too

DeepSeek released V4 Flash in public beta on July 31 and made the larger V4 Pro generally available on August 13. Flash runs on 284 billion parameters and is built for speed and volume. Pro uses 1.6 trillion parameters for heavier workflows.

Both models are now getting more expensive. DeepSeek said it will raise prices for V4 Flash and V4 Pro, and reporting puts some increases at roughly four times the earlier cost. The low price was a big part of the model's appeal, so the move changes the math for teams that picked it to cut bills.

How it got here

V4 Flash took over OpenRouter's usage leaderboard the moment it shipped and remains the platform's most-used model by weekly token volume. ML researcher Nathan Lambert called the early adoption numbers insane and noted the model scored about the same as GLM 5.2. Epoch AI rated it an ECI of 153, between Anthropic's Opus 4.5 and 4.6, and the second strongest open-weight model behind Moonshot's Kimi K3.

The result is a more nuanced picture of the model. It is cheap, popular, and strong on benchmarks, yet real agent work still breaks roughly half the time. For builders, the lesson is that the harness around the model may matter as much as the model itself.

Sources

- VentureBeat: DeepSeek's top-ranked V4 Flash stumbles on real agent tasks as its prices surge (Aug 16, 2026) - https://venturebeat.com/orchestration/deepseeks-top-ranked-v4-flash-stumbles-on-real-agent-tasks-as-its-prices-surge - Composio agent benchmark post on X - https://x.com/composio/status/2085330847951970801 - Engadget: DeepSeek AI models get four times pricier - https://www.engadget.com/2236912/deepseek-ai-models-get-four-times-pricier/ - Epoch AI on X: V4-Flash-0731 ECI 153 - https://x.com/EpochAIResearch/status/2084788991153586600

AIDeepSeekLLMAgentsBenchmark
ShareX (Twitter)LinkedIn

Written by

axc

More to read

  • Mental World Modeling: Why Autonomous AI Agents Fail Without Simulating Human Beliefs

    Mental World Modeling: Why Autonomous AI Agents Fail Without Simulating Human Beliefs Current foundation world models, from video simulators like Sora and Genie to spatial representations like JEPA and Marble, focus almost exclusively on the physical mechanics of an environment. They track geometry, object positions, motion trajectories, and visual continuity. However, when autonomous agents interact with humans in collaborative, medical, or domestic settings, physical state tracking alone cons

    1 min

  • Geo-Distributed LLM Inference in Production: Cross-Region Routing, Wide-Area KV Cache Dynamics, and Data Residency Architecture

    Serving large language models at enterprise scale quickly encounters geographic constraints. When users, microservices, and autonomous agents are distributed across North America, Europe, and Asia-Pacific, routing all traffic to a single centralized GPU cluster introduces significant wide-area network (WAN) round-trip times (RTT). An interactive request from Singapore to a US-East data center carries 180ms to 240ms of baseline network latency before the GPU executes a single token. However, dis

    1 min

  • Reversible Transformers: How Invertible Residual Blocks Eliminate Activation Memory in Deep Networks

    Training deep transformer models is primarily bounded by activation memory rather than parameter storage. During the forward pass of standard backpropagation, automatic differentiation engines cache intermediate activations across every attention head, layer normalization, and feed-forward sublayer so they can be referenced during the backward pass to evaluate gradients. For a transformer with N layers, sequence length L, batch size B, and hidden dimension d_model, storing these activations requ

    1 min