llms.blog
AI

French startup Kog says smarter software can pull 30x faster inference out of stock GPUs

The race for faster AI inference has pushed labs toward custom silicon, but a French startup argues the cheapest speedup is already sitting in servers companies bought. Kog says it can reach roughly thirty times faster LLM inference on standard data-center GPUs using software alone. Kog demonstrated three thousand tokens per second per request on AMD MI300X and NVIDIA H200 GPUs in May, running a small two-billion-parameter model it has since open-sourced as Laneformer 2B. Chief executive Gael D

axc
axc
2 min
French startup Kog says smarter software can pull 30x faster inference out of stock GPUs

The race for faster AI inference has pushed labs toward custom silicon, but a French startup argues the cheapest speedup is already sitting in servers companies bought. Kog says it can reach roughly thirty times faster LLM inference on standard data-center GPUs using software alone.

Kog demonstrated three thousand tokens per second per request on AMD MI300X and NVIDIA H200 GPUs in May, running a small two-billion-parameter model it has since open-sourced as Laneformer 2B. Chief executive Gael Delalleau told TechCrunch the preview drew about two hundred qualified business leads.

Software unlocks GPU headroom

The first target is software engineering. Claude Code users already wait hours for long runs, and Anthropic charges a premium for Fast Mode, so speed is something teams pay for. Kog is pitching shops that run AI workflows for paid work and cannot absorb those delays.

The bigger promise is thirty x on large models, but today it rests on a tiny demo. Delalleau expects to show a major model running at ten x speed in September, a milestone he says will unlock customer traction and a Series A raise. The approach is deliberately low-level.

Delalleau, a former white-hat hacker and Ecole Polytechnique physics graduate, says his team reverse-engineers each GPU down to assembly and binary to exploit unused memory bandwidth. The cost is time: weeks to months per chip with a team of eleven, which caps how many accelerators Kog can support.

Kog is not alone. France's ZML ships hardware-agnostic inference software that bypasses CUDA, and Stanford's Hazy Research explores similar GPU acceleration. Kog compares itself to Hazy Research but goes deeper. Backers include Scaleway, Bpifrance, and French Tech 2030, giving it a European sovereignty tailwind.

The open question is whether hand-tuned, per-chip optimization scales to the models customers actually want. If Kog hits ten x on a frontier model in September, the assumption that GPUs are already maxed out gets a serious challenge.

Sources

TechCrunch: https://techcrunch.com/2026/08/14/kog-is-going-deeper-to-squeeze-more-inference-out-of-gpus/

Kog tech preview: https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/

Laneformer 2B (Hugging Face): https://huggingface.co/kogai/laneformer-2b-it

AILLMinferenceStartupsGPU
ShareX (Twitter)LinkedIn

Written by

axc

More to read

  • Hallucination Detection and Faithfulness Verification in Production RAG: Architecture, NLI Claim Decomposition, and Runtime Guardrail Economics

    Retrieval-Augmented Generation (RAG) is commonly deployed under the assumption that grounding generation in retrieved passages eliminates factual inaccuracies. In practice, grounding provides an evidence boundary but does not guarantee factual fidelity. Production language models regularly synthesize claims absent from the retrieved context (extrinsic hallucinations) or directly assert statements conflicting with retrieved premises (intrinsic contradictions). As enterprise RAG pipelines scale i

    1 min

  • Sparse Attention and BigBird: How Window, Global, and Random Graphs Preserve Turing Completeness in Linear Time

    Standard self-attention in transformer architectures scales quadratically with sequence length. Computing full pairwise interactions between n tokens requires evaluating an n x n attention matrix, yielding O(n^2) computational complexity and memory consumption. While hardware accelerators and IO-aware tiling algorithms like FlashAttention optimize memory traffic, the quadratic compute and KV footprint remains a barrier for processing long contexts, document-level summarization, and genomic seque

    1 min

  • Oxford Study Details Chinese Gray-Market Proxies Reselling Claude Tokens at 90% Discounts

    An investigation by the Oxford China Policy Lab reveals that Chinese developers routinely access Anthropic's frontier Claude models at discounts between 70% and 90% below list price, bypassing geographical blocks, payment filters, and biometric identity verification through a decentralized network of API proxies known locally as "transfer stations" (中转站). The analysis, authored by Oxford researcher Zilan Qian and published via ChinaTalk, outlines the modular supply chain and economic mechanics

    1 min