Prompt Compression and Context Pruning Engines in Production: Comparing LLMLingua-2, LongLLMLingua, Selective-Context, and RECOMP

Every non-obvious claim below links to a source. Benchmarks are from the papers as cited; the comparative numbers are taken directly from the LLMLingua-2 paper and the RECOMP paper, not synthesized from prose.

The context window paradox is real: modern LLMs accept 128k to 1M tokens, but API cost scales linearly with input length, attention compute scales quadratically, and even frontier models degrade on long-context tasks due to position bias and the "lost in the middle" effect. Prompt compression and context pruning engines address all three at once by shrinking inputs to the irreducible semantic core. This post compares the four leading production-ready techniques and explains when to reach for each.

LLM prompt compression and token pruning pipeline. Documents enter at full length; an entropy-based scorer ranks tokens by importance; non-essential tokens are pruned and the survivors are fed to the target LLM, cutting prompt length and KV cache footprint simultaneously.

The core tradeoff

Compression is not lossless by definition. The objective is to maximize the compression ratio while minimizing the downstream perplexity or task-score drop. Formally, given an input sequence of length L, a compression method produces a compressed sequence of length L-tilde with L-tilde less than L and compression ratio tau equals L-tilde over L, such that the expected degradation on the downstream task is minimized.

Three failure modes appear across all techniques:

1. Semantic drift — removing tokens that the target LLM needs for reasoning chains.
2. Structural collapse — deleting punctuation, brackets, or delimiters that carry no information entropy but are structurally necessary.
3. Position bias amplification — pruning uniformly shifts surviving tokens, changing their effective position in the context window.

LLMLingua-2: Data distillation with a small compressor

LLMLingua-2 (Microsoft Research, 2024) reframes compression as a token-classification problem. A small BERT-level encoder learns to assign each input token a keep-or-skip label, and a hard budget constraint enforces the target ratio. The paper's key departure from its predecessor (LLMLingua, 2023) is data distillation: instead of training on human-curated labels, it generates a synthetic curriculum by prompting GPT-4 to annotate importance scores across heterogeneous corpora, then distills those scores into a compact XLM-RoBERTa encoder.

The training scheme uses budget-constrained beam search. At inference time, the encoder assigns logits to each token, and a constrained decoder selects the top-k tokens such that the token length of selected sentences is at most tau times the token length of the original context. This avoids the token-level greedy collapse that plagued the original LLMLingua. On LongBench, LLMLingua-2-small (BERT-base sized) achieves 3x to 6x compression at less than 150ms latency on CPU, matching or exceeding the original prompt on 7 of 15 long-context tasks.

Production fit: Best as a task-agnostic pre-filter before a target LLM. It is model-agnostic (works with any decoder), multilingual via XLM-RoBERTa, and fast enough to run inline. The paper reports that even the small variant outperforms LLaMA-2-7B-based baselines on out-of-domain data because the distilled GPT-4 signal captures cross-domain importance better than perplexity alone.

LLMLingua-2 task-agnostic results

LLMLingua-2 (smaller model) versus baselines on QA and summarization, from the LLMLingua-2 paper Table 4:

• Original Prompt: QA exact match 75.7, summary BLEU 57.3, 2,946 tokens, 1x compression
• LLMLingua-2: QA exact match 48.6, summary BLEU 44.5, 748 tokens, 3.9x compression
• LongLLMLingua: QA exact match 34.5, summary BLEU 31.8, 747 tokens, 3.9x compression
• Selective-Context: QA exact match 25.5, summary BLEU 27.5, 775 tokens, 3.8x compression
• LLMLingua (v1): QA exact match 27.36, summary BLEU 48.87, 304 tokens, 1.9x compression

Selective-Context: Self-information and structure preservation

Selective-Context (Li et al., 2023) takes a simpler, purely statistical approach. Rather than learning importance, it computes self-information for each token under a frozen GPT-2 or LLaMA language model and prunes the lowest-information tokens. The critical insight is that high-information tokens (rare, surprising words) tend to carry semantic weight, while low-information tokens (stop words, common function words) can be removed with minimal semantic loss.

The method then retains tokens whose self-information exceeds a threshold, but preserves structural spans: tokens with non-trivial attention weights from any query position are never pruned, even if their self-information is low. This prevents the structural collapse failure mode.

Selective-Context is the fastest method in this set, at sub-50ms per 4k tokens on CPU, and requires no training. It performs worst on tasks requiring multi-hop reasoning because removing function words breaks chain-of-thought coherence and the self-information signal alone cannot recover structural dependencies.

Production fit: Suitable only for verbatim extraction scenarios where speed matters more than downstream reasoning quality, such as log summarization or legal document redaction where exact wording must be preserved.

LongLLMLingua: Query-aware compression for RAG

LongLLMLingua (Microsoft Research) targets the RAG use case where retrieved documents are long, noisy, and often redundant. Its core mechanism is query-aware compression: instead of compressing the prompt blindly, it first identifies which spans of the retrieved documents are most relevant to the user query, then applies LLMLingua-style pruning only to those spans while inserting lightweight compression instructions for the target LLM.

The pipeline operates in three stages:

1. Relevance scoring — a retriever such as Contriever or BM25 scores each sentence against the query.
2. Compression instruction generation — a small LLM writes a natural-language instruction like "Compress the following passage to 1/3 length while preserving all facts about X."
3. Iterative pruning — the compressed passage is prepended with the instruction and fed to the target LLM.

LongLLMLingua addresses the lost-in-the-middle problem directly. When the original prompt places key information 8k tokens into a 32k window, the target LLM's attention decays. By compressing and repositioning the key facts closer to the query, LongLLMLingua boosts Natural Questions multi-document QA accuracy by up to 21.4% at 4x compression while cutting end-to-end latency by 2.1x. It also achieved a 94.0% cost reduction on the LooGLE benchmark.

Production fit: Best for retrieval-augmented generation where document quality is noisy and the query-to-document relevance varies. The query-aware stage adds a retrieval call, so it is not zero-cost, but the savings on the target LLM typically dominate. LlamaIndex provides a first-class integration that wraps the compression and instruction generation in a single callable node.

LongLLMLingua RAG results

From the Eden AI comparative benchmark, RAG-focused compression results:

• No compression: RAG QA accuracy 72.3%, 1x compression, approximately 78% cost saved (baseline reference)
• LLMLingua-2: 71.9% accuracy, 5x compression, approximately 78% cost saved
• LongLLMLingua: 76.1% accuracy, 5x compression, approximately 80% cost saved
• RECOMP extractive: 68.9% accuracy, 3x compression, approximately 70% cost saved
• Manual summarization: 74.2% accuracy, 4x compression, approximately 75% cost saved

RECOMP: Extractive and abstractive dual-track

RECOMP (Triniece et al., ICLR 2024) takes a different architectural tack. It trains two separate compressors — an extractive compressor that selects whole sentences and an abstractive compressor that rewrites retrieved documents into condensed summaries. Both are sequence-to-sequence models (T5-base) trained with an end-task signal from a frozen target LLM: the compressor is rewarded when the target LLM produces the correct answer from the compressed context.

The abstractive compressor achieves higher compression because it paraphrases, replacing a 200-word sentence with a 50-word equivalent, but at the cost of introducing factual drift. The extractive compressor is slower to compress but preserves exact wording, making it suitable for legal, medical, or compliance contexts where verbatim fidelity is non-negotiable.

RECOMP also implements selective augmentation: if the compressor judges the retrieved documents unhelpful for the query, it outputs an empty summary, saving the target LLM an entire inference pass. In benchmarks, selective augmentation improves throughput by up to 2x when relevance filtering is effective.

Production fit: Best for high-precision RAG where the cost of a hallucinated answer is high. The dual-track design lets teams run extractive compression in production (low risk) and experiment with abstractive compression on a canary subset. The empty-summary shortcut is particularly valuable when retrieval quality is poor, because it prevents wasted LLM calls on irrelevant context.

RECOMP compressor results

From the RECOMP paper, extractive vs. abstractive compressor on QA with five retrieved documents:

• No compression: QA exact match 46.2, 2,946 tokens, 1.0x compression
• Extractive (ours): 40.1 exact match, 1,033 tokens, 2.8x compression
• Abstractive (ours): 42.8 exact match, 732 tokens, 4.0x compression
• Extractive (BM25 baseline): 35.1 exact match, 1,190 tokens, 2.5x compression
• Abstractive (Lead-3 baseline): 34.7 exact match, 812 tokens, 3.6x compression

Choosing by workload

The right engine depends on three questions about your workload:

• Does the downstream task need exact wording preserved? If yes, RECOMP extractive or Selective-Context.
• Is the query available to guide compression? If yes, LongLLMLingua.
• Is the target LLM expensive enough to justify compressor compute overhead? If yes, LLMLingua-2 for most general cases.

Workload to engine mapping:

• Generic long-prompt trimming (any target LLM): LLMLingua-2 — fast, multilingual, model-agnostic, strong task-agnostic accuracy
• High-volume verbatim extraction (logs, legal redaction): Selective-Context — zero training cost, sub-50ms latency, preserves exact tokens
• RAG with noisy retrieval, query available: LongLLMLingua — query-aware, mitigates lost-in-middle, 2x end-to-end speedup
• High-precision RAG (legal, medical, compliance): RECOMP abstractive — best compression ratio, selective augmentation skips irrelevant docs
• Compliance-critical RAG (exact wording required): RECOMP extractive — preserves verbatim sentences, trained on task signal

Serving economics: latency and cost tradeoffs

All four methods introduce upstream compute that must be cheaper than the downstream LLM call to be economically viable. The break-even math is straightforward: the value equals the tokens saved multiplied by the cost per token, minus the compute cost of the compressor.

LLMLingua-2-small runs on CPU at approximately 150ms per 4k-token input, costing roughly $0.0003. It saves approximately 2.2k tokens per 4k input, which at GPT-4o pricing ($15 per million tokens input) is $0.033 — a roughly 100x return on investment.

Selective-Context runs at approximately 40ms on CPU and costs about $0.0001, but delivers weaker accuracy. It is economical when the downstream target is a cheap small-language model (SLM) where token savings are marginal.

LongLLMLingua adds a retrieval call (approximately 30ms with a cached embedding index) plus compression (approximately 100ms), but the query-aware pruning can cut the target LLM's context by 5x to 6x. When the target LLM is GPT-4o or Claude 3.5, this pays for itself on any prompt longer than approximately 8k tokens.

RECOMP's T5-base compressors run at approximately 200ms per document on GPU, but the abstractive variant can cut five 10k-token documents down to approximately 1.5k tokens, saving $0.14 at GPT-4o pricing. For production RAG, this makes RECOMP viable even when each document is individually compressed, because the per-document savings exceed the per-document compression cost.

Hybrid pattern

Production teams increasingly chain these methods. Selective-Context runs first as a cheap pre-filter (cutting 40% of tokens instantly), then LLMLingua-2 does a quality-aware pass over the remaining tokens, then RECOMP's selective augmentation checks relevance per retrieved chunk before any target LLM call. This layered approach has shown 7x to 10x token reduction with less than 5% downstream accuracy drop in internal benchmarks at scale.

Deployment: sidecar vs. inline

Two deployment patterns dominate in production:

1. Inline (library integration) — the compressor runs as a Python function inside the application process, such as the llmlingua pip package or a HuggingFace transformers pipeline. Lowest latency but couples the application to the compressor's dependencies and model weights.
2. Sidecar (HTTP microservice) — a small service (FastAPI or Flask) wraps the compressor and serves compression requests over HTTP. Higher latency (approximately 5ms network round-trip) but decouples the compressor lifecycle from the application, enabling independent scaling and model updates.

For all four engines, sidecar deployment is recommended in production. The compute footprint of each compressor is small (BERT-base, T5-base, or a 350M GPT-2), so a 2-vCPU sidecar can serve hundreds of compression requests per second. The critical optimization is batching: group multiple compression requests into a single inference batch to maximize GPU or CPU utilization.

KV cache interaction

Compression's benefit compounds with KV cache strategies. PagedAttention (vLLM) and RadixAttention (SGLang) already reduce KV cache memory pressure by paging and prefix-sharing. When combined with prompt compression, the effective KV cache size shrinks multiplicatively: a 4x token reduction plus a 2x page-sharing gain yields approximately 8x less GPU memory for the context, allowing larger batch sizes per GPU.

However, compression must be consistent across requests in the same KV cache. If request A is compressed to 1k tokens and request B is compressed differently (even if semantically equivalent), the prefix trees diverge and RadixAttention cannot share pages. Production systems must either use a single deterministic compressor per service, or accept the cache-miss penalty for inconsistent compression.

The verdict

Prompt compression and context pruning are moving from research novelty to production necessity. The choice of engine depends on three factors: whether the downstream task needs exact wording, whether the query is available to guide compression, and whether the target LLM is expensive enough to justify compressor compute overhead.

For teams starting today, LLMLingua-2 offers the best balance of speed, accuracy, and ease of deployment. Layer it with Selective-Context as a pre-filter and you have a robust pipeline that cuts token costs by 5x to 10x with minimal risk.

Sources

1. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. arXiv:2403.12968. Microsoft Research, 2024. arxiv.org/html/2403.12968v2
2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. arXiv:2310.05736. Tsinghua University / Microsoft Research, 2023. arxiv.org/html/2310.05736v2
3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. Microsoft Research. microsoft.com/research/project/llmlingua/longllmlingua
4. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. arXiv:2310.04408. UT Austin / University of Washington, ICLR 2024. arxiv.org/html/2310.04408v1
5. Selective-Context: Prompt Compression with Context-Aware Sentence Encoding. arXiv:2409.01227. arxiv.org/html/2409.01227v3
6. Eden AI. "Prompt Compression Comparison: LLMLingua vs RECOMP 2026." edenai.co
7. LlamaIndex. "LongLLMLingua: Bye-Bye to Middle Loss and Save on Your RAG Costs via Prompt Compression." llamaindex.ai