GAAC: compressing your local LLM's KV cache without losing a single bit

GAAC: Compressing Your Local LLM's KV Cache Without Losing a Single Bit

🔎 Full precision, the last luxury of local inference

Run an LLM locally in 2026 and you're already quantizing everything: the weights to 4 bits, sometimes the activations, and increasingly often the KV cache itself. Every byte saved is paid for in precision, and the bill shows up in responses that drift during long sessions.

A paper published on October 28, 2026 on arXiv tackles the problem from the other end. GAAC (Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices) compresses the KV cache losslessly — bit-for-bit — by recoding the BF16 exponents per channel, then reconstructing them on the fly inside the attention kernel. The advertised results: roughly 26% less resident KV memory, long-context attention up to 1.27x faster, and throughput gains of 13 to 29% at equal memory.

Figures measured where it counts: on an Apple M4 and an NVIDIA DGX Spark, the two archetypes of the personal machine with unified memory. The question remains what they're worth once you step outside the abstract. We went through the paper carefully, limitations included.

The essentials

• Lossless, truly: GAAC compresses the KV cache without altering a single bit — the reconstructed values are identical to the BF16 baseline, so the model's output doesn't change.
• Key figures: −26% KV storage, long-context attention up to 1.27x faster, throughput +13 to 29% at equal memory.
• Testing ground: Apple M4 (1.02x at 8K, 1.10x at 64K with Qwen3-8B, up to 1.25x depending on configurations) and NVIDIA DGX Spark (1.19–1.27x).
• Acknowledged limitations: no prefill acceleration, reduced gains on FP8 (E4M3) KV caches, no energy measurements.
• Status: research paper from October 28, 2026, no integration announced in Ollama, llama.cpp or LM Studio.

Recommended tools

GAAC is not a product: it's a method described in an arXiv paper. Here's the ecosystem you need to know in order to understand it, experiment with it, or get close to it.

Honest clarification: llama.cpp can already quantize the KV cache (Q8_0, Q4_0), but lossily. GAAC's lossless compression, on the other hand, has no public release to date.

Why the KV cache is choking your local LLMs

Because generating text is a bandwidth problem, not a compute problem: for every token produced, your machine re-reads the entire KV cache.

A quick refresher. To produce a token, attention compares the current token against all previous ones. Rather than recomputing keys and values (K/V) at every step, they're stored layer by layer: that's the KV cache. It grows linearly with context, and it gets read in full for every token generated.

Concretely, take Qwen3-8B, the model used in the paper's benchmarks: 36 layers, 8 KV heads (GQA), head dimension 128. In BF16, that's nearly 150 KB per token. At 64k context, the KV cache weighs about 9.5 GB — enough to eat up a third of a 32 GB Mac, just for attention memory.

Meanwhile, your weights are quantized to 4 bits: an 8B model then fits in 4 to 5 GB. Herein lies the paradox of modern local inference: we've compressed the weights to the max, and the full-precision KV cache has become the biggest memory consumer — and the biggest read workload.

Yet the bandwidth of an M4 (around 120 GB/s) is nothing like that of a datacenter GPU (several TB/s). Your decode speed largely boils down to a single equation: bytes to read divided by bandwidth. Cut the bytes, and you gain tokens per second.

If you're starting from scratch, our guide to installing a local LLM in 2026 covers these memory trade-offs in detail, and our pick of the best LLMs to run locally will help you choose a model that won't drown your RAM.

What GAAC Does, in Plain Terms

GAAC targets only the exponents of the BF16 numbers in the KV cache — half the bits of each element — and recodes them without losing a single bit of information.

A bit of structure first. A BF16 number is written across 16 bits: 1 sign bit, 8 exponent bits, 7 mantissa bits. Now, in a KV cache, the exponents are massively redundant: within a given channel, values stay concentrated in a narrow range, so their exponents repeat. This redundancy is fat — trimming it changes nothing about the values.

GAAC encodes these exponents channel by channel. Mantissa and sign remain untouched; only the exponents are compacted. The operation is reversible down to the last bit: when the attention kernel reads the KV cache, it reconstructs the original values on the fly, right before the computation.

The result is therefore bit-for-bit identical to the BF16 baseline. No calibration, no reference dataset, no quality drift over long generations. The model's output is, literally, the same.

What this buys: roughly 26% less KV storage, and above all fewer bytes crossing memory at every decode step. Since long-context attention is bandwidth-bound, those saved bytes convert directly into speed.

The trade-off must be named: 2-bit quantization divides memory by 4, GAAC reduces it by 26%. Lossless gains less — but it costs nothing in quality.

The numbers, without makeup

The gains are real but very uneven: nearly nil at short context on Apple M4, solid at 64K, genuinely interesting on DGX Spark.

Source: GAAC paper, arXiv 2610.38210, October 2026.

Let's start with the inconvenient number: 1.02x at 8K on the M4. Two percent is measurement noise. The reason is simple: at short context, the KV cache is small compared to the weights, and compressing it barely changes the total memory traffic. If your sessions fit within 8K tokens, GAAC won't do anything for you.

The gain grows with context. At 64K, the KV cache dominates the bytes read at each decode step, and the compressed kernel takes a clear lead: 1.10x on the M4, up to 1.25x depending on the configurations tested.

DGX Spark confirms the trend with 1.19 to 1.27x. The machine's memory bandwidth is limited — that's the price of its compactness — and every byte saved shows up directly there. The most telling test is W4A16 decode (4-bit weights, 16-bit activations) at batch 1: when the weights are already compressed, the KV cache accounts for an even larger share of the traffic, and lossless compression pays off more.

The last reading, the most interesting one: throughput at equal memory, +13 to 29%. At constant RAM, the space saved allows for a longer context or a larger batch. For a personal machine, that's often the right way to frame the question.

My verdict: −26% storage and 1.1 to 1.27x on attention — that's not a revolution. It's a free lunch — in the strict sense, since the model's output doesn't change by a single bit. And in local inference, free lunches are rare.

Lossless vs lossy: complementary, not competing

GAAC does not replace KV cache quantization. It plugs in after it, to recover whatever remains unnecessary.

The current consensus is quantize-everything, KV cache included: FP8 at API providers, 2 bits in research work. We have already covered the 2-bit KV quantization open-sourced by Together AI: memory cut by four, but a rounding error that can accumulate over long generations.

GAAC is orthogonal to these approaches. It does not touch the values, it compacts the way they are written. The ideal pipeline therefore chains the two: quantize first for the big gains, compress afterwards for the residual gains — with no additional risk.

And GAAC is not alone. A genuine lossless wave is taking shape:

• SplitZip exploits the same redundancy of BF16 exponents, with throughput of 613.3 GB/s for compression and 2,181.8 GB/s for decompression — faster than memory bandwidth, the sine qua non condition for decompressing on the fly without creating a new bottleneck.
• AdaCode (EMNLP 2026) proposes a bit-exact compression framework for BF16 KV cache.
• Ferguson Finn's analysis of speculative KV coding shows that a predictor model can push lossless compression up to around 4x, and that starting from native FP8 you get down to 2.05–2.59 bits per element, i.e., 3.08 to 3.90x versus raw FP8.

What sets GAAC apart from its cousins: it does not settle for measuring a compression throughput. The paper evaluates the end-to-end gain — kernel included — on consumer hardware, M4 and DGX Spark. It is the only one of the four that answers the question we actually care about: "does it speed up my machine?"

The limitations the paper acknowledges

GAAC doesn't speed up prefill, offers little gain on FP8, and nobody has measured energy consumption — three limitations that seriously frame the scope of the method.

First limitation: the efficiency depends on redundancy. If the exponents of a given model are not very redundant, the compressor yields less. This is not a guaranteed-by-construction gain, it's a statistical gain, verified on the configurations tested in the paper.

Second limitation: FP8. In E4M3, exponents are only 4 bits and are already dense — there's little left to compress losslessly. If FP8 KV caches become the norm, GAAC's BF16 target shrinks. Important nuance: the majority of local stacks still run their KV cache in 16 bits today, llama.cpp included. The window is open, but it won't stay open forever.

Third limitation: prefill is not accelerated. Ingesting a long prompt uses standard attention, unmodified by GAAC. If your bottleneck is time-to-first-token on RAG with 100k tokens to swallow, this method won't change anything.

Fourth limitation, awkward for a paper targeting "personal devices": no energy measurements. Fewer bytes read should reduce power consumption — that's even the natural argument for battery-powered use — but the paper doesn't demonstrate it.

Finally, the meta limitation: this is a research paper. As of this writing (October 2026), no integration has been announced in Ollama, llama.cpp, or LM Studio. The path from arXiv to your Mac is measured in months, not days.

Mac, DGX Spark: why local is the ideal playground

The more constrained your machine's memory bandwidth is, the more value GAAC delivers — and that's exactly the profile of personal inference machines.

Apple Silicon is the perfect illustration: unified memory shared between CPU and GPU, roughly 120 GB/s on a base M4 versus several TB/s for a datacenter GPU. Decoding there is massively memory-bound, not compute-bound. Hence measurable gains as soon as the context grows.

The DGX Spark follows the same logic. Our article on NVIDIA's compact machine with unified memory details its trade-offs: plenty of memory, deliberately constrained bandwidth, a single goal — running large models outside the datacenter. This is precisely the environment where reducing the bytes read pays off more than stacking FLOPS.

It's also the philosophy behind 1-bit LLMs that fit on a smartphone: the frontier of local is no longer compute, it's data movement. GAAC, SplitZip, and KV quantization belong to the same family of techniques — reduce the bytes, not the operations.

Who benefits first? Those who load long contexts locally: agents that accumulate sessions, RAG over documents, code analysis at repository scale. With models like DeepSeek V4 Flash or an MoE such as Qwen3.5-122B-A10B, the KV cache quickly becomes the biggest RAM consumer — and the first optimization lever.

What to do concretely while waiting for an integration

Three actions are worth taking: watch the kernels, experiment with kvpress, and reduce your KV cache at the source.

First reflex: keep an eye out for integrations. llama.cpp and Ollama regularly absorb kernels from research, and GAAC dates from October 28, 2026 — too recent to be integrated, but significant enough to end up in a pull request. Papers that measure gains on M4 have historically been a good omen on the Apple Silicon side.

Second reflex: experiment. NVIDIA/kvpress brings together several KV cache compression methods in an open source library. It's not GAAC, but it's the closest playground available today for understanding what KV compression does to your models.

Third reflex, the most immediately profitable one: reduce the KV cache at the source. Choose architectures with GQA, which already divide the KV cache by 4 to 8 depending on the models. Cut out obese system prompts, prune dead context. And before inflating the context window, ask yourself whether a RAG that reasons wouldn't do better than a garbage context — our article on PageIndex, the vector-free RAG explores this path.

One use case illustrates the stakes well. If you run an AI avatar for customer service locally, long sessions mechanically blow up the KV cache — that's exactly the profile that will benefit first from this type of kernel. To choose the model to put behind it, our comparison of the best LLMs for agents remains the best starting point.

❌ Common Mistakes

Mistake 1: confusing lossless compression with quantization

2-bit KV quantization reduces the precision of values; GAAC restores them bit-for-bit. The orders of magnitude are not comparable (−75% versus −26%), and neither are the risks. The two techniques combine; they don't compete.

Mistake 2: expecting a gain on prefill

GAAC only accelerates decoding. Ingesting a long prompt still uses standard attention. If your problem is time-to-first-token on RAG, this method won't help you — look into prefill or pipeline architecture instead.

Mistake 3: extrapolating the 1.27x to all configurations

The maximum was measured on the DGX Spark, at long context. On an M4 at 8K, the gain is 2% — imperceptible. The general rule: the gain follows the share of the KV cache in the total memory traffic of inference.

Mistake 4: assuming it will work the same on FP8

FP8 KV caches (E4M3) offer little exponent redundancy to exploit. GAAC targets BF16, the de facto standard of today's local stacks — but not necessarily tomorrow's.

❓ Frequently Asked Questions

Is GAAC available in Ollama or llama.cpp?

No. The paper was published on October 28, 2026, and no integration has been announced to date. Kernels of this type typically take several months to make their way into mainstream stacks. In the meantime, NVIDIA/kvpress lets you experiment with KV cache compression.

Does GAAC work with quantized models?

Yes, in principle. The paper measures a W4A16 scenario — 4-bit weights, 16-bit activations — on DGX Spark, with a measured speedup at the decode step at batch 1. This is the typical local setup: quantized weights, 16-bit KV cache, as in most GGUF stacks.

What's the difference with 2-bit KV quantization?

2-bit quantization is lossy: it divides memory by four but introduces rounding error, which can accumulate over long generations. GAAC is lossless: −26% memory, zero degradation, output identical bit for bit. The two are complementary in an inference pipeline.

On which hardware are the gains most visible?

On DGX Spark, with 1.19 to 1.27x in long-context attention: its unified memory bandwidth is limited, so every byte saved counts. On Apple M4, the gain depends on context: 1.02x at 8K, 1.10x at 64K with Qwen3-8B, up to 1.25x depending on configurations.

Do you need a long context to benefit?

Yes. GAAC's target is long-context attention, where the KV cache dominates memory traffic. At short context, weights dominate everything and the gain becomes negligible. If your sessions fit within 8k tokens, this technique won't change anything for you.

✅ Conclusion

GAAC isn't revolutionizing local inference, but it validates an idea that's all too rare in this field: you can still gain bandwidth without sacrificing a single bit of precision. If you're setting up a local stack, start by choosing the right model with our comparison of the best current LLMs — the kernels, for their part, will arrive eventually.