Reuse, Don't Recompute

Governed semantic KV reuse on a real research workload

Your GPUs already computed the context. Why pay to compute all of it again?

We are testing governed semantic KV cache reuse for open weight models running on customer controlled GPUs. It complements the exact prefix caching an inference engine already performs. When a request contains eligible reusable context, the system can reuse validated model computation. When it cannot establish that reuse is safe, it runs the request normally. In this article, security validated refers to the authorization controls exercised in calibration. It is not an independent security certification.

That is the idea, and this is the benchmark behind it. We began with a real research workload, rebuilt its operating pattern in an open benchmark designed for reproducible testing, and measured three different paths. We tested exact reuse, similarity based semantic reuse, and semantic reuse controlled by policy and security checks. For each path, we examined time to first token, avoided prefill computation, and answer quality.

The workload

The original run was a structured research effort covering more than 100 entities. Its subject and participants remain private. The aggregate workload is what matters here.

The ledger shows 12.33 million input and cache related token events plus 1.92 million output tokens. The 14.25 million total is recorded activity, not unique content. The same material can appear first as a cache write and later as a cache read.

The workflow made 2,074 model calls. Each entity generated roughly 20 calls containing evidence, a question, and a structured answer contract. Evidence appeared again under different questions and request wrappers. Comparable entities also had to remain strictly isolated from one another. That combination makes the workload a useful test of semantic reuse. It contains repeated evidence that could create value, plus deceptively similar evidence that must never cross a boundary.

We reproduced the pattern in SemBench, an open benchmark designed for reproducible KV reuse testing. The frozen workload used in this study contains 504 paired cases across seven transformation families, with prompts ranging from 4,000 to about 18,700 tokens. Each answer is scored for its required structured values. The primary run used Qwen2.5 7B Instruct on one NVIDIA A10G. SGLang served the model, and SemBlend supplied semantic candidates. Every case ran first without a donor and then with one. We counted reuse only when engine audit events confirmed that KV state was physically materialized.

Time to first token

Time to first token is the wait before an answer begins. Long prompts can spend much of that interval in prefill, when the model processes the input and builds its KV state.

The exact path reduced mean time to first token by about 97.2 percent. It establishes the performance ceiling for prompts whose token prefixes align. The governed SGLang calibration case demonstrates the broader opportunity. It reduced time to first token by 88.5 percent on a request that was not eligible for a full exact prefix hit. The result is promising, but it is one calibration case rather than a fleet wide performance claim. For scale, applying the measured 6.32 second cold mean to 2,074 calls yields about 3.6 cumulative hours before first tokens. Actual latency varies with prompt shape, model, hardware, concurrency, and scheduling. Deployment savings therefore need to be measured on the intended runtime rather than inferred from a single GPU.

Exact reuse and the semantic opportunity

Exact prefix reuse is narrow and dependable. When a previously processed token prefix appears again, the engine can reuse its KV state. In our 72 usable exact pairs, it produced a 35.6 times mean improvement with no measured structured quality regression. Semantic reuse targets work that exact matching misses. A familiar example is the same evidence presented behind a different instruction wrapper or a reworded question. The prompt can differ near the beginning even though most of its evidence has already been processed.

Our first semantic configuration used similarity as the primary admission signal. It discovered related donors, materialized 102,176 semantic KV tokens, and gave unrelated controls no reuse. It also caused structured quality regressions in 60.14 percent of usable semantic positive pairs. Faster was not better. The failures explain why similarity cannot be the authorization decision. A donor may contain the right evidence for the wrong task. A reused span can also cross an instruction or response boundary even when the underlying documents are related.

The governed path adds task, response, security, and token checks before physical reuse. In the clearest SGLang calibration example, semantic discovery found a related donor. Token verification then established that 4,760 of approximately 4,880 prompt positions contained identical tokens and approved those positions for copying. The reuse was semantic in how the donor was found and exact in what was physically copied. Time to first token fell from 2.34 seconds to 269 milliseconds, while the required answer values remained correct. The answer was fast and correct. Immediately afterward, however, SGLang detected a KV pool accounting invariant failure and stopped the scheduler. We therefore did not open the final held out test. Correcting that lifecycle path is part of completing the initial release through the upstream contribution.

What the original run might have avoided

The word “tokens” needs care here. Submitted input tokens avoided were zero. The complete prompt still entered the service and was tokenized. Generated output tokens avoided were also zero. The model still produced a new answer. What semantic KV reuse can avoid is GPU prefill computation for positions approved by the reuse policy. The incremental opportunity applies most naturally to the 6.84 million uncached input tokens in the original workload.

These are planning scenarios, not reconstructed historical savings. The benchmark did not run inside the original hosted model calls, and the original workload did not have the same measured donor eligibility distribution. The original API ledger totaled $45.02. Multiplying that number directly by the percentages above would be misleading. The bill combines input, cache, and output categories that may have different prices. Output generation was not avoided. The benchmark also ran a different model on customer controlled infrastructure.

The defensible financial conclusion is narrower. Mature semantic reuse can reduce the compute attached to approved uncached prefill. Converting that reduction into dollars requires the target model, GPU throughput, utilization, accepted reuse length, fallback rate, and local infrastructure cost. At the scale of one research project, the immediate value may appear primarily as lower latency. At fleet scale, avoided prefill also becomes capacity on GPUs the organization already operates.

Does it work

The answer is different at each layer.

• Exact cache reuse works. Across 72 usable pairs, it produced a 35.6 times mean improvement with no measured structured regression.

• Semantic discovery works. The system found related candidates while unrelated controls received no semantic materialization.

• Similarity alone does not work as a safe admission policy. The 60.14 percent structured regression rate makes that unambiguous.

• Policy authorization worked in calibration. The gate denied all 30 candidates across the changed task, entity swap, and unrelated cohorts. Denied requests received no semantic materialization.

• Governed semantic realization worked in a limited vLLM confirmation slice. Ten shifted repeat pairs materialized 20,320 semantic tokens, improved mean paired time to first token by about 1.12 times, and preserved the required values with some wording drift. That is about a 10.7 percent reduction in time to first token.

• Token verified SGLang realization worked for the highlighted calibration response. It was 8.7 times faster and preserved the required values.

• The initial SGLang release path remains in progress. Its invariant checker found a KV pool accounting defect after the response. That correction remains part of the upstream contribution work.

That is a useful result. The benchmark separated a genuine optimization from unsafe admission, and then caught an engine problem that a fast and correct looking answer would have concealed.

Why the gates matter

Every gate can lower the raw hit rate. That is expected. The right optimization target is avoided computation per successful task, not the largest possible number of cache hits.

• Tenant and namespace checks prevent a related donor from crossing an authorization boundary.

• Signed provenance makes altered eligibility data detectable.

• Expiration and version checks prevent stale policy state from authorizing reuse.

• Personal data checks can refuse unsafe donors before they enter the reusable pool.

• Task matching prevents shared evidence from authorizing a different job.

• Response contract matching protects the required answer structure.

• Token verification limits physical reuse to approved positions.

• Cold fallback preserves availability whenever a gate denies reuse.

• Engine invariants test whether resource ownership remains correct after the answer is returned.

The vLLM calibration illustrates the selectivity. The gate denied 30 of 30 candidates in the three clearly unsafe cohorts. It accepted the intended exact, shifted repeat, and question cohorts in that calibration. After one instruction variant regressed, a response contract check denied all ten instruction variants in the final check. These are calibration results, not universal security percentages. Their value is architectural. A reuse decision can be constrained, explained, audited, and reversed to an ordinary cold execution. Semantic reuse is therefore more than a similarity problem. It is a controlled execution problem. The policy and security gates are what can make the optimization suitable for enterprise inference.

Test the mechanism

The work is open. Our SGLang contribution, PR 31057, adds semantic KV reuse as an optional radix cache backend with a provider interface. The default engine path remains unchanged. SemBlend supplies the semantic provider. SemBench supplies the frozen workload and paired protocol.

The next engineering milestone is specific. We need to correct and instrument KV ownership around semantic realization, repeat the single calibration pair until pool accounting remains stable, rerun the seven pair calibration, and only then open the untouched confirmation set. That work defines the initial release path. As the contribution moves through upstream review, the same foundation can support additional innovations. These include reuse across several regions of one prompt, selective recomputation where a copied layer needs reinforcement, and governed evidence reuse across different tasks.

That sequence matters. A compelling speed result is useful. A speed result that survives quality checks, policy denial, cold fallback, and resource accounting is what earns trust.

Bring a long context workload and we will help identify where reusable computation may be hiding. Contact Us.