Estimate how semantic KV reuse changes the monthly cost and capacity profile of a self-hosted LLM serving fleet. The model always runs and generates a fresh answer; reuse removes recomputation, not generation.
What each request can do
| Outcome | Meaning | Modeled work |
|---|
| Semantic reuse hit | The context is semantically similar to prior traffic; validated KV computation is reused and prefill is skipped. | Decode only |
| Exact-prefix hit | A byte-identical KV/prompt-cache prefix is available. Serving engines do this today. | Decode only |
| Miss | No usable reuse. | Full prefill and decode |
Both hit types cost the same per request. The difference is coverage: exact-prefix fires only on byte-identical prefixes, while semantic reuse extends the same prefill savings to similar-but-not-identical context. The gap between the chart's two lines is the increment semantic reuse adds on top of what the serving engine already provides.
Formulas
cost_per_sec = fleet_cost_per_hour / 3600
prefill_sec = input_context_tokens / prefill_tokens_per_sec
decode_sec = output_tokens / decode_tokens_per_sec
miss_cost = (prefill_sec + decode_sec) x cost_per_sec
prefix_hit_cost = decode_sec x cost_per_sec
semantic_hit_cost = decode_sec x cost_per_sec
blended_cost = semantic_rate x semantic_hit_cost
+ prefix_rate x prefix_hit_cost
+ miss_rate x miss_cost
monthly_cost = blended_cost x requests_per_day x 30
exact_prefix_only = same blend, semantic traffic treated as misses
semantic_added = exact_prefix_only - monthly_cost
reclaimed_hours = (semantic_rate + prefix_rate) x prefill_sec
x requests_per_day x 30 / 3600
Defaults
| Input | Default | Basis |
|---|
| Requests per day | 100,000 | Illustrative enterprise workload; agentic traffic multiplies per-user calls |
| Input context | 20,000 tokens | Long-context assistant or agent session |
| Output | 500 tokens | Typical non-trivial answer |
| Semantic reuse hit rate | 25% | Illustrative; depends on workload repetition |
| Exact-prefix hit rate | 40% | Illustrative; depends on byte-identical prefix reuse and TTL fit |
| Currency | USD | Switching converts values with fixed July 2026 reference rates |
| Fleet cost | $160/hour | Illustrative mid-size serving fleet, all-in |
| Prefill throughput | 100,000 tok/s | Illustrative fleet-level throughput |
| Decode throughput | 5,000 tok/s | Illustrative fleet-level throughput |
Replace the throughput and fleet-cost inputs with measured numbers before using the output as a business case.
Known simplifications
- A semantic hit is modeled as skipping the full prefill. Real systems reuse validated KV blocks and pay embedding, lookup, validation, and policy-check overhead.
- Hit rates are inputs, not a TTL simulation. Real hit rates depend on request timing, cache size, routing, and eviction behavior.
- Reclaimed capacity only becomes cash savings when the fleet can be scaled down, deferred, or used to absorb growth.
- Request shape is uniform. Real traffic has a distribution of context and output lengths; run separate cases for p50, p90, and p99 workloads.
- Reuse safety gates are outside the calculator. KV computation is only reused when it passes validation and policy checks for the new request.
- Currency selection converts values with fixed July 2026 reference rates; it does not change the underlying model.
MIT license. Built as a WorldFlow AI public artifact.