Semantic Caching for LLM APIs: A Practical Guide

Every LLM-powered application has repeated questions. Your support bot answers "how do I reset my password?" fifty times a day. Your code assistant explains the same API patterns to different developers. Your content tool rewrites the same product descriptions with minor variations.

Each duplicate request costs the same as the first one. There's no volume discount, no automatic deduplication. Unless you cache.

This guide covers how to implement caching for LLM APIs, from simple exact-match caching to vector-based semantic similarity, with real numbers on hit rates and savings.

Why LLM Caching Is Different

Traditional API caching is straightforward: same input, same output, cache the response. LLM caching is harder for three reasons:

• Inputs are natural language. "How do I reset my password?", "Reset password steps", and "I forgot my password, how to fix?" are semantically identical but textually different. An exact-match cache misses all of these after the first.
• Outputs are non-deterministic. LLMs produce different responses to the same input (unless temperature is 0). A cached response is a valid answer, but it's one valid answer. This is usually fine for factual queries but can be an issue for creative tasks.
• Context matters. The same user message in different conversation contexts should produce different responses. Your cache key needs to account for conversation history, system prompts, and any other contextual inputs.

These challenges are solvable. Let's look at the two caching layers that handle them.

Layer 1: Exact Match Caching

The first layer is a straightforward hash-based cache. Take the full request body (system prompt + messages + model + parameters), compute a SHA-256 hash, and use it as the cache key. If you've seen this exact request before, return the cached response.

Implementation

The basic flow is:

1. Serialize the request body deterministically (sorted keys, no whitespace variation).
2. Compute SHA-256 hash of the serialized body.
3. Check Redis/Memcached for the hash key.
4. On hit: return cached response with a X-Cache: HIT header.
5. On miss: forward to the LLM provider, cache the response, return it with X-Cache: MISS.

What to cache

• Do cache: non-streaming 200 responses. These are complete, well-formed responses.
• Don't cache: streaming responses (partial data), error responses (transient issues), or requests with high temperature (>0.7) where response variety is desired.

Cache key design

Your cache key should include:

• Organization/tenant ID (multi-tenant isolation)
• Model name (different models produce different responses)
• Full message array (system + user + assistant messages)
• Temperature and other generation parameters

Exclude request IDs, timestamps, and anything that varies per request but doesn't affect the response content.

TTL strategy

LLM responses don't go stale the way database queries do, but they're not infinitely valid either. A reasonable default is 1 hour for general-purpose caching. Adjust based on your use case:

Hit rates

Exact match caching alone typically produces 5-12% hit rates in production. The exact number depends heavily on your application:

• Support chatbots: 10-15% (many users ask the same questions verbatim)
• Code assistants: 5-8% (prompts are more varied)
• Content tools: 3-5% (each request tends to be unique)
• Classification pipelines: 15-25% (repeated documents, batch reprocessing)

Those percentages translate directly into cost savings. A 10% hit rate means 10% fewer API calls and 10% lower token costs.

Layer 2: Semantic Similarity Caching

Exact match misses paraphrased queries. Semantic caching catches them by comparing the meaning of requests rather than their exact text.

How it works

1. Extract the prompt text from the request (typically the user messages).
2. Generate an embedding vector for the prompt using a fast embedding model (e.g., text-embedding-3-small).
3. Search a vector store for existing cached prompts with high cosine similarity.
4. If similarity exceeds a threshold (typically 0.95), return the cached response.
5. On miss: forward to the LLM, cache both the response and the embedding.

Choosing a similarity threshold

The threshold is the most important parameter. Too low and you return irrelevant cached responses. Too high and you rarely get hits.

Start at 0.95 and adjust based on observed false positive rates. Monitor a sample of cache hits weekly to verify the responses are actually relevant to the queries they're serving.

Embedding model costs

Semantic caching has its own cost: generating the embedding for each request. With text-embedding-3-small at $0.02 per million tokens, a typical 200-token prompt costs $0.000004 to embed. Even at a million requests per month, embedding costs total $4 — negligible compared to the savings from cache hits.

Vector storage options

You need a vector store for similarity search. The main options:

For most teams, pgvector is the right starting point. If you're already running PostgreSQL, it requires zero additional infrastructure. It handles millions of vectors with sub-100ms search times when properly indexed. You can always migrate to a dedicated vector store later if you outgrow it.

Combined hit rates

When you combine exact match and semantic similarity caching, hit rates improve significantly:

Semantic caching roughly doubles the hit rate of exact match alone. For a support chatbot spending $10,000/month on LLM calls, going from 12% to 25% cache hits saves an additional $1,300/month.

What Not to Cache

Not every LLM response should be cached. Some cases where caching does more harm than good:

• Personalized responses — if the response depends on user-specific data not in the prompt (fetched via function calling), a cached response for a similar prompt may contain the wrong user's data.
• Time-sensitive queries — "What's the weather today?" or "What's the latest news?" should not be cached beyond a few minutes.
• Creative generation with high temperature — users asking for creative content expect variety. Returning the same cached poem or story defeats the purpose.
• Multi-turn conversations — caching is most effective on single-turn queries. For multi-turn conversations, the context window changes with every message, making cache hits unlikely and potentially confusing if they do match.

Cache Invalidation

The classic hard problem. For LLM caches, the good news is that most responses don't have strong invalidation requirements. They're not tied to mutable database records. A few strategies that work well:

TTL-based expiry

The simplest and most effective approach. Set a TTL and let entries expire naturally. This handles 90% of invalidation needs. When underlying information changes (product updates, policy changes), the old cached responses expire within the TTL window.

Prompt-version keying

If you update your system prompt, you want the cache to invalidate. Include a prompt version or hash in the cache key. When the system prompt changes, the keys change, and old entries are effectively invalidated.

Manual purge

Provide an API endpoint or dashboard button to flush the cache for a specific organization or globally. This is useful when you know cached responses are wrong (a product name changed, a feature was removed) and can't wait for TTL expiry.

Measuring Cache Effectiveness

To know if your cache is working, track these metrics:

• Hit rate — the percentage of requests served from cache. This is your primary metric.
• Cost avoided — hit rate multiplied by the average request cost. This is the dollar value of your cache.
• Latency improvement — cached responses are typically served in 5-20ms vs 500-5000ms for an LLM call. This is a major UX win.
• False positive rate — for semantic caching, what percentage of cache hits return responses that don't actually match the query? Sample and review weekly.
• Cache size — monitor storage usage to ensure costs stay proportional to savings.

A healthy cache has a hit rate above 10%, a false positive rate below 2%, and cost-avoided that's at least 5x the cache infrastructure cost.

Architecture: Putting It Together

A production-grade LLM cache has two layers working together:

1. Request arrives — compute SHA-256 hash of the full request body.
2. Exact match check — look up hash in Redis. If hit, return immediately (~2ms).
3. Semantic check — generate embedding, search vector store for similarity ≥ 0.95. If hit, return (~50ms).
4. Cache miss — forward to LLM provider, wait for response.
5. Store — on successful response, store in both exact cache (Redis) and semantic cache (vector store + response).
6. Return — send response to client with cache status header.

The exact match layer is checked first because it's faster (2ms vs 50ms). The semantic layer catches what exact match misses. Together, they maximize hit rate while keeping latency overhead minimal on cache misses.

Cost-Benefit Analysis

Let's put real numbers on it. For a team processing 500,000 LLM requests per month at an average cost of $0.012 per request:

The infrastructure cost is negligible relative to the savings. Even a modest 10% hit rate pays for the cache 20 times over. With semantic caching added, ROI exceeds 40x.

Getting Started

If you want to add caching to your LLM pipeline:

1. Start with exact match. It takes an afternoon to implement with Redis, catches the easy wins, and teaches you your application's cache hit patterns.
2. Log cache misses. After a week, sample your cache misses and look for semantic duplicates. If you find many, semantic caching will have high impact.
3. Add semantic caching. Use pgvector if you already run PostgreSQL. Set the similarity threshold to 0.95 and monitor false positives.
4. Expose cache headers. Return X-Cache: HIT or MISS so your application can log and monitor cache effectiveness.
5. Review weekly. Check hit rates, false positives, and cost avoided. Adjust TTLs and similarity thresholds based on data.

Or, use a proxy that has both layers built in from day one.