Caching strategies

Caching Strategies

Concept of the skill

A cache stores a copy of data closer to the consumer so repeated reads avoid the expensive recompute or refetch. The win is real — an expensive query becomes a fast lookup — but the copy creates a coherence problem: it can drift from the source of truth. Caching is the discipline of capturing the speed-and-load win while controlling the staleness it introduces.

Every caching decision is two coupled choices made for a specific layer:

1. Staleness budget — how out-of-date may this data be? (A price quote: seconds. A user's avatar: hours. A blog post: minutes.)
2. Invalidation strategy — how does the copy get corrected when the source changes? (Expiry, explicit invalidation, versioned keys, write-through.)

The two famously hard problems, per the field's oldest joke, are cache invalidation and naming things — and caching forces you to confront the first one directly.

Coverage

• The cache layers — client/browser, CDN/edge, application in-memory, distributed cache (Redis/Memcached), and the database's own caches — and what each is good for.
• Read/write patterns — cache-aside (lazy loading), read-through, write-through, write-behind (write-back), refresh-ahead — with consistency and failure tradeoffs.
• TTL and expiry design and the staleness budget.
• Eviction policies — LRU, LFU, FIFO, TTL — and sizing.
• Cache coherence and invalidation — time-based expiry, explicit/event-driven invalidation, key/tag versioning, write-through coherence.
• Cache stampede / thundering herd — and mitigations: request coalescing (single-flight), locks/leases, TTL jitter, stale-while-revalidate, probabilistic early refresh.
• HTTP caching semantics — Cache-Control, ETag and conditional requests, stale-while-revalidate / stale-if-error — and CDN/edge caching.
• Cache key design — cardinality, namespacing, and the per-user-in-a-shared-cache trap.
• Negative caching of misses.
• Measurement — hit ratio and the cost of a miss.

Philosophy of the skill

Caching is a correctness risk traded for a performance win — treat it as such. A cache that serves stale or cross-tenant data is worse than no cache: it is fast and wrong. Before adding a cache, state the staleness budget and the invalidation strategy explicitly. If you cannot say how the cache gets invalidated, you are not ready to add it.

The miss path is the design. A cache makes the hot path fast, but the system's behavior under misses — cold start, mass expiry, a hot key invalidated — is where caches cause outages. Design the miss path (coalescing, jitter, stale-while-revalidate) with as much care as the hit path.

The cache layers

client/browser ─► CDN/edge ─► app in-memory ─► distributed cache ─► database
   (per user)      (shared,     (fastest,        (shared across       (source of
                    geo)         per-instance)     instances, Redis)     truth)

Choosing the layer is half the skill. Per-user data must not sit in a shared CDN (you'll leak one user's response to another). Cross-instance consistency needs a distributed cache, not per-instance memory. Static assets belong at the edge. The closer to the consumer, the faster — and the harder to invalidate.

Read/write patterns

• Cache-aside (lazy loading): the app checks the cache; on a miss it loads from the source, populates the cache, and returns. Simple and resilient (cache down ≠ app down), but the first request after expiry pays the miss, and there's a small write-skew window.
• Read-through: the cache itself loads from the source on a miss (the app only talks to the cache). Cleaner app code; couples app to the cache.
• Write-through: writes go to the cache and the source synchronously — strong coherence, higher write latency.
• Write-behind (write-back): writes hit the cache and flush to the source asynchronously — fast writes, risk of loss if the cache dies before flush.
• Refresh-ahead: proactively refresh hot keys before they expire — hides miss latency, wastes work on cold keys.

Pick by the consistency you need and the failure you can tolerate.

Invalidation strategies

• TTL / time-based expiry — simplest; data is correct-enough within the staleness budget, wrong after a source change until expiry.
• Explicit / event-driven invalidation — on a write to the source, delete or update the cached key. Coherent, but you must find every key affected by a change.
• Key/tag versioning — embed a version (or a tag) in the key; bump it to invalidate a whole class at once without enumerating keys.
• Write-through coherence — the write updates the cache, so it never goes stale (at write-latency cost).

There is no free invalidation: TTL is easy but stale-prone; explicit is coherent but hard to get complete; versioning trades memory for simplicity.

Cache stampede (thundering herd)

When a hot key expires (or many keys share an expiry), the next requests all miss simultaneously and slam the backend with identical work — a self-inflicted load spike that can cascade into an outage. Mitigations:

• Request coalescing / single-flight — only one request recomputes the key; the rest wait and share the result.
• Locks / leases — the first miss takes a lease to recompute; others serve stale or wait.
• TTL jitter — randomize expiry so keys don't all expire together.
• stale-while-revalidate — serve the stale value immediately and refresh in the background.
• Probabilistic early refresh — refresh a key slightly before expiry with rising probability, smoothing the miss.

A caching design that ignores the miss-storm is incomplete.

HTTP caching

For cacheable HTTP responses, the protocol gives you the tools: Cache-Control (max-age, s-maxage, public/private, no-store), ETag + conditional If-None-Match for cheap revalidation (304 Not Modified), and stale-while-revalidate / stale-if-error for graceful freshness. CDNs honor these directives at the edge. private vs public is a correctness control — never let a private (per-user) response be cached by a shared CDN.

Verification

A caching design is sound when you can answer yes to:

• Is the staleness budget stated for each cached datum?
• Is there a real invalidation strategy (not just a TTL) for data that changes?
• Is the layer correct for the data's sharing scope (no per-user data in a shared cache)?
• Is the read/write pattern chosen for the consistency you actually need?
• Is the miss storm handled (coalescing, jitter, or stale-while-revalidate)?
• Do you measure hit ratio and know the cost of a miss?
• For HTTP, are private/public and validation headers correct?

If any answer is no, name the gap before shipping the cache.

Do NOT Use When

• The task is making the source query itself faster via an index — that is indexing-strategy. A cache fronts the query; it does not optimize it.
• The task is rewriting the query (N+1 → join, removing a full scan) — that is query-optimization.
• The task is HTTP protocol semantics in general (methods, status codes, content negotiation) — that is http-semantics; caching uses only its Cache-Control/ETag subset.
• The task is connection pooling or CDN/infrastructure provisioning — out of scope.

References

• RFC 9111 — HTTP Caching: https://www.rfc-editor.org/rfc/rfc9111
• Cache-Aside pattern (Azure): https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside
• Redis patterns: https://redis.io/docs/latest/develop/use/patterns/
• stale-while-revalidate: https://web.dev/articles/stale-while-revalidate
• Scaling Memcache at Facebook (NSDI 2013): https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final197.pdf
• Cloudflare cache concepts: https://developers.cloudflare.com/cache/concepts/cache-control/