Caching

Caching

There are only two hard things in computer science: cache invalidation and naming things. This skill covers the first one.

When to Cache

Cache when ALL of these are true:

1. Read-heavy: Data is read much more often than written
2. Expensive to compute/fetch: Database query, API call, heavy computation
3. Tolerant of staleness: It's OK if users see data that's a few seconds/minutes old
4. Stable enough: Data doesn't change every request

Don't cache when:

• Data must be real-time (account balance, inventory count at checkout)
• Data changes on every request
• The source is already fast enough
• You can't reason about when the cache is wrong

Cache Placement

User → CDN Cache → Reverse Proxy Cache → App Cache → Database Cache → Database

Layer	What to Cache	TTL	Example
Browser	Static assets, API responses	Minutes to forever	Cache-Control: max-age=31536000 for hashed assets
CDN	Static files, public API responses	Minutes to hours	CloudFront, Fastly
Reverse Proxy	Full page/response cache	Seconds to minutes	Nginx, Varnish
Application	Computed results, API responses	Seconds to minutes	Redis, Memcached, in-memory
Database	Query results, materialized views	Varies	Query cache, materialized views

Rule: Cache as close to the user as possible. Each layer closer to the user avoids more work.

Cache Strategies

Cache-Aside (Lazy Loading)

READ:
  1. Check cache
  2. Cache hit → return cached data
  3. Cache miss → fetch from source → write to cache → return

WRITE:
  1. Write to source
  2. Invalidate cache (delete, don't update)

• Pro: Only caches data that's actually requested
• Con: First request always slow (cache miss)
• Best for: Most cases. Default choice.

Write-Through

WRITE:
  1. Write to cache AND source simultaneously

READ:
  1. Always read from cache (guaranteed fresh)

• Pro: Cache always consistent
• Con: Write latency higher, caches data that may never be read
• Best for: Data that's written and immediately re-read

Write-Behind (Write-Back)

WRITE:
  1. Write to cache
  2. Asynchronously flush to source (batched)

READ:
  1. Read from cache

• Pro: Very fast writes
• Con: Risk of data loss if cache fails before flush
• Best for: High write throughput, analytics, counters

Invalidation Patterns

TTL (Time-To-Live)

cache.set(key, value, ttl=300)  # Expire after 5 minutes

• Simplest approach. Data is eventually consistent.
• Choose TTL based on: how stale can users tolerate? How expensive is a miss?

Event-Based Invalidation

# On data change, explicitly delete the cache entry
def update_user(user_id, data):
    db.update(user_id, data)
    cache.delete(f"user:{user_id}")
    cache.delete(f"user_list:page:*")  # Related caches too

• More complex but fresher data.
• Must track ALL caches affected by each data change.

Versioned Keys

cache_key = f"user:{user_id}:v{user.updated_at}"

• Old versions naturally expire. No explicit invalidation needed.
• Cache size grows until old versions TTL out.

Cache Stampede Prevention

When a popular cache key expires, hundreds of requests simultaneously hit the source. Solutions:

Lock / Mutex

# Only one request rebuilds the cache; others wait
if cache.miss(key):
    if cache.acquire_lock(key):
        value = expensive_query()
        cache.set(key, value)
        cache.release_lock(key)
    else:
        wait_for_key(key)  # or return stale

Stale-While-Revalidate

# Serve stale data while one request refreshes in the background
cache.set(key, value, ttl=300, stale_ttl=600)
# Serves stale data for 5 extra minutes while refreshing

Jittered TTL

# Spread expirations to avoid thundering herd
ttl = base_ttl + random(0, jitter_seconds)

Common Anti-Patterns

Anti-Pattern	Problem	Fix
Cache everything	Memory bloat, stale data everywhere	Cache only what's expensive and read-heavy
No TTL	Stale data lives forever	Always set a TTL, even if long
Cache & update (not invalidate)	Race conditions between concurrent updates	Invalidate (delete) on write, not update
Caching errors	Error response served to all users	Only cache successful responses
Unbounded cache	Memory grows until OOM	Set max size with eviction policy (LRU)
Cache key without version	Deploy new code, old cache format causes errors	Include version/schema in cache keys

Cache Key Design

{service}:{entity}:{id}:{version}

# Examples
api:user:123:v2
api:user_list:page:1:per:20
api:search:q=shoes:page:1

• Deterministic: same input → same key
• Namespaced: no collisions between services
• Inspectable: you can read the key and know what it caches

Review Checklist

• Cache has a TTL (no unbounded caching)
• Cache is invalidated on write (not just TTL-dependent)
• Error responses are not cached
• Cache keys are deterministic and namespaced
• Cache stampede handled for hot keys
• Cache size bounded with eviction policy
• Metrics: hit rate, miss rate, eviction rate
• Fallback works when cache is down (graceful degradation)