This week was focused on addressing a risk identified before the holiday period.

The issue involved a service that used Redis without setting TTLs for keys, instead relying entirely on Redis eviction policies. The instance was configured with an LRU-based eviction strategy, which may appear sufficient at first glance, but introduces significant risks under bursty write workloads.

During periods of high write traffic, Redis is forced to aggressively trigger eviction in order to free up memory. This process consumes considerable resources, which can degrade normal operations such as read and write requests. As a result, latency at the application layer can fluctuate significantly. I have encountered this issue multiple times in production.

Another problem with not setting TTLs is that Redis memory usage tends to remain near 100% at all times. This makes capacity planning difficult: it becomes unclear how much memory is actually required for the workload, and whether cost optimizations (such as downsizing during off-peak periods) are feasible. Maintaining sufficient free memory is critical for system stability.

To mitigate these risks, I decided to modify the service to assign a TTL to each newly written key. According to Redis behavior, if a key already exists, writing to it will also update its TTL, which aligns well with the intended design.

For stability reasons, I avoided scanning the entire Redis dataset to retroactively assign TTLs. Although this can be done using cursor-based iteration, it introduces unnecessary overhead and operational risk.

Another important consideration is how TTLs are distributed. If a large number of keys expire at the same time, it can lead to sudden spikes in load and latency. To prevent this, I introduced randomized expiration.

Specifically, if the base TTL is T, the actual TTL is set within the range [T, 2T]. This spreads expiration events over time, ensuring smoother load patterns and reducing pressure on the system.

This approach improves both system stability and predictability, while also enabling more effective capacity management.