This week I contracted COVID-19 and stayed at home for nine days. Despite being unwell, the most critical task was assessing the impact of launching a high-traffic mini program on the core services I was responsible for.
This mini program aligned closely with user demand at the time, and a significant traffic surge was expected. Previously, I had already evaluated and scaled the system for a major feature launch. However, after sending notifications to hundreds of millions of users, the system experienced a large number of timeouts.
At around 8:00 a.m., I was informed that the login interface was experiencing severe timeout issues. After connecting remotely, I observed that traffic had increased more than twenty times.
My initial assumption was that the number of service containers was insufficient to handle the surge. However, log analysis quickly ruled this out.
The issue was traced to two main causes.
First, an internal interface was rate-limited, and the incoming traffic exceeded its configured threshold. This was resolved quickly by coordinating with the relevant team to increase the rate limit.
Second, the cache layer became a bottleneck. A large number of returning users triggered heavy reads from the database to rebuild cache entries. As traffic continued to grow, the backend database reached its I/O limit, leading to cascading timeouts.
To address this, we expanded several core cache clusters, increasing both capacity and the number of nodes to better handle subsequent traffic spikes.
In addition to incident handling, I also worked on improving the stability of a gateway service by adjusting its Kubernetes scheduling strategy.
Previously, the service was deployed on compute nodes that were themselves virtualized from on-premise physical machines. On top of this layer, containers were created and Pods were scheduled within them. This multi-layer setup led to resource contention and weak isolation, as multiple Pods shared the same underlying operating system and competed for CPU, memory, and network resources.
Furthermore, when such a compute node required maintenance or reboot, all Pods on that node would be affected simultaneously. This made the setup unsuitable for gateway services, which require high reliability and low latency.
To improve this, I migrated the service to a single-layer scheduling model provided by the cloud platform. In this model, each Pod runs on a dedicated virtual machine instance provisioned directly from the cloud resource pool.
Kubernetes treats these instances as specialized nodes, allowing Pods to be scheduled directly onto them. This approach significantly improves isolation, as each Pod runs in its own environment and does not interfere with others.
After updating the scheduling strategy and migrating all Pods to this model, the overall stability of the service improved noticeably. During peak traffic periods, the previous issues of high timeout rates combined with abnormal CPU behavior were largely eliminated.