<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Kubernetes on Tech Philosopher</title><link>https://blog.bktus.com/en/tags/kubernetes/</link><description>Recent content in Kubernetes on Tech Philosopher</description><generator>Hugo</generator><language>en-US</language><lastBuildDate>Thu, 19 Jan 2023 19:31:54 +0800</lastBuildDate><atom:link href="https://blog.bktus.com/en/tags/kubernetes/index.xml" rel="self" type="application/rss+xml"/><item><title>Technical Review for the Fourth Week of December 2022</title><link>https://blog.bktus.com/en/archives/uxk641/</link><pubDate>Thu, 19 Jan 2023 19:31:54 +0800</pubDate><guid>https://blog.bktus.com/en/archives/uxk641/</guid><description>&lt;p>This week I contracted COVID-19 and stayed at home for nine days. Despite being
unwell, the most critical task was assessing the impact of launching a
high-traffic mini program on the core services I was responsible for.&lt;/p>
&lt;p>This mini program aligned closely with user demand at the time, and a
significant traffic surge was expected. Previously, I had already evaluated and
scaled the system for a major feature launch. However, after sending
notifications to hundreds of millions of users, the system experienced a large
number of timeouts.&lt;/p></description><content:encoded><![CDATA[<p>This week I contracted COVID-19 and stayed at home for nine days. Despite being
unwell, the most critical task was assessing the impact of launching a
high-traffic mini program on the core services I was responsible for.</p>
<p>This mini program aligned closely with user demand at the time, and a
significant traffic surge was expected. Previously, I had already evaluated and
scaled the system for a major feature launch. However, after sending
notifications to hundreds of millions of users, the system experienced a large
number of timeouts.</p>
<p>At around 8:00 a.m., I was informed that the login interface was experiencing
severe timeout issues. After connecting remotely, I observed that traffic had
increased more than twenty times.</p>
<p>My initial assumption was that the number of service containers was insufficient
to handle the surge. However, log analysis quickly ruled this out.</p>
<p>The issue was traced to two main causes.</p>
<p>First, an internal interface was rate-limited, and the incoming traffic exceeded
its configured threshold. This was resolved quickly by coordinating with the
relevant team to increase the rate limit.</p>
<p>Second, the cache layer became a bottleneck. A large number of returning users
triggered heavy reads from the database to rebuild cache entries. As traffic
continued to grow, the backend database reached its I/O limit, leading to
cascading timeouts.</p>
<p>To address this, we expanded several core cache clusters, increasing both
capacity and the number of nodes to better handle subsequent traffic spikes.</p>
<p>In addition to incident handling, I also worked on improving the stability of a
gateway service by adjusting its Kubernetes scheduling strategy.</p>
<p>Previously, the service was deployed on compute nodes that were themselves
virtualized from on-premise physical machines. On top of this layer, containers
were created and Pods were scheduled within them. This multi-layer setup led to
resource contention and weak isolation, as multiple Pods shared the same
underlying operating system and competed for CPU, memory, and network resources.</p>
<p>Furthermore, when such a compute node required maintenance or reboot, all Pods
on that node would be affected simultaneously. This made the setup unsuitable
for gateway services, which require high reliability and low latency.</p>
<p>To improve this, I migrated the service to a single-layer scheduling model
provided by the cloud platform. In this model, each Pod runs on a dedicated
virtual machine instance provisioned directly from the cloud resource pool.</p>
<p>Kubernetes treats these instances as specialized nodes, allowing Pods to be
scheduled directly onto them. This approach significantly improves isolation, as
each Pod runs in its own environment and does not interfere with others.</p>
<p>After updating the scheduling strategy and migrating all Pods to this model, the
overall stability of the service improved noticeably. During peak traffic
periods, the previous issues of high timeout rates combined with abnormal CPU
behavior were largely eliminated.</p>
]]></content:encoded></item></channel></rss>