<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Troubleshooting on Tech Philosopher</title><link>https://blog.bktus.com/en/tags/troubleshooting/</link><description>Recent content in Troubleshooting on Tech Philosopher</description><generator>Hugo</generator><language>en-US</language><lastBuildDate>Wed, 15 Nov 2023 20:25:05 +0800</lastBuildDate><atom:link href="https://blog.bktus.com/en/tags/troubleshooting/index.xml" rel="self" type="application/rss+xml"/><item><title>Weekly Technical Report for the Fourth Week of February 2023</title><link>https://blog.bktus.com/en/archives/oebjdm/</link><pubDate>Wed, 15 Nov 2023 20:25:05 +0800</pubDate><guid>https://blog.bktus.com/en/archives/oebjdm/</guid><description>&lt;p>This week, I encountered a performance issue that led me to re-evaluate the
quality of some service frameworks, particularly certain Java-based ones.&lt;/p>
&lt;p>Under moderate load—around 35% to 40% CPU utilization—the services began to
experience a large number of timeouts. This was unexpected, as neither CPU nor
memory resources were saturated, and the number of worker threads was
sufficient.&lt;/p>
&lt;p>Using Java profiling tools, I found that most worker threads were in &lt;code>Idle&lt;/code> or&lt;code>Waiting&lt;/code> states. There were no obvious hotspots in business logic, and no
threads appeared to be fully utilized. This suggested that the bottleneck was
not CPU-bound, but likely related to I/O or some form of blocking or
coordination overhead.&lt;/p></description><content:encoded><![CDATA[<p>This week, I encountered a performance issue that led me to re-evaluate the
quality of some service frameworks, particularly certain Java-based ones.</p>
<p>Under moderate load—around 35% to 40% CPU utilization—the services began to
experience a large number of timeouts. This was unexpected, as neither CPU nor
memory resources were saturated, and the number of worker threads was
sufficient.</p>
<p>Using Java profiling tools, I found that most worker threads were in <code>Idle</code> or<code>Waiting</code> states. There were no obvious hotspots in business logic, and no
threads appeared to be fully utilized. This suggested that the bottleneck was
not CPU-bound, but likely related to I/O or some form of blocking or
coordination overhead.</p>
<p>Despite using NIO and the Netty framework, overall throughput remained low.
Based on thread analysis, the system appeared to spend a significant amount of
time waiting rather than executing useful work. The exact root cause was still
unclear, but the symptoms strongly indicated inefficiencies in how the framework
handled concurrency or I/O.</p>
<p>During this period, I attempted to reduce infrastructure costs by scaling down
the number of nodes. However, my initial approach was overly simplistic—I relied
mainly on CPU utilization as the indicator of system capacity.</p>
<p>After reducing the number of nodes and increasing average CPU utilization in the
Beijing region to around 35%–40%, the entire service began to experience
widespread timeouts. Monitoring showed that almost all nodes entered a degraded
state simultaneously.</p>
<p>This incident highlighted an important lesson: CPU utilization alone is not a
reliable indicator of system capacity, especially for I/O-bound systems. A
system can appear underutilized while still being unable to handle additional
load due to hidden bottlenecks.</p>
<p>In parallel, I continued refactoring an old PHP-based gateway service, replacing
it with a new implementation in Java. However, I have reservations about using
Java for high-concurrency gateway scenarios.</p>
<p>The execution model of Java(especially with JDK 8) relies heavily on OS threads.
Without lightweight threads (such as virtual threads), the number of threads per
container is limited (e.g., around 800 threads for a 4-core, 8GB container).
High thread counts introduce significant context-switching overhead, which
limits per-instance throughput and increases the number of containers required
to handle the same traffic.</p>
<p>Initially, I considered implementing the gateway in Go, which provides better
support for high-concurrency workloads and has a mature ecosystem. However, due
to team and organizational constraints, I proceeded with the internally
developed Java framework.</p>
<p>This experience reinforced the importance of aligning technology choices with
workload characteristics, as well as the need to evaluate system capacity beyond
simple resource metrics.</p>
]]></content:encoded></item><item><title>Weekly Technical Report for the Second Week of January 2023</title><link>https://blog.bktus.com/en/archives/1p2hnf/</link><pubDate>Tue, 14 Feb 2023 20:13:32 +0800</pubDate><guid>https://blog.bktus.com/en/archives/1p2hnf/</guid><description>&lt;p>This week was primarily focused on ensuring the stability of various services
ahead of the Chinese New Year.&lt;/p>
&lt;p>Recently, a particular service frequently reported timeouts during peak traffic
hours. I initially notified the service owner to investigate the issue. However,
after several days, no clear root cause was identified.&lt;/p>
&lt;p>Given the severity of the alarms—with timeout rates reaching up to 20% on some
nodes—I decided to step in and troubleshoot the issue myself. During this
period, overall traffic had increased significantly, roughly doubling compared
to the end of December, likely due to the approaching holiday. My first
assumption was insufficient service capacity, so I performed a capacity
expansion.&lt;/p></description><content:encoded><![CDATA[<p>This week was primarily focused on ensuring the stability of various services
ahead of the Chinese New Year.</p>
<p>Recently, a particular service frequently reported timeouts during peak traffic
hours. I initially notified the service owner to investigate the issue. However,
after several days, no clear root cause was identified.</p>
<p>Given the severity of the alarms—with timeout rates reaching up to 20% on some
nodes—I decided to step in and troubleshoot the issue myself. During this
period, overall traffic had increased significantly, roughly doubling compared
to the end of December, likely due to the approaching holiday. My first
assumption was insufficient service capacity, so I performed a capacity
expansion.</p>
<p>However, the expansion did not improve the situation. The timeout rate and alarm
frequency remained largely unchanged.</p>
<p>I then began analyzing the service’s source code. I found that each request
first called a downstream service and then performed an asynchronous database
write. Since the database operation was asynchronous and non-blocking, it was
unlikely to be the bottleneck—although I had initially spent quite some time
investigating that direction. This led me to suspect the downstream service call
instead.</p>
<p>I logged into one of the nodes and examined the logs. It turned out that all
downstream requests were consistently routed to the same IP and
port—specifically, through a Scarecrow node. To eliminate potential issues
caused by Scarecrow forwarding, I migrated the service to the cloud.</p>
<p>However, after switching traffic, I observed a large number of timeouts
occurring in a different geographic region. Even after scaling resources, the
issue persisted, which was quite puzzling. I temporarily rolled back the change.</p>
<p>Curious about why the problem worsened after moving to the cloud, I conducted
further monitoring and analysis. I discovered that after traffic was redirected,
one of the downstream service nodes in the cloud had extremely high CPU usage,
while the others remained underutilized.</p>
<p>This strongly suggested a load balancing issue.</p>
<p>I revisited the service code and confirmed the suspicion: after retrieving the
list of available downstream nodes from the service registry, the client always
selected the first node in the list. As a result, nearly all traffic from a
specific geographic region was routed to a single node.</p>
<p>Previously, when traffic passed through Scarecrow, load balancing was implicitly
handled at that layer. This masked the issue, as Scarecrow nodes are proxy
services capable of handling high concurrency without significant performance
degradation. Once traffic bypassed Scarecrow and was routed directly to business
nodes, the imbalance became critical, leading to severe overload and timeouts.</p>
<p>To validate this hypothesis, I checked the monitoring data for Scarecrow and
found that one node had indeed been heavily overloaded, with CPU usage exceeding
95%.</p>
<p>After identifying the root cause, I proceeded with a two-step solution. First,
to quickly mitigate the issue before the holiday, I scaled up the downstream
service by doubling the CPU cores of each node. This significantly increased
processing capacity and reduced timeout rates.</p>
<p>Then, I gradually shifted traffic back to the cloud. As expected, one node
initially experienced higher CPU usage, but the system remained stable overall.
Eventually, the service stabilized, alarms were cleared, and the timeout rate
dropped to zero.</p>
<p>The final step was to fix the underlying issue. After the holiday, I modified
the service code to introduce a proper load balancing mechanism using a random
selection strategy. Once the new version was deployed, traffic was evenly
distributed across all downstream nodes, and CPU usage became balanced.</p>
<p>At that point, the problem was fully resolved.</p>
]]></content:encoded></item><item><title>Technical Review for the Fourth Week of December 2022</title><link>https://blog.bktus.com/en/archives/uxk641/</link><pubDate>Thu, 19 Jan 2023 19:31:54 +0800</pubDate><guid>https://blog.bktus.com/en/archives/uxk641/</guid><description>&lt;p>This week I contracted COVID-19 and stayed at home for nine days. Despite being
unwell, the most critical task was assessing the impact of launching a
high-traffic mini program on the core services I was responsible for.&lt;/p>
&lt;p>This mini program aligned closely with user demand at the time, and a
significant traffic surge was expected. Previously, I had already evaluated and
scaled the system for a major feature launch. However, after sending
notifications to hundreds of millions of users, the system experienced a large
number of timeouts.&lt;/p></description><content:encoded><![CDATA[<p>This week I contracted COVID-19 and stayed at home for nine days. Despite being
unwell, the most critical task was assessing the impact of launching a
high-traffic mini program on the core services I was responsible for.</p>
<p>This mini program aligned closely with user demand at the time, and a
significant traffic surge was expected. Previously, I had already evaluated and
scaled the system for a major feature launch. However, after sending
notifications to hundreds of millions of users, the system experienced a large
number of timeouts.</p>
<p>At around 8:00 a.m., I was informed that the login interface was experiencing
severe timeout issues. After connecting remotely, I observed that traffic had
increased more than twenty times.</p>
<p>My initial assumption was that the number of service containers was insufficient
to handle the surge. However, log analysis quickly ruled this out.</p>
<p>The issue was traced to two main causes.</p>
<p>First, an internal interface was rate-limited, and the incoming traffic exceeded
its configured threshold. This was resolved quickly by coordinating with the
relevant team to increase the rate limit.</p>
<p>Second, the cache layer became a bottleneck. A large number of returning users
triggered heavy reads from the database to rebuild cache entries. As traffic
continued to grow, the backend database reached its I/O limit, leading to
cascading timeouts.</p>
<p>To address this, we expanded several core cache clusters, increasing both
capacity and the number of nodes to better handle subsequent traffic spikes.</p>
<p>In addition to incident handling, I also worked on improving the stability of a
gateway service by adjusting its Kubernetes scheduling strategy.</p>
<p>Previously, the service was deployed on compute nodes that were themselves
virtualized from on-premise physical machines. On top of this layer, containers
were created and Pods were scheduled within them. This multi-layer setup led to
resource contention and weak isolation, as multiple Pods shared the same
underlying operating system and competed for CPU, memory, and network resources.</p>
<p>Furthermore, when such a compute node required maintenance or reboot, all Pods
on that node would be affected simultaneously. This made the setup unsuitable
for gateway services, which require high reliability and low latency.</p>
<p>To improve this, I migrated the service to a single-layer scheduling model
provided by the cloud platform. In this model, each Pod runs on a dedicated
virtual machine instance provisioned directly from the cloud resource pool.</p>
<p>Kubernetes treats these instances as specialized nodes, allowing Pods to be
scheduled directly onto them. This approach significantly improves isolation, as
each Pod runs in its own environment and does not interfere with others.</p>
<p>After updating the scheduling strategy and migrating all Pods to this model, the
overall stability of the service improved noticeably. During peak traffic
periods, the previous issues of high timeout rates combined with abnormal CPU
behavior were largely eliminated.</p>
]]></content:encoded></item><item><title>Technical Review for the Third Week of December 2022</title><link>https://blog.bktus.com/en/archives/3nav5g/</link><pubDate>Thu, 19 Jan 2023 17:40:11 +0800</pubDate><guid>https://blog.bktus.com/en/archives/3nav5g/</guid><description>&lt;p>This week, my main focus was optimizing a Java service that had long been
experiencing abnormal CPU behavior.&lt;/p>
&lt;p>At first glance, the issue appeared to be that CPU usage could not scale up
under load. A natural starting point was to check whether the service was
constrained by an insufficient number of worker threads. However, further
observation revealed a different pattern: CPU usage could increase, but once it
did, the system quickly ran into severe timeout issues.&lt;/p></description><content:encoded><![CDATA[<p>This week, my main focus was optimizing a Java service that had long been
experiencing abnormal CPU behavior.</p>
<p>At first glance, the issue appeared to be that CPU usage could not scale up
under load. A natural starting point was to check whether the service was
constrained by an insufficient number of worker threads. However, further
observation revealed a different pattern: CPU usage could increase, but once it
did, the system quickly ran into severe timeout issues.</p>
<p>This suggested that the problem was not about CPU utilization itself, but rather
about what happened when the system attempted to utilize more CPU.</p>
<p>Historically, this service had already been flagged for poor performance, and
there were even suggestions to deprecate it. Based on this background, I
suspected that the root cause might lie in the framework rather than the
business logic.</p>
<p>I began by reviewing the framework implementation. The service uses Netty as its
NIO server framework. Incoming requests are handled by I/O threads and then
dispatched to worker threads for business processing.</p>
<p>In a previous issue, insufficient worker threads had caused performance
bottlenecks under high I/O load. However, in this case, logs indicated that the
number of worker threads was sufficient. This led me to consider whether the
issue might originate from the client side.</p>
<p>Within the microservice architecture, this service also acts as a client when
invoking other services. This became a key investigation direction.</p>
<p>By analyzing the code, I found that the framework uses a proxy class
(<code>ObjectProxy</code>) based on Java’s <code>InvocationHandler</code> to intercept RPC calls. When
a service method is invoked, <code>ObjectProxy</code> interacts with a <code>ProtocolInvoker</code> to
retrieve a list of available downstream nodes from the service registry
(refreshed every 30 seconds). The list is then passed to a <code>LoadBalancer</code> to
select a target node.</p>
<p>The actual invocation is performed by a protocol-specific <code>Invoker</code>, which
maintains long-lived connections to downstream services. Depending on whether
the call is synchronous or asynchronous, the request is sent and handled
differently.</p>
<p>Each downstream service consists of multiple nodes. For each node, the framework
creates:</p>
<ul>
<li>Two I/O threads (for NIO event handling)</li>
<li>Multiple TCP connections (by default equal to the number of CPU cores)</li>
</ul>
<p>Each I/O thread manages a <code>Selector</code>, which polls for events on its associated
connections. For each TCP connection, a <code>TCPSession</code> is maintained.</p>
<p>When a request is sent, a <code>Ticket</code> object is created to track the
request-response lifecycle:</p>
<ul>
<li>For synchronous calls, the thread blocks until a response is received</li>
<li>For asynchronous calls, the <code>Ticket</code> stores a callback, which is triggered
upon response arrival</li>
</ul>
<p>At the network level, the framework relies on Java NIO (<code>SocketChannel</code>).
Incoming data is buffered, and packet boundaries are determined using a
length-prefixed protocol. The framework checks whether the buffer contains
enough bytes to form a complete packet before processing it.</p>
<p>Once a full packet is received, it is dispatched for further processing. At this
stage, a worker thread from a thread pool is used to handle the <code>Ticket</code>. This
thread pool is relatively small, with:</p>
<ul>
<li>Core size = number of CPU cores</li>
<li>Maximum size = 2 × number of CPU cores</li>
</ul>
<p>From a design perspective, this architecture is fairly standard and widely used.
After thoroughly reviewing the client-side implementation, I did not find any
obvious flaws.</p>
<p>At this point, it became increasingly likely that the issue was not on the
client side, but rather on the server side or in the interaction between
services. Further investigation would need to focus on the downstream services
and their behavior under load.</p>
]]></content:encoded></item><item><title>Weekly Technical Report for the Fourth Week of November 2022</title><link>https://blog.bktus.com/en/archives/99lv21/</link><pubDate>Mon, 28 Nov 2022 00:47:29 +0800</pubDate><guid>https://blog.bktus.com/en/archives/99lv21/</guid><description>&lt;p>This week, my main task was optimizing a Java service that exhibited abnormal
timeout behavior.&lt;/p>
&lt;p>Even under low production traffic, the service experienced batch timeouts. At
the same time, CPU usage remained consistently low and never increased under
load. This strongly suggested that the system was not CPU-bound, but rather
blocked on some operations.&lt;/p>
&lt;p>Most services I have worked on, including this one, are IO-intensive. They rely
heavily on RPC calls, where worker threads often block while waiting for
responses. To mitigate this, such systems typically use a large number of worker
threads to ensure that new requests can still be processed even when many
threads are blocked.&lt;/p></description><content:encoded><![CDATA[<p>This week, my main task was optimizing a Java service that exhibited abnormal
timeout behavior.</p>
<p>Even under low production traffic, the service experienced batch timeouts. At
the same time, CPU usage remained consistently low and never increased under
load. This strongly suggested that the system was not CPU-bound, but rather
blocked on some operations.</p>
<p>Most services I have worked on, including this one, are IO-intensive. They rely
heavily on RPC calls, where worker threads often block while waiting for
responses. To mitigate this, such systems typically use a large number of worker
threads to ensure that new requests can still be processed even when many
threads are blocked.</p>
<p>However, this approach has inherent limitations. Although Java uses NIO for
network I/O, the actual business logic is still executed by worker threads. Once
a worker thread is blocked on an RPC call, it cannot proceed until the call
completes or times out. This makes the system highly sensitive to downstream
latency.</p>
<p>After investigation, I discovered that the issue was caused by a framework
configuration problem. The configured number of worker threads was not being
properly loaded, causing the service to fall back to its default setting—where
the number of worker threads equals the number of CPU cores.</p>
<p>As a result, when worker threads were occupied with downstream RPC calls, even
small fluctuations in latency would quickly exhaust available threads, leading
to a large number of timeouts. Meanwhile, CPU usage remained low because most
threads were blocked waiting on I/O.</p>
<p>To troubleshoot the issue, I first enabled thread name logging by adding <code>%t</code> to
the logback pattern. Then, I performed load testing on a single node to observe
request handling under controlled QPS. I also added <code>StopWatch</code> instrumentation
to measure processing time within the request flow.</p>
<p>From the logs, I noticed that only a small number of threads were handling
requests, which was clearly abnormal. Using <code>jstack</code>, I confirmed that only four
threads were actively processing the main request logic.</p>
<p><img loading="lazy" src="https://img.bktus.com/DGCRxfEA.webp"></p>
<p>Further analysis showed that the service relies on Netty, and request handling
ultimately occurs within the Netty worker thread pool (e.g., threads with prefix<code>nioEventLoopGroup-*</code>). CPU time distribution also confirmed that only these few
threads were actively processing requests.</p>
<p>In production, services of this type typically configure hundreds of worker
threads (e.g., 800–1000), along with strict RPC timeouts to prevent excessive
blocking. After fixing the framework configuration, the issue was resolved.</p>
<p>In addition, I identified another risk in one of the service’s core interfaces.
It made downstream HTTP calls using OkHttp, but no timeout was configured. In
extreme cases, this could cause worker threads to block indefinitely if the
downstream service was slow to respond.</p>
<p>To address this, I configured proper timeouts for all HTTP calls:</p>
<ul>
<li>connection timeout</li>
<li>read timeout</li>
<li>write timeout</li>
</ul>
<p>This is essential to prevent cascading failures and maintain system stability.</p>
<p>I also enabled OkHttp’s <code>ConnectionPool</code> to improve performance under high
concurrency.</p>
<p><img loading="lazy" src="https://img.bktus.com/tQJn9FVd.webp"></p>
<p>According to the documentation, <code>ConnectionPool</code> allows multiple HTTP/HTTP2
requests to reuse the same connection, reducing overhead. However, this requires
the server side to support persistent connections.</p>
<p>Besides this optimization work, I also spent time preparing for Tencent Cloud’s
Advanced Architect TCP exam. Compared to previous certifications, this exam was
significantly more challenging and required deeper understanding.</p>
<p>Due to limited time during workdays, I concentrated my review late into the
night before the exam—studying until 5 a.m. on Saturday. Fortunately, the effort
paid off, and I passed the exam. I plan to write a separate article summarizing
that experience.</p>
]]></content:encoded></item></channel></rss>