This week, my main task was optimizing a Java service that exhibited abnormal timeout behavior.

Even under low production traffic, the service experienced batch timeouts. At the same time, CPU usage remained consistently low and never increased under load. This strongly suggested that the system was not CPU-bound, but rather blocked on some operations.

Most services I have worked on, including this one, are IO-intensive. They rely heavily on RPC calls, where worker threads often block while waiting for responses. To mitigate this, such systems typically use a large number of worker threads to ensure that new requests can still be processed even when many threads are blocked.

However, this approach has inherent limitations. Although Java uses NIO for network I/O, the actual business logic is still executed by worker threads. Once a worker thread is blocked on an RPC call, it cannot proceed until the call completes or times out. This makes the system highly sensitive to downstream latency.

After investigation, I discovered that the issue was caused by a framework configuration problem. The configured number of worker threads was not being properly loaded, causing the service to fall back to its default setting—where the number of worker threads equals the number of CPU cores.

As a result, when worker threads were occupied with downstream RPC calls, even small fluctuations in latency would quickly exhaust available threads, leading to a large number of timeouts. Meanwhile, CPU usage remained low because most threads were blocked waiting on I/O.

To troubleshoot the issue, I first enabled thread name logging by adding %t to the logback pattern. Then, I performed load testing on a single node to observe request handling under controlled QPS. I also added StopWatch instrumentation to measure processing time within the request flow.

From the logs, I noticed that only a small number of threads were handling requests, which was clearly abnormal. Using jstack, I confirmed that only four threads were actively processing the main request logic.

Further analysis showed that the service relies on Netty, and request handling ultimately occurs within the Netty worker thread pool (e.g., threads with prefix nioEventLoopGroup-*). CPU time distribution also confirmed that only these few threads were actively processing requests.

In production, services of this type typically configure hundreds of worker threads (e.g., 800–1000), along with strict RPC timeouts to prevent excessive blocking. After fixing the framework configuration, the issue was resolved.

In addition, I identified another risk in one of the service’s core interfaces. It made downstream HTTP calls using OkHttp, but no timeout was configured. In extreme cases, this could cause worker threads to block indefinitely if the downstream service was slow to respond.

To address this, I configured proper timeouts for all HTTP calls:

  • connection timeout
  • read timeout
  • write timeout

This is essential to prevent cascading failures and maintain system stability.

I also enabled OkHttp’s ConnectionPool to improve performance under high concurrency.

According to the documentation, ConnectionPool allows multiple HTTP/HTTP2 requests to reuse the same connection, reducing overhead. However, this requires the server side to support persistent connections.

Besides this optimization work, I also spent time preparing for Tencent Cloud’s Advanced Architect TCP exam. Compared to previous certifications, this exam was significantly more challenging and required deeper understanding.

Due to limited time during workdays, I concentrated my review late into the night before the exam—studying until 5 a.m. on Saturday. Fortunately, the effort paid off, and I passed the exam. I plan to write a separate article summarizing that experience.