This week, I encountered a performance issue that led me to re-evaluate the quality of some service frameworks, particularly certain Java-based ones.

Under moderate load—around 35% to 40% CPU utilization—the services began to experience a large number of timeouts. This was unexpected, as neither CPU nor memory resources were saturated, and the number of worker threads was sufficient.

Using Java profiling tools, I found that most worker threads were in Idle or Waiting states. There were no obvious hotspots in business logic, and no threads appeared to be fully utilized. This suggested that the bottleneck was not CPU-bound, but likely related to I/O or some form of blocking or coordination overhead.

Despite using NIO and the Netty framework, overall throughput remained low. Based on thread analysis, the system appeared to spend a significant amount of time waiting rather than executing useful work. The exact root cause was still unclear, but the symptoms strongly indicated inefficiencies in how the framework handled concurrency or I/O.

During this period, I attempted to reduce infrastructure costs by scaling down the number of nodes. However, my initial approach was overly simplistic—I relied mainly on CPU utilization as the indicator of system capacity.

After reducing the number of nodes and increasing average CPU utilization in the Beijing region to around 35%–40%, the entire service began to experience widespread timeouts. Monitoring showed that almost all nodes entered a degraded state simultaneously.

This incident highlighted an important lesson: CPU utilization alone is not a reliable indicator of system capacity, especially for I/O-bound systems. A system can appear underutilized while still being unable to handle additional load due to hidden bottlenecks.

In parallel, I continued refactoring an old PHP-based gateway service, replacing it with a new implementation in Java. However, I have reservations about using Java for high-concurrency gateway scenarios.

The execution model of Java(especially with JDK 8) relies heavily on OS threads. Without lightweight threads (such as virtual threads), the number of threads per container is limited (e.g., around 800 threads for a 4-core, 8GB container). High thread counts introduce significant context-switching overhead, which limits per-instance throughput and increases the number of containers required to handle the same traffic.

Initially, I considered implementing the gateway in Go, which provides better support for high-concurrency workloads and has a mature ecosystem. However, due to team and organizational constraints, I proceeded with the internally developed Java framework.

This experience reinforced the importance of aligning technology choices with workload characteristics, as well as the need to evaluate system capacity beyond simple resource metrics.