Weekly Technical Report for the Fourth Week of February 2023

This week, I encountered a performance issue that led me to re-evaluate the quality of some service frameworks, particularly certain Java-based ones. Under moderate load—around 35% to 40% CPU utilization—the services began to experience a large number of timeouts. This was unexpected, as neither CPU nor memory resources were saturated, and the number of worker threads was sufficient. Using Java profiling tools, I found that most worker threads were in Idle orWaiting states. There were no obvious hotspots in business logic, and no threads appeared to be fully utilized. This suggested that the bottleneck was not CPU-bound, but likely related to I/O or some form of blocking or coordination overhead. ...

November 15, 2023 · Saturneric

Weekly Technical Report for the Second Week of January 2023

This week was primarily focused on ensuring the stability of various services ahead of the Chinese New Year. Recently, a particular service frequently reported timeouts during peak traffic hours. I initially notified the service owner to investigate the issue. However, after several days, no clear root cause was identified. Given the severity of the alarms—with timeout rates reaching up to 20% on some nodes—I decided to step in and troubleshoot the issue myself. During this period, overall traffic had increased significantly, roughly doubling compared to the end of December, likely due to the approaching holiday. My first assumption was insufficient service capacity, so I performed a capacity expansion. ...

February 14, 2023 · Saturneric

Technical Review for the Fourth Week of December 2022

This week I contracted COVID-19 and stayed at home for nine days. Despite being unwell, the most critical task was assessing the impact of launching a high-traffic mini program on the core services I was responsible for. This mini program aligned closely with user demand at the time, and a significant traffic surge was expected. Previously, I had already evaluated and scaled the system for a major feature launch. However, after sending notifications to hundreds of millions of users, the system experienced a large number of timeouts. ...

January 19, 2023 · Saturneric

Technical Review for the Third Week of December 2022

This week, my main focus was optimizing a Java service that had long been experiencing abnormal CPU behavior. At first glance, the issue appeared to be that CPU usage could not scale up under load. A natural starting point was to check whether the service was constrained by an insufficient number of worker threads. However, further observation revealed a different pattern: CPU usage could increase, but once it did, the system quickly ran into severe timeout issues. ...

January 19, 2023 · Saturneric

Weekly Technical Report for the Fourth Week of November 2022

This week, my main task was optimizing a Java service that exhibited abnormal timeout behavior. Even under low production traffic, the service experienced batch timeouts. At the same time, CPU usage remained consistently low and never increased under load. This strongly suggested that the system was not CPU-bound, but rather blocked on some operations. Most services I have worked on, including this one, are IO-intensive. They rely heavily on RPC calls, where worker threads often block while waiting for responses. To mitigate this, such systems typically use a large number of worker threads to ensure that new requests can still be processed even when many threads are blocked. ...

November 28, 2022 · Saturneric