Tencent

Recently, compliance requirements have increasingly started to affect technical work. There are frequent requests to implement compliance-related features or to complete compliance assessments. From my perspective, these requirements mainly focus on user data governance, including standardized access control and gradually giving users more control over their own data. At the same time, organizational and personnel changes are also having a noticeable impact on the system. For example, when a business domain is transferred to another department, shared resources such as databases become a point of contention. Cost allocation then becomes a practical issue. Even within the same company, internal accounting mechanisms make these discussions unavoidable, and disputes over resource usage are quite common. ...

This week, I encountered a performance issue that led me to re-evaluate the quality of some service frameworks, particularly certain Java-based ones. Under moderate load—around 35% to 40% CPU utilization—the services began to experience a large number of timeouts. This was unexpected, as neither CPU nor memory resources were saturated, and the number of worker threads was sufficient. Using Java profiling tools, I found that most worker threads were in Idle or Waiting states. There were no obvious hotspots in business logic, and no threads appeared to be fully utilized. This suggested that the bottleneck was not CPU-bound, but likely related to I/O or some form of blocking or coordination overhead. ...

This week was focused on addressing a risk identified before the holiday period. The issue involved a service that used Redis without setting TTLs for keys, instead relying entirely on Redis eviction policies. The instance was configured with an LRU-based eviction strategy, which may appear sufficient at first glance, but introduces significant risks under bursty write workloads. During periods of high write traffic, Redis is forced to aggressively trigger eviction in order to free up memory. This process consumes considerable resources, which can degrade normal operations such as read and write requests. As a result, latency at the application layer can fluctuate significantly. I have encountered this issue multiple times in production. ...

From the end of January to early February, the Chinese New Year period required on-call support to ensure system stability. During this time, the best possible outcome is simple: no incidents. Fortunately, everything remained stable, and no major issues occurred. This week, I mainly evaluated the impact of a new business requirement. One important lesson I have learned is that, when working on complex systems—especially those you are not fully familiar with—the safest approach is to minimize changes as much as possible. ...

This week was primarily focused on ensuring the stability of various services ahead of the Chinese New Year. Recently, a particular service frequently reported timeouts during peak traffic hours. I initially notified the service owner to investigate the issue. However, after several days, no clear root cause was identified. Given the severity of the alarms—with timeout rates reaching up to 20% on some nodes—I decided to step in and troubleshoot the issue myself. During this period, overall traffic had increased significantly, roughly doubling compared to the end of December, likely due to the approaching holiday. My first assumption was insufficient service capacity, so I performed a capacity expansion. ...

As we enter 2023, the year ahead is expected to be challenging. One major task is migrating all services from physical servers to the cloud. In parallel, several new team members need to be onboarded quickly and brought up to a level where they can independently handle production issues and optimize services. This will allow me to gradually transfer routine work and focus on more long-term and critical goals. At a personal level, I also feel that I have reached a stage where my technical growth will significantly influence my direction over the next 7–8 years. ...

This week I contracted COVID-19 and stayed at home for nine days. Despite being unwell, the most critical task was assessing the impact of launching a high-traffic mini program on the core services I was responsible for. This mini program aligned closely with user demand at the time, and a significant traffic surge was expected. Previously, I had already evaluated and scaled the system for a major feature launch. However, after sending notifications to hundreds of millions of users, the system experienced a large number of timeouts. ...

This week, my main focus was optimizing a Java service that had long been experiencing abnormal CPU behavior. At first glance, the issue appeared to be that CPU usage could not scale up under load. A natural starting point was to check whether the service was constrained by an insufficient number of worker threads. However, further observation revealed a different pattern: CPU usage could increase, but once it did, the system quickly ran into severe timeout issues. ...

This week’s work focused on reviewing the systems I am responsible for and identifying potential risks, particularly in preparation for cloud migration. Several key issues were identified, mainly around: how to safely migrate existing data to the cloud how to transform the current single-region deployment into a multi-region architecture how to resolve inconsistencies between on-premise and cloud data Another important finding was that some services still rely on on-premise databases, which should eventually be phased out. However, these are legacy systems, and making direct changes carries risk. Before taking action, it is necessary to thoroughly analyze: ...

This week’s work, to summarize, is mainly to put a core service on the cloud, and then constantly switch the nodes under the cloud into traffic forwarding nodes. The first step in the cloud is to deploy the service node in the cloud environment: migrate the configuration files, environment, and then compile the image for the cloud environment according to the stable version of the code, and then let the service run up in the cloud environment. ...