---
title: "Weekly Technical Report for the Second Week of January 2023"
url: https://blog.bktus.com/en/archives/1p2hnf/
author: "Saturneric"
date: 2023-02-14
language: en-US
categories: ["Feelings"]
tags: ["weeklies","Tencent","Cloud Migration","Troubleshooting"]
translation_zh: https://blog.bktus.com/archives/14abwa/
---

# Weekly Technical Report for the Second Week of January 2023

This week was primarily focused on ensuring the stability of various services
ahead of the Chinese New Year.

Recently, a particular service frequently reported timeouts during peak traffic
hours. I initially notified the service owner to investigate the issue. However,
after several days, no clear root cause was identified.

Given the severity of the alarms—with timeout rates reaching up to 20% on some
nodes—I decided to step in and troubleshoot the issue myself. During this
period, overall traffic had increased significantly, roughly doubling compared
to the end of December, likely due to the approaching holiday. My first
assumption was insufficient service capacity, so I performed a capacity
expansion.

However, the expansion did not improve the situation. The timeout rate and alarm
frequency remained largely unchanged.

I then began analyzing the service’s source code. I found that each request
first called a downstream service and then performed an asynchronous database
write. Since the database operation was asynchronous and non-blocking, it was
unlikely to be the bottleneck—although I had initially spent quite some time
investigating that direction. This led me to suspect the downstream service call
instead.

I logged into one of the nodes and examined the logs. It turned out that all
downstream requests were consistently routed to the same IP and
port—specifically, through a Scarecrow node. To eliminate potential issues
caused by Scarecrow forwarding, I migrated the service to the cloud.

However, after switching traffic, I observed a large number of timeouts
occurring in a different geographic region. Even after scaling resources, the
issue persisted, which was quite puzzling. I temporarily rolled back the change.

Curious about why the problem worsened after moving to the cloud, I conducted
further monitoring and analysis. I discovered that after traffic was redirected,
one of the downstream service nodes in the cloud had extremely high CPU usage,
while the others remained underutilized.

This strongly suggested a load balancing issue.

I revisited the service code and confirmed the suspicion: after retrieving the
list of available downstream nodes from the service registry, the client always
selected the first node in the list. As a result, nearly all traffic from a
specific geographic region was routed to a single node.

Previously, when traffic passed through Scarecrow, load balancing was implicitly
handled at that layer. This masked the issue, as Scarecrow nodes are proxy
services capable of handling high concurrency without significant performance
degradation. Once traffic bypassed Scarecrow and was routed directly to business
nodes, the imbalance became critical, leading to severe overload and timeouts.

To validate this hypothesis, I checked the monitoring data for Scarecrow and
found that one node had indeed been heavily overloaded, with CPU usage exceeding
95%.

After identifying the root cause, I proceeded with a two-step solution. First,
to quickly mitigate the issue before the holiday, I scaled up the downstream
service by doubling the CPU cores of each node. This significantly increased
processing capacity and reduced timeout rates.

Then, I gradually shifted traffic back to the cloud. As expected, one node
initially experienced higher CPU usage, but the system remained stable overall.
Eventually, the service stabilized, alarms were cleared, and the timeout rate
dropped to zero.

The final step was to fix the underlying issue. After the holiday, I modified
the service code to introduce a proper load balancing mechanism using a random
selection strategy. Once the new version was deployed, traffic was evenly
distributed across all downstream nodes, and CPU usage became balanced.

At that point, the problem was fully resolved.
