<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Cloud Migration on Tech Philosopher</title><link>https://blog.bktus.com/en/tags/cloud-migration/</link><description>Recent content in Cloud Migration on Tech Philosopher</description><generator>Hugo</generator><language>en-US</language><lastBuildDate>Tue, 14 Feb 2023 20:13:32 +0800</lastBuildDate><atom:link href="https://blog.bktus.com/en/tags/cloud-migration/index.xml" rel="self" type="application/rss+xml"/><item><title>Weekly Technical Report for the Second Week of January 2023</title><link>https://blog.bktus.com/en/archives/1p2hnf/</link><pubDate>Tue, 14 Feb 2023 20:13:32 +0800</pubDate><guid>https://blog.bktus.com/en/archives/1p2hnf/</guid><description>&lt;p>This week was primarily focused on ensuring the stability of various services
ahead of the Chinese New Year.&lt;/p>
&lt;p>Recently, a particular service frequently reported timeouts during peak traffic
hours. I initially notified the service owner to investigate the issue. However,
after several days, no clear root cause was identified.&lt;/p>
&lt;p>Given the severity of the alarms—with timeout rates reaching up to 20% on some
nodes—I decided to step in and troubleshoot the issue myself. During this
period, overall traffic had increased significantly, roughly doubling compared
to the end of December, likely due to the approaching holiday. My first
assumption was insufficient service capacity, so I performed a capacity
expansion.&lt;/p></description><content:encoded><![CDATA[<p>This week was primarily focused on ensuring the stability of various services
ahead of the Chinese New Year.</p>
<p>Recently, a particular service frequently reported timeouts during peak traffic
hours. I initially notified the service owner to investigate the issue. However,
after several days, no clear root cause was identified.</p>
<p>Given the severity of the alarms—with timeout rates reaching up to 20% on some
nodes—I decided to step in and troubleshoot the issue myself. During this
period, overall traffic had increased significantly, roughly doubling compared
to the end of December, likely due to the approaching holiday. My first
assumption was insufficient service capacity, so I performed a capacity
expansion.</p>
<p>However, the expansion did not improve the situation. The timeout rate and alarm
frequency remained largely unchanged.</p>
<p>I then began analyzing the service’s source code. I found that each request
first called a downstream service and then performed an asynchronous database
write. Since the database operation was asynchronous and non-blocking, it was
unlikely to be the bottleneck—although I had initially spent quite some time
investigating that direction. This led me to suspect the downstream service call
instead.</p>
<p>I logged into one of the nodes and examined the logs. It turned out that all
downstream requests were consistently routed to the same IP and
port—specifically, through a Scarecrow node. To eliminate potential issues
caused by Scarecrow forwarding, I migrated the service to the cloud.</p>
<p>However, after switching traffic, I observed a large number of timeouts
occurring in a different geographic region. Even after scaling resources, the
issue persisted, which was quite puzzling. I temporarily rolled back the change.</p>
<p>Curious about why the problem worsened after moving to the cloud, I conducted
further monitoring and analysis. I discovered that after traffic was redirected,
one of the downstream service nodes in the cloud had extremely high CPU usage,
while the others remained underutilized.</p>
<p>This strongly suggested a load balancing issue.</p>
<p>I revisited the service code and confirmed the suspicion: after retrieving the
list of available downstream nodes from the service registry, the client always
selected the first node in the list. As a result, nearly all traffic from a
specific geographic region was routed to a single node.</p>
<p>Previously, when traffic passed through Scarecrow, load balancing was implicitly
handled at that layer. This masked the issue, as Scarecrow nodes are proxy
services capable of handling high concurrency without significant performance
degradation. Once traffic bypassed Scarecrow and was routed directly to business
nodes, the imbalance became critical, leading to severe overload and timeouts.</p>
<p>To validate this hypothesis, I checked the monitoring data for Scarecrow and
found that one node had indeed been heavily overloaded, with CPU usage exceeding
95%.</p>
<p>After identifying the root cause, I proceeded with a two-step solution. First,
to quickly mitigate the issue before the holiday, I scaled up the downstream
service by doubling the CPU cores of each node. This significantly increased
processing capacity and reduced timeout rates.</p>
<p>Then, I gradually shifted traffic back to the cloud. As expected, one node
initially experienced higher CPU usage, but the system remained stable overall.
Eventually, the service stabilized, alarms were cleared, and the timeout rate
dropped to zero.</p>
<p>The final step was to fix the underlying issue. After the holiday, I modified
the service code to introduce a proper load balancing mechanism using a random
selection strategy. Once the new version was deployed, traffic was evenly
distributed across all downstream nodes, and CPU usage became balanced.</p>
<p>At that point, the problem was fully resolved.</p>
]]></content:encoded></item><item><title>Weekly Technical Report for the Second Week of December 2022</title><link>https://blog.bktus.com/en/archives/qeqg8u/</link><pubDate>Wed, 14 Dec 2022 00:16:19 +0800</pubDate><guid>https://blog.bktus.com/en/archives/qeqg8u/</guid><description>&lt;p>This week’s work focused on reviewing the systems I am responsible for and
identifying potential risks, particularly in preparation for cloud migration.&lt;/p>
&lt;p>Several key issues were identified, mainly around:&lt;/p>
&lt;ul>
&lt;li>how to safely migrate existing data to the cloud&lt;/li>
&lt;li>how to transform the current single-region deployment into a multi-region
architecture&lt;/li>
&lt;li>how to resolve inconsistencies between on-premise and cloud data&lt;/li>
&lt;/ul>
&lt;p>Another important finding was that some services still rely on on-premise
databases, which should eventually be phased out. However, these are legacy
systems, and making direct changes carries risk. Before taking action, it is
necessary to thoroughly analyze:&lt;/p></description><content:encoded><![CDATA[<p>This week’s work focused on reviewing the systems I am responsible for and
identifying potential risks, particularly in preparation for cloud migration.</p>
<p>Several key issues were identified, mainly around:</p>
<ul>
<li>how to safely migrate existing data to the cloud</li>
<li>how to transform the current single-region deployment into a multi-region
architecture</li>
<li>how to resolve inconsistencies between on-premise and cloud data</li>
</ul>
<p>Another important finding was that some services still rely on on-premise
databases, which should eventually be phased out. However, these are legacy
systems, and making direct changes carries risk. Before taking action, it is
necessary to thoroughly analyze:</p>
<ul>
<li>the existing data migration mechanisms</li>
<li>supporting services involved in synchronization</li>
<li>detailed code paths that may affect data consistency</li>
</ul>
<p>Early identification of potential issues is critical to avoid failures during
migration.</p>
<p>To ensure a safe migration process, comprehensive monitoring is required. Key
metrics include:</p>
<ul>
<li>request success rate</li>
<li>timeout rate</li>
<li>write failure rate</li>
<li>data inconsistency rate</li>
</ul>
<p>These should be monitored through both metrics dashboards and log analysis to
provide full visibility into system behavior.</p>
<p>For multi-region deployment, the current design direction is to adopt a
master–replica architecture:</p>
<ul>
<li>writes are directed to the primary node</li>
<li>reads are served from replicas</li>
<li>replication is one-way from primary to replicas</li>
</ul>
<p>The goals of this architecture are:</p>
<ul>
<li>improving system stability</li>
<li>reducing latency for geographically distributed users</li>
<li>minimizing the impact of cross-region network instability</li>
</ul>
<p>However, replication delay is an unavoidable aspect of this design. It is
necessary to define an acceptable delay threshold and manage it through both
theoretical understanding and real-time monitoring.</p>
<p>In addition to system-level considerations, I also realized the importance of
mastering a scripting language. For tasks involving repetitive operations or
data analysis, tools like Python can significantly improve efficiency.</p>
<p>That said, it is not always appropriate to use Python for building large-scale
systems. Different programming languages serve different purposes while many can
accomplish similar tasks, some are better suited for specific scenarios.
Choosing the right tool for the right problem is essential.</p>
]]></content:encoded></item><item><title>Weekly Technical Report for the First Week of December 2022</title><link>https://blog.bktus.com/en/archives/uud8xf/</link><pubDate>Fri, 09 Dec 2022 21:48:38 +0800</pubDate><guid>https://blog.bktus.com/en/archives/uud8xf/</guid><description>&lt;p>This week’s work, to summarize, is mainly to put a core service on the cloud,
and then constantly switch the nodes under the cloud into traffic forwarding
nodes. The first step in the cloud is to deploy the service node in the cloud
environment: migrate the configuration files, environment, and then compile the
image for the cloud environment according to the stable version of the code, and
then let the service run up in the cloud environment.&lt;/p></description><content:encoded><![CDATA[<p>This week’s work, to summarize, is mainly to put a core service on the cloud,
and then constantly switch the nodes under the cloud into traffic forwarding
nodes. The first step in the cloud is to deploy the service node in the cloud
environment: migrate the configuration files, environment, and then compile the
image for the cloud environment according to the stable version of the code, and
then let the service run up in the cloud environment.</p>
<p>After the service runs up and the test is completed, the node on the cloud is
currently not any traffic, this time you need to forward some of the traffic
under the cloud to the cloud, the first thing is to replace some of the nodes
under the cloud with forwarding nodes, the role of the forwarding node is to
forward the traffic from the main call to the node on the cloud. Later, this
part of the traffic can be used to observe the working status of the nodes on
the cloud and check for anomalies, which can be called “traffic grayscale”.</p>
<p>Traffic gray scale is generally controlled in the overall traffic 1 %-5% role,
depending on the criticality of the service to adjust the proportion of the test
environment can be more appropriate, 25 %-50% can be. Traffic gray scale is when
a problem occurs, or when you receive a problem ticket, you just need to shut
down the traffic forwarding node under the cloud.</p>
<p>Of course, everything above must first be operated in the test environment, test
properly and then follow the same program to carefully operate the production
environment. Some of the background system is intricate, a call involving
multiple services, business logic is confusing, this time try to control the
variables for a period of time to make only one change, and so observe a period
of time, the situation stabilized and then do the next operation. Or you can
split the big step into several small steps, a period of time to do only a small
step, after observing a period of time and then continue to promote the next
small step. This is less prone to error.</p>
<p>Next, if the gray scale verification passes (usually lasts a week). The next
step is to switch routes. Switching routes means switching the routes from the
service node on the cloud directly to the on-cloud node, after which the other
on-cloud master service will directly access the on-cloud node for that service
and will not visit the off-cloud node again.</p>
<p>Special care must be taken while switching routes as a lot of online traffic
will hit the on-cloud node directly. Before the operation first of all, you need
to calculate the number of nodes needed under each geographic region on the
cloud based on usual data, generally in accordance with the peak traffic level,
if not enough to expand the capacity, so as not to cause a large number of
timeouts on the line due to insufficient capacity.</p>
<p>At this time, the number of nodes on the cloud would rather have more, because
more can be slowly scaled back later, the cost is not high. However, if there is
a large number of timeouts caused by less nodes, node expansion takes time
(mainly resource scheduling and service startup time consuming), this time is
likely to cause user complaints. If the timeout triggers a large number of
client retries, it may bring down the whole service and cause online accidents.</p>
<p>At that time, I picked a time when the traffic was low, so as to minimize the
impact caused by switching jitter. After switching the route, the nodes under
the cloud can gradually switch to forwarding nodes, forwarding all the traffic
under the cloud to the cloud for processing. Then the person in charge of the
primary service is notified to try to bring the service to the cloud. Because,
forwarding nodes forwarding to the cloud has overhead and will increase the
latency.</p>
]]></content:encoded></item></channel></rss>