<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Java on Tech Philosopher</title><link>https://blog.bktus.com/en/tags/java/</link><description>Recent content in Java on Tech Philosopher</description><generator>Hugo</generator><language>en-US</language><lastBuildDate>Wed, 15 Nov 2023 20:25:05 +0800</lastBuildDate><atom:link href="https://blog.bktus.com/en/tags/java/index.xml" rel="self" type="application/rss+xml"/><item><title>Weekly Technical Report for the Fourth Week of February 2023</title><link>https://blog.bktus.com/en/archives/oebjdm/</link><pubDate>Wed, 15 Nov 2023 20:25:05 +0800</pubDate><guid>https://blog.bktus.com/en/archives/oebjdm/</guid><description>&lt;p>This week, I encountered a performance issue that led me to re-evaluate the
quality of some service frameworks, particularly certain Java-based ones.&lt;/p>
&lt;p>Under moderate load—around 35% to 40% CPU utilization—the services began to
experience a large number of timeouts. This was unexpected, as neither CPU nor
memory resources were saturated, and the number of worker threads was
sufficient.&lt;/p>
&lt;p>Using Java profiling tools, I found that most worker threads were in &lt;code>Idle&lt;/code> or&lt;code>Waiting&lt;/code> states. There were no obvious hotspots in business logic, and no
threads appeared to be fully utilized. This suggested that the bottleneck was
not CPU-bound, but likely related to I/O or some form of blocking or
coordination overhead.&lt;/p></description><content:encoded><![CDATA[<p>This week, I encountered a performance issue that led me to re-evaluate the
quality of some service frameworks, particularly certain Java-based ones.</p>
<p>Under moderate load—around 35% to 40% CPU utilization—the services began to
experience a large number of timeouts. This was unexpected, as neither CPU nor
memory resources were saturated, and the number of worker threads was
sufficient.</p>
<p>Using Java profiling tools, I found that most worker threads were in <code>Idle</code> or<code>Waiting</code> states. There were no obvious hotspots in business logic, and no
threads appeared to be fully utilized. This suggested that the bottleneck was
not CPU-bound, but likely related to I/O or some form of blocking or
coordination overhead.</p>
<p>Despite using NIO and the Netty framework, overall throughput remained low.
Based on thread analysis, the system appeared to spend a significant amount of
time waiting rather than executing useful work. The exact root cause was still
unclear, but the symptoms strongly indicated inefficiencies in how the framework
handled concurrency or I/O.</p>
<p>During this period, I attempted to reduce infrastructure costs by scaling down
the number of nodes. However, my initial approach was overly simplistic—I relied
mainly on CPU utilization as the indicator of system capacity.</p>
<p>After reducing the number of nodes and increasing average CPU utilization in the
Beijing region to around 35%–40%, the entire service began to experience
widespread timeouts. Monitoring showed that almost all nodes entered a degraded
state simultaneously.</p>
<p>This incident highlighted an important lesson: CPU utilization alone is not a
reliable indicator of system capacity, especially for I/O-bound systems. A
system can appear underutilized while still being unable to handle additional
load due to hidden bottlenecks.</p>
<p>In parallel, I continued refactoring an old PHP-based gateway service, replacing
it with a new implementation in Java. However, I have reservations about using
Java for high-concurrency gateway scenarios.</p>
<p>The execution model of Java(especially with JDK 8) relies heavily on OS threads.
Without lightweight threads (such as virtual threads), the number of threads per
container is limited (e.g., around 800 threads for a 4-core, 8GB container).
High thread counts introduce significant context-switching overhead, which
limits per-instance throughput and increases the number of containers required
to handle the same traffic.</p>
<p>Initially, I considered implementing the gateway in Go, which provides better
support for high-concurrency workloads and has a mature ecosystem. However, due
to team and organizational constraints, I proceeded with the internally
developed Java framework.</p>
<p>This experience reinforced the importance of aligning technology choices with
workload characteristics, as well as the need to evaluate system capacity beyond
simple resource metrics.</p>
]]></content:encoded></item><item><title>Technical Review for the Third Week of December 2022</title><link>https://blog.bktus.com/en/archives/3nav5g/</link><pubDate>Thu, 19 Jan 2023 17:40:11 +0800</pubDate><guid>https://blog.bktus.com/en/archives/3nav5g/</guid><description>&lt;p>This week, my main focus was optimizing a Java service that had long been
experiencing abnormal CPU behavior.&lt;/p>
&lt;p>At first glance, the issue appeared to be that CPU usage could not scale up
under load. A natural starting point was to check whether the service was
constrained by an insufficient number of worker threads. However, further
observation revealed a different pattern: CPU usage could increase, but once it
did, the system quickly ran into severe timeout issues.&lt;/p></description><content:encoded><![CDATA[<p>This week, my main focus was optimizing a Java service that had long been
experiencing abnormal CPU behavior.</p>
<p>At first glance, the issue appeared to be that CPU usage could not scale up
under load. A natural starting point was to check whether the service was
constrained by an insufficient number of worker threads. However, further
observation revealed a different pattern: CPU usage could increase, but once it
did, the system quickly ran into severe timeout issues.</p>
<p>This suggested that the problem was not about CPU utilization itself, but rather
about what happened when the system attempted to utilize more CPU.</p>
<p>Historically, this service had already been flagged for poor performance, and
there were even suggestions to deprecate it. Based on this background, I
suspected that the root cause might lie in the framework rather than the
business logic.</p>
<p>I began by reviewing the framework implementation. The service uses Netty as its
NIO server framework. Incoming requests are handled by I/O threads and then
dispatched to worker threads for business processing.</p>
<p>In a previous issue, insufficient worker threads had caused performance
bottlenecks under high I/O load. However, in this case, logs indicated that the
number of worker threads was sufficient. This led me to consider whether the
issue might originate from the client side.</p>
<p>Within the microservice architecture, this service also acts as a client when
invoking other services. This became a key investigation direction.</p>
<p>By analyzing the code, I found that the framework uses a proxy class
(<code>ObjectProxy</code>) based on Java’s <code>InvocationHandler</code> to intercept RPC calls. When
a service method is invoked, <code>ObjectProxy</code> interacts with a <code>ProtocolInvoker</code> to
retrieve a list of available downstream nodes from the service registry
(refreshed every 30 seconds). The list is then passed to a <code>LoadBalancer</code> to
select a target node.</p>
<p>The actual invocation is performed by a protocol-specific <code>Invoker</code>, which
maintains long-lived connections to downstream services. Depending on whether
the call is synchronous or asynchronous, the request is sent and handled
differently.</p>
<p>Each downstream service consists of multiple nodes. For each node, the framework
creates:</p>
<ul>
<li>Two I/O threads (for NIO event handling)</li>
<li>Multiple TCP connections (by default equal to the number of CPU cores)</li>
</ul>
<p>Each I/O thread manages a <code>Selector</code>, which polls for events on its associated
connections. For each TCP connection, a <code>TCPSession</code> is maintained.</p>
<p>When a request is sent, a <code>Ticket</code> object is created to track the
request-response lifecycle:</p>
<ul>
<li>For synchronous calls, the thread blocks until a response is received</li>
<li>For asynchronous calls, the <code>Ticket</code> stores a callback, which is triggered
upon response arrival</li>
</ul>
<p>At the network level, the framework relies on Java NIO (<code>SocketChannel</code>).
Incoming data is buffered, and packet boundaries are determined using a
length-prefixed protocol. The framework checks whether the buffer contains
enough bytes to form a complete packet before processing it.</p>
<p>Once a full packet is received, it is dispatched for further processing. At this
stage, a worker thread from a thread pool is used to handle the <code>Ticket</code>. This
thread pool is relatively small, with:</p>
<ul>
<li>Core size = number of CPU cores</li>
<li>Maximum size = 2 × number of CPU cores</li>
</ul>
<p>From a design perspective, this architecture is fairly standard and widely used.
After thoroughly reviewing the client-side implementation, I did not find any
obvious flaws.</p>
<p>At this point, it became increasingly likely that the issue was not on the
client side, but rather on the server side or in the interaction between
services. Further investigation would need to focus on the downstream services
and their behavior under load.</p>
]]></content:encoded></item><item><title>Weekly Technical Report for the Fourth Week of November 2022</title><link>https://blog.bktus.com/en/archives/99lv21/</link><pubDate>Mon, 28 Nov 2022 00:47:29 +0800</pubDate><guid>https://blog.bktus.com/en/archives/99lv21/</guid><description>&lt;p>This week, my main task was optimizing a Java service that exhibited abnormal
timeout behavior.&lt;/p>
&lt;p>Even under low production traffic, the service experienced batch timeouts. At
the same time, CPU usage remained consistently low and never increased under
load. This strongly suggested that the system was not CPU-bound, but rather
blocked on some operations.&lt;/p>
&lt;p>Most services I have worked on, including this one, are IO-intensive. They rely
heavily on RPC calls, where worker threads often block while waiting for
responses. To mitigate this, such systems typically use a large number of worker
threads to ensure that new requests can still be processed even when many
threads are blocked.&lt;/p></description><content:encoded><![CDATA[<p>This week, my main task was optimizing a Java service that exhibited abnormal
timeout behavior.</p>
<p>Even under low production traffic, the service experienced batch timeouts. At
the same time, CPU usage remained consistently low and never increased under
load. This strongly suggested that the system was not CPU-bound, but rather
blocked on some operations.</p>
<p>Most services I have worked on, including this one, are IO-intensive. They rely
heavily on RPC calls, where worker threads often block while waiting for
responses. To mitigate this, such systems typically use a large number of worker
threads to ensure that new requests can still be processed even when many
threads are blocked.</p>
<p>However, this approach has inherent limitations. Although Java uses NIO for
network I/O, the actual business logic is still executed by worker threads. Once
a worker thread is blocked on an RPC call, it cannot proceed until the call
completes or times out. This makes the system highly sensitive to downstream
latency.</p>
<p>After investigation, I discovered that the issue was caused by a framework
configuration problem. The configured number of worker threads was not being
properly loaded, causing the service to fall back to its default setting—where
the number of worker threads equals the number of CPU cores.</p>
<p>As a result, when worker threads were occupied with downstream RPC calls, even
small fluctuations in latency would quickly exhaust available threads, leading
to a large number of timeouts. Meanwhile, CPU usage remained low because most
threads were blocked waiting on I/O.</p>
<p>To troubleshoot the issue, I first enabled thread name logging by adding <code>%t</code> to
the logback pattern. Then, I performed load testing on a single node to observe
request handling under controlled QPS. I also added <code>StopWatch</code> instrumentation
to measure processing time within the request flow.</p>
<p>From the logs, I noticed that only a small number of threads were handling
requests, which was clearly abnormal. Using <code>jstack</code>, I confirmed that only four
threads were actively processing the main request logic.</p>
<p><img loading="lazy" src="https://img.bktus.com/DGCRxfEA.webp"></p>
<p>Further analysis showed that the service relies on Netty, and request handling
ultimately occurs within the Netty worker thread pool (e.g., threads with prefix<code>nioEventLoopGroup-*</code>). CPU time distribution also confirmed that only these few
threads were actively processing requests.</p>
<p>In production, services of this type typically configure hundreds of worker
threads (e.g., 800–1000), along with strict RPC timeouts to prevent excessive
blocking. After fixing the framework configuration, the issue was resolved.</p>
<p>In addition, I identified another risk in one of the service’s core interfaces.
It made downstream HTTP calls using OkHttp, but no timeout was configured. In
extreme cases, this could cause worker threads to block indefinitely if the
downstream service was slow to respond.</p>
<p>To address this, I configured proper timeouts for all HTTP calls:</p>
<ul>
<li>connection timeout</li>
<li>read timeout</li>
<li>write timeout</li>
</ul>
<p>This is essential to prevent cascading failures and maintain system stability.</p>
<p>I also enabled OkHttp’s <code>ConnectionPool</code> to improve performance under high
concurrency.</p>
<p><img loading="lazy" src="https://img.bktus.com/tQJn9FVd.webp"></p>
<p>According to the documentation, <code>ConnectionPool</code> allows multiple HTTP/HTTP2
requests to reuse the same connection, reducing overhead. However, this requires
the server side to support persistent connections.</p>
<p>Besides this optimization work, I also spent time preparing for Tencent Cloud’s
Advanced Architect TCP exam. Compared to previous certifications, this exam was
significantly more challenging and required deeper understanding.</p>
<p>Due to limited time during workdays, I concentrated my review late into the
night before the exam—studying until 5 a.m. on Saturday. Fortunately, the effort
paid off, and I passed the exam. I plan to write a separate article summarizing
that experience.</p>
]]></content:encoded></item><item><title>Technical Review for the Third Week of November 2022</title><link>https://blog.bktus.com/en/archives/t6yqs7/</link><pubDate>Mon, 21 Nov 2022 00:07:35 +0800</pubDate><guid>https://blog.bktus.com/en/archives/t6yqs7/</guid><description>&lt;p>This is the first of my technical reviews, mainly intended as a reflection on my
experiences so far and as a starting point for future summaries.&lt;/p>
&lt;p>I have been working at the company for nearly six months, and recently
transitioned from client-side development to backend development. This shift
aligns with my long-term thinking, as backend work tends to expose more complex
system-level problems and offers broader technical exploration.&lt;/p>
&lt;p>That said, I do not believe there is an inherent hierarchy between client-side
and backend development. My earlier open-source project, GpgFrontend, is a
client-side application where I dealt with complex issues such as cross-platform
compilation, compatibility, and stability. These problems are by no means
simpler. The perceived difference often comes from mindset rather than the
nature of the work itself.&lt;/p></description><content:encoded><![CDATA[<p>This is the first of my technical reviews, mainly intended as a reflection on my
experiences so far and as a starting point for future summaries.</p>
<p>I have been working at the company for nearly six months, and recently
transitioned from client-side development to backend development. This shift
aligns with my long-term thinking, as backend work tends to expose more complex
system-level problems and offers broader technical exploration.</p>
<p>That said, I do not believe there is an inherent hierarchy between client-side
and backend development. My earlier open-source project, GpgFrontend, is a
client-side application where I dealt with complex issues such as cross-platform
compilation, compatibility, and stability. These problems are by no means
simpler. The perceived difference often comes from mindset rather than the
nature of the work itself.</p>
<p>From my perspective, the core ideas behind different technical domains are
fundamentally similar. However, industry roles tend to divide them too rigidly.
Even though I am currently working as a backend developer, I do not want to
limit myself to that identity or stop exploring other areas.</p>
<p>Looking back, my earliest backend project was an astronomy forum I built at the
age of 15 using Node.js and Express. At that time, I was first exposed to
concepts such as MongoDB, session management, and asynchronous programming.</p>
<p>Asynchronous callbacks were particularly difficult for me to understand. At that
time, I did not even have a clear understanding of processes or threads, which
made it challenging to grasp concepts like non-blocking execution and closures.
Gradually, I came to understand that asynchronous programming allows the main
execution flow to continue while deferring blocking operations, and later
handling results through callbacks.</p>
<p>Later, I worked with Python to build a simple academic management system, where
I encountered the MVC pattern for the first time. This introduced me to
structured system design, where controllers handle requests, models manage data,
and views render output.</p>
<p>At that stage, server-side rendering was still dominant. The backend was
responsible for everything, including page generation, data processing, and even
aspects of UI design. This experience gave me a more holistic understanding of
system development.</p>
<p>In the following years, I began using Spring Boot and gained a deeper
understanding of relational databases. I also encountered the concept of
frontend-backend separation, which I found very compelling. Offloading rendering
to the client while keeping the server focused on data processing improves
decoupling and scalability.</p>
<p>At one point, I was very enthusiastic about RESTful API design, even believing
that strict adherence to it could eliminate the need for documentation. In
practice, however, I learned that real-world systems are more complex, and
maintaining strict REST conventions is not always feasible.</p>
<p>After joining the company, I found that the primary technology stack is Java and
Spring Boot. Although I personally find Java somewhat verbose, its ecosystem is
mature, well-documented, and highly suitable for production systems. It provides
stability and a wide range of readily available components.</p>
<p>Over the past months, my work has largely focused on system optimization,
including caching strategies and thread management. I frequently analyze alerts,
some related to business logic and others to technical issues. Business problems
require understanding context, while technical problems require deeper knowledge
and continuous learning.</p>
<p>There are still areas where my understanding is not sufficient. For example,
during containerized deployments, I have observed occasional timeout issues that
are difficult to attribute definitively to either the container environment or
the application itself. I believe resolving such problems requires deeper
knowledge of system-level topics such as CPU scheduling, memory management, and
network virtualization.</p>
<p>This marks the starting point of my technical reviews, and I expect future
entries to become more focused and structured as my experience grows.</p>
]]></content:encoded></item></channel></rss>