My last weekly technical report, the Weekly Technical Report for the Second Week of April 2023, was written after the fact in November 2025, and after that I stopped for a long time. The reason is actually simple. One is that typing is a pain. When I write, I change a bit here and a bit there, and after enough changes the sentence just breaks. The sentence in my head is one thing, the sentence I end up writing is another, and sometimes words or prepositions just go missing. The other is that I used AI to help with two posts before, but it didn’t work well. You could tell at a glance they were written by AI. They didn’t read like something I wrote, or like something I would say.

These weekly technical reports are actually the reports I really sent to my manager every week back then. But I don’t put every item into the posts. One reason is confidentiality: even though I’ve left the company, I don’t think it’s a good idea to name the detailed services and things like that. So from the very first post, I’ve been reading the report for that week, thinking about which things were more important, and then picking those things and my thoughts at the time to write a post.

Now I’ve finally found a better way: I read the reports and dictate as I go, say whatever comes up, try to recall how I felt that week, and then organize it into a post. How it gets organized is explained in the AI Policy. This way, the reports are no longer one post per week, but several weeks in one post. The posts are a bit longer, but I can actually record a lot more.

If you’re interested in what backend development really looks like on the front line, this series is about what actually happened each week back then: an account system with a lot of traffic, how compliance requirements get implemented, how a license model gets designed, how an API gets hardened, and what went wrong when I scaled down a Redis. Besides how things were done at the time, I’ll also talk about how I see them now, looking back.

Third Week of April

Compliance Requirements: Soft Deletion Only

This week was mostly all kinds of small things, and most of them were compliance requirements. I think new laws and regulations had just been introduced in China at the time, and they required two things: one was that users can delete their accounts, and the other was that the user’s IP address has to be recorded permanently at registration. Neither of these is complicated in itself, but account deletion is a bit more complicated. I had to look at the database model at the time, the tech stack we used, how account data was stored and where it was scattered, and then figure out how to do the deletion, and keep the changes to the existing system small.

In the end we did a kind of soft deletion. The account isn’t really deleted. It’s soft-deleted in the database, and the data scattered in other places is no longer referenced by any microservice, so we consider the account deleted, dead. Why didn’t we just delete it? Mainly because it was a lot of trouble. The storage structure inside the company at the time was mainly a cache in front of a database, but the problem wasn’t really the database, it was the business logic. There was a lot of very old business logic across dozens of microservices. One place uses the account ID, another uses some fields related to the ID. If we deleted the data and some place we didn’t know about still referenced it, it would find nothing in the database and throw an error. That could easily make the microservice unstable, or make it exit, and affect other users. The account service is a very core service with quite a lot of user traffic, so it can’t go down. If it’s down for more than 15 seconds, the downstream customers, who are monitoring us too, will see it, and the complaints come right away. So soft deletion was the only option.

Recording IP Addresses: An Audit Service Built on Assumptions

Recording IP addresses should have been simple too. We maintained a fairly central Elasticsearch (ES) instance that had every registration record from the beginning until now, so we just needed to add a field there. But before this, I did something that I now think was a bit naive. We had just started doing microservices, and I felt that auditing accounts was hard: when an account registered, when it logged in, when it was deleted, all these records were spread across different microservices. So I proposed writing an account audit microservice. The microservices that use accounts would report events asynchronously to this audit service, and it would organize them and then write them to ES, instead of every service writing a bit here and a bit there.

The idea was very ideal, but now I feel it wasn’t really necessary. A lot of the time, after you build something like this, you find it’s not that useful: unified async reporting, some merging and analysis, then writing to ES. It sounds good, but in practice you rarely use what you’ve analyzed. The main business doesn’t dig data out of places like this. What it’s mainly good for is debugging and tracking down production issues, and that doesn’t need so much processing. So with how I think now, I wouldn’t do this. But since we already had the audit service, when it came to recording the registration IP, I just put it there: at registration, the IP goes into the tracking events, gets reported asynchronously to the audit service together with what was already being reported, and the audit service writes it to ES. It takes a detour, and in the end it lands in ES just like before. I don’t know how things are there now, and I suspect this audit service will most likely be removed by whoever took over. Or maybe the person who took over is more cautious and won’t touch it, to avoid instability from context nobody really understands. Then they’ll have to take the blame for it.

Standardizing Error Codes

Another thing was standardizing error codes. The microservices left by the previous team needed to bind accounts with downstream customers in the cloud, communicating cloud-to-cloud, that is, B2B. We actually had specs and documents for this, but the previous team didn’t seem to care much. They just wanted the integration with customers to work, and once it was integrated and something went wrong, developers had to dig through the code by hand to find the problem. To save manpower, our team decided to collect these error codes and standardize them: which error code is returned when, for what reason, and how to troubleshoot it, all sorted out and written into documentation. Once that’s written, we can send it to the “classmates” in front who deal with customers (calling colleagues “classmates” was the trendy way to say it there), so they can tell customers directly what the cause is and stop bothering us developers. I think I was the one who collected these error codes; the weekly report doesn’t mention anyone else on the team doing this step. After that I aligned with the product side, reached agreement, and wrote it in our Wiki. The formal documentation after that wasn’t written by me. Other people on the team turned my Wiki into standardized documentation, and then it was sent to customers through the product process.

A New Account Model: From Per-Head Billing to Licenses

The main thing this week was a new account model. There was a new customer with fairly high requirements, and they needed us to adjust and strengthen the existing account process. The old account linking process went roughly like this: the customer’s end users use the customer’s product, the product uses our service when it gets activated and registers an account with us, then we notify the customer cloud-to-cloud about which of their end users used our service, and charge based on that. But the old reconciliation, I can only say, was rather loose and not very rigorous. We exported a report on our side, the customer exported a report on their side based on their end users and the cloud-to-cloud notifications, and then the two sides reconciled them. The billing model also charged per head: if you used it, it counted. It didn’t consider whether the user was still using it, whether they had a license, how long the license was, or whether, after the customer’s contract expired, the user could still get access to our services through our account system.

So we needed a stricter model that took contracts into account, and also which product the customer’s end users were using. For example, a user can only use it for two years, and after two years they have to pay to keep using our service, and the customer collects that money from their end users. Every time an end user activates the product and uses our account, we must tell the customer which license was issued, for how long, and how long it’s valid. All of this needed to be standardized. My manager pushed this model. The idea was that since this customer had high requirements and needed this kind of model, we’d make it a standard thing and use the same one with new customers later.

The model became clear gradually during negotiations with the customer. My manager told me what the customer wanted, what we should do, what the constraints were, what we had to do and what we couldn’t do. Then I built the model from that, designed the abstractions, how licenses should work and how contracts should work, and wrote it into a document. It went to the customer, and to my manager and my manager’s manager for review. Once everyone agreed, I put it into practice: I went through the existing microservice architecture to see where the logic needed adjusting, and started a new license service. On top of the existing accounts, this license service handled the new model, including pricing, issuing, activating and managing licenses and contracts, and whitelist management for some test licenses, all that messy stuff.

Licenses Are Immutable, and Permits Decide Whether They Can Be Issued

The model is worth talking about a little. First, a license is an object, or rather a domain object. I gave it one constraint: once a license is created, it can’t be changed. If it needs to change, the only way is to revoke the current license and issue a new one. Why do it this way? Mainly because of auditing. If some action by the operations team, or some bug in the code, adjusted a license without us knowing, a lot of the time there would be no way to notice it unless someone looked into it carefully. Revoking and issuing a new license are at a different granularity, and both are recorded in the ES I mentioned earlier, so you can look them up at any time. That’s why the system has no API at all that can change a license’s duration, subject, validity period or effect. It can only be revoked.

Above the license there’s another layer, the customer’s license, which is tied to the length of the contract. I’ll call it a “permit” for now (the documents at the time probably didn’t call it that, and I can’t remember the real name anymore). When issuing a license, we first need to find the customer’s permit. Then, based on whether the permit has expired, which product of which customer it belongs to, and how long a license it allows, we decide which licenses can be issued. If no permit is found, that means the customer hasn’t signed any contract for this product, or hasn’t renewed, so issuing new licenses is refused. In other words, the permit is the ability to issue licenses.

For contracts, I abstracted it like this: during the contract, we provide services to the customer and the customer’s end users. After the contract expires, if an existing end user’s license hasn’t expired yet, we must let them keep using it, and only stop them once their license expires. The contract expiring only means we no longer accept new users, and users whose licenses have expired can’t activate their service again, can’t activate our account again.

With these two layers of abstraction, it turned out we could flexibly express all of the customer’s requirements, and ours too. When a contract is about to expire, it also makes negotiation easier for us. Even if the contract has expired and the renewal has been agreed but the process isn’t finished yet, we can first give the customer a temporary permit, so issuing doesn’t stop, and users whose licenses expired can keep paying and keep using our service. I felt pretty good about it. All of this was implemented later, including reminders when a license is about to expire: we notify the customer, and the customer reminds their end users. The structure of these messages, which fields and what content they have, and terms like the permit, were also designed by me and written in the documentation.

After Launch

The service went live before I left, and it was officially activated. What happened after that, I don’t know: how the service is doing now, whether it has bugs. It definitely has bugs. I’ve been thinking about this all along: if I scanned the code I wrote back then with AI now, it would definitely find a lot of bugs. But it ran at the time and nothing went wrong, and that’s good. As for problems that only show up in some extreme cases, as long as they aren’t big and don’t cause incidents, that’s just how it is. Having bugs, that’s human slop, and there’s no way to avoid it. A person writing code can’t think of everything the way AI does.

Besides that, there were all kinds of governance-related services this week that I followed up on a lot, so I won’t go through them one by one.

Fourth Week of April

Hardening the API: Why Not Public-Key Signatures

The more important thing this week was upgrading the API that clients use to access the account service. This isn’t the B2B communication I mentioned earlier, but the API our own client uses to access accounts. The old API was fairly weak, its authentication was weak too, and it didn’t do much validation. A lot of the time we worried that fake clients were bypassing our usual authentication and using our service illegally. I was the main person responsible for accounts at the time, so the more important items naturally went to me. I upgraded this API and moved the clients over to the new one. The new API does a lot more: it requires a signature, many fields are mandatory, and they’re validated. The signature depends on a secret that only the client knows and no one else does, and we sign and verify based on that secret.

But we didn’t use public-key signatures. We used a signature based on a hash algorithm. The main reason was that our clients were all over the place, running on different end products, on old platforms and new ones, and old platforms don’t necessarily support signature verification or anything PKI-related. Also, it’s complicated. The thinking at the time was: if something is complex, isn’t really necessary, doesn’t bring in any revenue, and leaving it out won’t cause big losses, then don’t do it. Doing it is useless, and it would cause all sorts of problems in all those messy real environments. There are too many variables: this one says the CPU can’t handle it, that one says the library is missing, this library fails as soon as you call it. Once the client is released, you can’t change it anymore, and it just doesn’t work. End users don’t care whether it’s secure or not. When they see it doesn’t work, the complaints come right away, and we can’t update their clients directly. The losses would be big, and it’s bad for both customers and end users.

Looking Back Now

Looking back, I was still pretty inexperienced then. I’m now doing a master’s degree at a university and have picked up more knowledge, especially after taking courses in cryptography and network security. If I did this again now, I’d think more about some of the design. For how to design a protocol like this, there are actually many existing best practices, and you can just follow them. How security protocols have changed from the 1970s to now, what new and better patterns came out of each lesson: with that theory, my design could probably be a bit better.

But actually, that doesn’t help much. In real business, launching as soon as possible, keeping things stable, and making sure users are never stuck unable to use it, that’s the most fundamental thing. Hardening the API is a side benefit. Done well, it just means a bit less freeloading, and it doesn’t show much cost reduction, unless the company asks for it or there’s a fairly serious security hole.

Fields Can Only Be Added, Never Removed

Another thing was protocol-related: adding a field between the client and the API. What I can mainly say about this is: fields can only be added, never removed. Once you remove a field, you don’t know what the old client logic will do when a field is missing. In theory most clients should handle this, but who knows if they do. A client might just throw an error, or just exit, both are possible. Adding fields also needs care: you need to understand things clearly and test, and old clients can’t use the new field anyway. So everything should be based on the end user’s experience with the product, not on showing off technology. The main thing is to think about all kinds of compatibility: your old services and old clients are still in use, they can’t be updated, so what should they do?

There was also monitoring of API abuse. I was asked to design a plan this week. It’s related to the API hardening too, and the main worry was that our current API was being abused by gray-market operators.

Fifth Week of April

This week was still about the same things as before, mostly with some new items added. In the weekly report, a lot of items had a progress percentage every week. For example, the API abuse item: last week it was still under investigation, and this week I had an approach and was basically doing the development and writing the documentation.

From a Single Availability Zone to Multiple

The other thing was service governance. The account business was deployed in the cloud at the time, and there was already a related item last week: the storage layer was deployed in a single availability zone, for example in Guangzhou, or in Shenzhen. Once something like a cut network cable happens in that availability zone, there’s a problem. We actually had a cable cut once, the whole availability zone was degraded, and our service latency went up. When latency goes up, clients time out, and customer complaints come straight to us. We can’t just say the cloud provider had a problem. We can pass on part of the blame, but the blame still ends up with us. So a single-zone deployment is risky, and it needed to move to a multi-zone deployment, carefully. At the time I was working on a plan: sorting out which services were in which zones, which were single-zone, which were half single-zone and half multi-zone, and what state their databases were in, whether it was the cloud’s distributed database or Redis, and how the Redis was set up.

Second Week of May

The first week of May was the May Day holiday, a whole week off, so next is the second week. This week was mainly development for requirements, that is, the items from the previous week, nothing new.

Redis Replay Protection: The Lesson of a 20% Cut

What was new was a problem found around May Day. Our gateway service uses Redis for replay protection; I can’t share the exact algorithm. Because the traffic is huge, this Redis had a fairly large capacity. After May Day, to cut costs, I reduced it by 20%, thinking it normally never reached 80%, so it should be fine. But that’s a story for later. At some point afterward, of all things, it actually shot up to 100% again. Then Redis had to evict keys, latency spiked, the gateway fluctuated badly, and alerts went off. I regretted it right then: why did I cut that 20%? Because you never know when it will reach the line you thought it would never cross.

As for why this Redis had to be so big, it’s because we couldn’t change the protocol. Lots of old clients had to use this algorithm, and the algorithm takes up a lot of Redis space, so the only option was a huge Redis to make sure replay protection worked. The earliest generation of the service didn’t consider replay protection in its design, and there was nothing for it in the fields. The products and clients were already out there and couldn’t be updated, and we couldn’t add these fields either, so we just had to live with the cost.

Not a Technical Problem, but an Organizational One

This is a dilemma too. When you’re just starting out, it’s hard to foresee this much traffic later. Whether the business would work out, you didn’t know either. You launch first and worry later, race for time, keep up the pace. Later you look back and see that these things really weren’t considered. Actually, thinking about it now, it should still be possible to migrate to a new algorithm step by step. But that’s slow and hard to push, involves multiple teams and multiple clients, and takes a lot of energy. A lot of the time it’s not a technical problem, it’s an organizational one: do you have enough clout to bring this up for discussion, and does it really need this much manpower? Otherwise, doubling the Redis might even be cheaper than getting all these people together to discuss, implement and test.

Timeouts in Two Java Services

Also, there were the two Java services I mentioned before. Their framework was probably not well written, and they kept having all kinds of timeout problems. Back then I was still an undergraduate and hadn’t started my master’s. I used a lot of monitoring methods I could find online to see whether it was the JVM or the framework itself, but I couldn’t see anything, probably because I wasn’t experienced enough yet. Up until May 2023, these two services still had problems. I found some clues at the time, did some optimization and some work related to thread bloat, but the framework was very hard to change. It was mainly a heavily modified Spring Boot, modified a bit here and a bit there, who knows where, and then the people who modified it left, leaving behind this mess, and I could only analyze it piece by piece.

I was still pretty clueless at the time. Looking at all the things I did, it might seem like my skills were great, but actually I was still lacking in theory. Performance analysis, the whole computer architecture, how the CPU, the three levels of cache and memory really work, and assembly too: I had no real understanding of them, and it was all still a fog. That fog only started to clear when I was doing my master’s, which is now. There will probably be some items in later weekly reports that show the actual analysis, so I won’t go into it here.

Finally

That’s all for these few weeks. I’ll talk about the third week of May and later slowly, saying whatever comes to mind, so my train of thought doesn’t break and the posts don’t get too long. Dictating like this feels much better than typing it myself, and I can say things I couldn’t get out by typing: ideas I forget while typing, details from back then, or things I didn’t want to type because it was too much trouble and then slowly forgot. My memory is still pretty good now, and I still remember these things. So I’ll slowly catch up on these weekly reports, at least up to when I left in 2024, so I won’t forget them later. For my future self, it’s also a valuable asset.