Updates have always been one of GpgFrontend’s biggest pain points. From the very first versions, GpgFrontend relied on GitHub Release to publish versions and to check for updates, and it stayed that way for many years. Starting with 2.2.5, and including the current main line, the version numbering rules, the release process and the way the client checks for updates have all moved to a new model and mechanism. How to use it and how to verify things is already in the user manual. But why I did it this way, and what I was thinking at the time, doesn’t come across in the documentation.

For security software like GpgFrontend, transparency is a basic requirement. And updates are directly about what actually ends up on the user’s computer: where the client connects in the background and what it sends. Users have every reason to be suspicious about that. So I think it’s necessary to explain it clearly, so that users know what the update side really looks like. If you want to know how GpgFrontend got here, you can read my previous post.

Many Years of GitHub Release

In the beginning, GitHub Release was actually pretty good, at least until a couple of years ago. One thing is that it gives you free distribution. Another is that its model is simple enough: a release is tied to a Git tag. When I released a version, I just created a tag, wrote a Release Note, and uploaded all the installers built by GitHub Actions in one go, and that was it.

Later, the project started moving faster, especially after I brought in AI coding assistance, and now agents too, and a lot of features were added. You don’t always notice these features day to day, but when you really use them, you still find some problems. On the old tick-tock rhythm, I released a version every two months. A lot of the time, a problem I missed before a release would turn up one, two or three days after it, found by me or by someone else. At that point it was hard for me to release again right away, because the rhythm was one version every two months, and releasing in between was a hassle.

But on the other hand, releasing every day isn’t good either. Most users care about stability and don’t want to try new things. They might check for updates once every two months, or even less often. Anyway, most users don’t need most of the new features. What they use is encryption, decryption, key generation and so on. Take email, for example. In the past it could basically only work with EML files, and I’ll admit myself that it was pretty hard to use. Now the email module has been reworked quite a bit and can cover most everyday email use, but the people using it are still a minority.

So for most users, waiting two months probably doesn’t feel like anything, because I check things before every release and the basic features don’t break. But users who really do hit a problem have to wait two months to get a fixed version, and that’s pretty uncomfortable. I can feel this myself. Still, when I find a problem I can just fix it, build it myself and use it, so anyway it doesn’t affect me. But imagine you don’t have the means to build it yourself, or you’re not a technical user. Then you have to wait two months.

That’s why I later introduced Nightly Builds, which follow my development line. They used to follow the develop branch, and now they follow the main branch. As soon as I commit and push to GitHub, the pipeline starts a build automatically. Nightly and the stable release are of course different. Whatever you say about it, the stable release is a set of features that have been verified, while the main line brings in a lot of new features. Most of the time these new features don’t conflict with the stable ones, but sometimes I make architectural changes or optimizations, and at that point I can’t yet verify that they won’t affect stability. So Nightly uses a different profile from the stable release: it runs as a separate app, GpgFrontend Testing, with its own keys and settings.

As for how it was published, Nightly had only one nightly tag. When the pipeline ran, it pulled the latest commit on the branch and built it, then force-overwrote the existing nightly tag, used an action plugin to create the release, and filled in the Release Note template for the Nightly Build automatically. The stable release kept the usual vX.Y.Z tags. Nightly and the stable release lived side by side on GitHub Release like this for at least a year or two, and it felt fine. But Nightly had no numbering like 1, 2, 3, so users could only see the latest one.

The Turning Point: 2.2 and Multiple Release Lines

The real turning point was version 2.2. 2.2 introduced a second engine, so the GnuPG engine and the RPGP engine can be used at the same time. It was a fairly big architectural change. Back then, I needed time to prove that the changes that came with RPGP wouldn’t affect existing users who mainly use GnuPG. So I kept the 2.1 line, the original single-engine release line, and kept updating it as usual, while the 2.2 line also kept moving forward.

At that point, 2.1 users would very likely want to stay on 2.1. For them, I could only push 2.1 updates first, and then recommend upgrading to 2.2 once 2.2 was stable. If you use GitHub Release for this, the only way is to make the client logic thicker. The client calls the GitHub Release API, works out which versions 2.2 has and which versions 2.1 has, gets the metadata of each version, and then calculates locally: does 2.1 have a new version, is 2.2 already stable or still in pre-release, has anything been withdrawn, and so on. Only then does it notify the user. It’s doable, but it always felt pretty awkward to me.

GitHub’s release model can express two release lines, but it’s, how should I put it, missing some flexibility. For example, when you publish a release, you can mark it as latest, but that latest is the latest across all versions. For an older release line, say one kept around for users on older systems, its newest version isn’t the newest version of the whole project.

Another thing is EOL (End of Life). Say I plan to end 2.1 on December 31 of some year. With GitHub Release, I can’t just take all of 2.1 down, because all of it has to be archived and kept on record. So the only way is to hard-code it in the client: after December 31, 2.1 is EOL, and it recommends upgrading to 2.2. But the client is software installed on the user’s computer. Once it’s released, that value is very hard to change. If I want to extend support, or end it earlier, there’s no way to do it unless the client reads the value remotely from somewhere.

Then there’s the rate limit. GitHub’s API limits how often you can call it. Once at my company, I downloaded GpgFrontend on my own computer and wanted to check for updates, and it turned out the whole company’s exit IP had already hit the limit, so it simply didn’t work. With a VPN it was often the same, unless you log in, and the client can’t log in, which is pretty awkward. On top of that, it’s tied too tightly to the platform: you can only distribute through GitHub. If I wanted my website to detect the user’s platform automatically and offer a download link, that link could only point to GitHub. Downloads weren’t very stable either, and with a VPN they could be pretty slow, so it wasn’t a great experience. (As for the many people who don’t want to touch anything GitHub-related since Microsoft bought it, that’s a side note, not the main reason.)

Of course, all of this isn’t completely impossible on GitHub Release. For example, you can write some agreed-upon data into the Release Note, have the client read it, and decide based on that. But one thing is that it’s not a proper standard, and another is that it’s fragile. It’s like a kitchen knife: you can use it for almost anything, but in some special cases using it is just clumsy.

And precisely because the GitHub approach needed so much extra logic and was so easy to get wrong, I simply didn’t add it. I kept the update check plugin very simple: it only talked to GitHub, read some simple metadata and made a decision. The result was that the plugin wasn’t very useful, and it did almost no verification. How would the client know that a newly published version was signed? It’s hard to conveniently get these signatures from GitHub, or metadata like the “root” behind the signatures. Actually, I rarely used this update plugin myself.

The New Version Numbering Rules

Before getting to the new release mechanism, let me talk about the change in version numbers. Under the old rules, a change in the last number didn’t mean the release was just a patch. It meant another step forward in a certain direction. For example, the 2.1 direction was about steadily improving all kinds of features on a single engine. 2.2 was multi-engine, and 2.2.1 and 2.2.2 just kept exploring in the multi-engine direction, which didn’t mean those updates were small. Someone did raise this as an issue, and that’s how I explained it back then. From 1.0 all the way to 2.2.4, that was how I thought about it.

Starting with 2.2.5, to fit the new release model, or rather to make it more sensible, the last number now really is just a patch:

  • The first number is the “Generation”, a change like the one from 1.0 to 2.0. It only changes for a really major architectural change, or when the whole idea of the product changes. It has nothing to do with how many lines of code went in.
  • The second number is the feature release, which is what the “direction” used to be. 2.2 to 2.3, or 2.3 to 2.4, really means big changes, roughly every two months.
  • The third number is the fix release. 2.3.1 to 2.3.2, or 2.3.2 to 2.3.3, goes back to what it originally meant: mainly security updates and fixes. It may bring a few features, but as few as possible, unless it’s really necessary.

The benefit is that stable users can get fixes without needing new features. For example, a user might not want two engines but still want security fixes for the GnuPG engine, or might want to stay on 2.2, which works well for them, rather than move to 2.3. For me, it also makes it fairly easy to control the pace of each release line: I cherry-pick fixes from the main line onto each line, and then push them to the right users through the current update mechanism. Moving from 2.2 to 2.3 is only a recommendation. You can still stay on the 2.2 line and keep receiving patches.

But it’s hard for me alone to maintain a lot of release lines. Usually it goes like this: I want to move from 2.2 to 2.3, but 2.3 brings in a lot of new things and doesn’t settle down that quickly, so 2.2 definitely has to stay around for a while, and at that point there are two lines. Add the fairly important single-engine 2.1 line, and that’s three. Three is the most I’ll maintain. The current state is: 2.3 is the main line, 2.2 is stable, and 2.1 is legacy, which only receives security fixes until December 31, 2027. It’s also the last series that builds with Qt 5. With the new mechanism, I can update 2.1 once in a while, update 2.2 at the usual pace, and keep pushing 2.3 forward. All of this state is written in release/policy.toml in the repository.

Also, the Nightly version numbers changed. Before, I wrote the version number, say 2.3.0, into the CMake project, so every Nightly Build was 2.3.0, no matter which commit it was built from. Users could see the commit in the build information, but it wasn’t convenient to refer to. Now the Nightly version number is derived from git and looks like 2.3.0-alpha.138+g<commit>. The number after alpha is the number of commits since the most recent base tag. This makes it easy for other people to refer to and easy for me to find. I don’t need to dig through long commit messages and match them up in the repository anymore. When I see alpha.250, I roughly know which day’s version it is. Users can also keep the last three or four Nightlies. If a new one doesn’t work well, they can go back to the previous one for a while and update again after that stretch of development is over.

Besides stable releases and Nightly, there are also betas and RCs (Release Candidates). Say I’m about to release 2.4, and the features on the main line feel about done, but it’s not very stable yet, and I want users who are willing to try it to test it first. That’s when there’s an RC like 2.4.0-rc.1. A beta is a test build of the next fix release of a line, with a version number like 2.3.2-alpha.3. Betas and RCs are only offered to users who have ticked “Also offer beta versions” in the settings, and that option is off by default.

The Release Mechanism: Three Buckets and One Update Server

With the background of the version model covered, let me talk about the release mechanism. I don’t really want to use GitHub Release as the main release channel anymore. The biggest pain points are still the ones I mentioned before: one is that it’s pretty slow, and another is the rate limit.

Actually, I’ve always been fairly interested in object storage, and I felt something like AWS S3 should host this. A normal web page would also work, but in comparison, the client only needs a metadata file, say a JSON file, and doesn’t need the JS and CSS that come with a web page. Also, a web server might let people scan its directories, which is a hassle. I don’t want people to be able to grab everything at once. I’d rather keep the metadata separate: the client first gets the metadata at the root, and when it needs something else, that metadata tells it where the next index is, and then it goes and fetches that. A lot of technical things can be done in different ways, but sometimes you just have an intuition about which one is better and which one feels more comfortable to me.

I originally wanted to use S3, but its outbound traffic is billed separately, and that’s hard for me to control. What if this software gets hit by a DDoS attack and a few hundred TB of traffic gets pulled out at once? With the little money that comes from donations, a few hundred TB every day is not something I could pay for. Later I found that Cloudflare’s R2 doesn’t charge for distribution traffic, and I’ve been using Cloudflare’s CDN all along. It did have some incidents a while ago, but for something like software distribution, which doesn’t need high availability, it’s still OK. So I chose R2.

Then the question is what to store in the buckets, and who puts things into them. The principle I set for myself is that the new mechanism must free up my personal time and make releasing faster, especially for stable releases. In the past, to release a stable version, I had to download the installers one by one, sign them with my Authenticode certificate on my Windows computer, then do an OpenPGP signature on all of them, and then upload them one by one. It took two or three hours and was easy to get wrong. Sometimes I even got the file names wrong. I’ve been using scripts since 2.0, and later I had AI write a script for me that downloads automatically from GitHub, asks me to confirm and enter a password for the certificate signature, then a password for the GPG signature, puts everything in one folder, and then slowly uploads it to GitHub Release. That was a lot faster. But I still had to wait for the pipeline to build, with one eye on the pipeline and the other on my own work. And where I live now, access to GitHub Release is pretty slow: the bandwidth isn’t high and the VPN is throttled, which made it pretty painful. Another pain point was that Nightly never had a signature. There was no way I could download every build and sign it with my own GPG key, and putting my GPG private key into the pipeline was even more out of the question.

The current structure looks like this:

  flowchart LR
  CI[GitHub Actions] -->|builds| Staging[(Staging bucket)]
  Staging -->|stable builds| Me[Maintainer machine]
  Me -->|signed| Staging
  Staging --> Server[Release server]
  Repo[GpgFrontend repository] -->|policy, trust, scripts| Server
  Server -->|promote| Builds[(Builds bucket)]
  Server -->|signed index| Update[(Update bucket)]
  Builds --> Client[GpgFrontend]
  Update --> Client

Let me go through them one by one:

  • The pipeline: after a build finishes, it uploads the artifacts and SHA256SUMS.txt to the staging bucket. It also signs SHA256SUMS.txt with Sigstore (cosign), which proves that these files really were built by the build.yml pipeline of the GpgFrontend repository at the matching commit. This kind of signature doesn’t need me to hand any private key to the pipeline, so Nightly finally has a signature.
  • The staging bucket: private and not open to users. It holds things that haven’t been checked yet.
  • My computer: only involved for stable releases. I download the stable build from staging, sign the Windows installers locally with Authenticode using the certificate on my hardware token, sign with my OpenPGP key, and then upload everything back to staging. That certificate lives on a hardware token and can’t be exported, so signing in the cloud is impossible, and it wouldn’t be safe anyway.
  • The update server: it checks what’s in staging, moves it to the builds bucket once the checks pass, and then updates and signs the index.
  • The builds bucket (builds.bktus.com): holds the installers, the descriptor of each version and the Release Notes. Users download from here.
  • The update bucket (update.bktus.com): holds the signed index index.json, plus trust-related files like the root and the index-keys.

Why keep builds and update apart? One reason is not to put everything in one basket, so build artifacts and metadata are stored separately. Another is that it’s easier to manage: one bucket only holds metadata and the other only holds installers, so they don’t get mixed up. Add staging, and three buckets is about right.

The update server sits in the middle and acts as a checkpoint for users. For Nightly, beta and RC, it uses cosign to verify the Sigstore signature and confirms that the build came from build.yml and that both the ref and the commit match. Every file has to match the signed SHA256SUMS.txt, and there can’t be any extra files. It derives the version number itself from git instead of trusting the version number the build claims. For stable releases, it also checks my OpenPGP signatures and the signed tag, as well as the Authenticode signer of the Windows installers. Besides that, it reads the guidance file release/policy.toml in the repository, things like when 2.1 reaches EOL and what state 2.2 and 2.3 are in, and generates metadata that’s easy for the client to read, which is the index.

The update server is important, because it decides what the client can see. But it’s not the most critical link. It doesn’t take requests from clients directly, so it doesn’t need high concurrency or DDoS protection. It just moves things from the unverified place to the verified place and then updates the related metadata. High availability, high concurrency and DDoS protection are left to Cloudflare’s infrastructure. Writing code myself specifically to handle these would be a huge project, and there’s no need for it. So the load sits on the buckets, and management and admission sit on the update server. And the client does its own verification too. It treats the update server with suspicion as well, which I’ll get to later.

The Signing Root: 2-of-3 and minisign

Another question is the signing root. A signing key can’t go without rotation forever. I can’t use the same key for 30 years. So how do I manage it? I didn’t use CA certificates, because that would actually be more complicated. I know X.509 is very flexible and you can add all kinds of fields, but managing it already takes a lot of tools, OpenSSL and so on. It’s too flexible and has too many features. This is just a software update. There’s no need to do it like Apple: set up a CA root, self-sign it, hold some kind of ceremony, use the CA certificate to sign an intermediate certificate, and then use the intermediate certificate to sign other things. It can be done, but there’s no need. Day to day, 2-of-3 plus minisign is already fully enough to handle all of this.

Specifically:

  • The root is made up of 3 keys. As long as 2 of them sign, the next version of the root can take over. The root is valid for two years, which means I update the root roughly every two years, and a new root has to be signed by enough keys from both the old root and the new root.
  • The root doesn’t sign the update server’s key directly. It names another offline key, which isn’t allowed to be one of the root keys, and that key is only used to sign an index-keys list. This list is valid for one year and states which key the update server may use to sign the index.
  • The update server only has the one minisign key that signs the index. It has no root key, and none of my OpenPGP keys or my Authenticode certificate. The index is valid for 14 days, and the server re-signs it every day.

So the whole chain is: root → offline key → index-keys list → index key on the update server → index.json. If something goes wrong with the update server’s key, or if I just feel like changing it once in a while, I only need to sign a new list with the offline key, without touching the root.

There’s no way I would write all this JSON by hand, or generate three keys manually and configure them one by one. So I wrote a set of scripts that generate everything automatically according to the design of this mechanism, and I only need to enter passwords. What they generate goes into the project repository under release/root/ and release/index-keys/, next to more readable files meant for humans, that is, for me or for other people reviewing the code. The three keys in the current root are 4E179325A34DCF45, 27458E01B54EAF5D and 4D875241843652E8.

Why minisign instead of OpenPGP? GpgFrontend is itself an OpenPGP tool, so this is a natural question. The problem is that OpenPGP needs a context to be prepared before you can use it, and key management is fairly complex. It’s also usually tied to a person’s identity, so you can’t just casually generate a new key, and using subkeys makes it even more complex. At the end of the day, OpenPGP is mainly for individual users in all kinds of situations. For something read by machines and verified by the client itself, it’s best to depend on as little context as possible. minisign verification can be done with just libsodium. There’s no need to pull in all of GnuPG or RPGP, or to have GpgFrontend’s own context load anything. Something lightweight does the job. And here I only need verification, and only the key itself matters. There’s no need to look at a User ID like with a personal identity. So in the end I went with minisign.

But the bottom line is still there. Stable releases are still signed in the end with my offline OpenPGP key. No matter how much gets automated, every stable release needs me to personally confirm the version, type in the password by hand and sign with the key on my hardware token. This OpenPGP key is tied to my identity and is trusted by other people. Its fingerprint is 12F7E8858CF15BEC9975FF3C5CA3DA246843FD03. This is the last line of defense, and going back to what this project is about, I think it’s enough.

The Client: Not Blindly Trusting the Update Server

With the new mechanism, the client can have less logic of its own and rely more on the metadata from the update server: which release lines there are, what state each line is in, which betas are available and which version to recommend. The update server can adjust these for security or other reasons. If version numbers or release lines change later, it can also guide the client fairly easily, without old clients breaking as soon as the rules change.

But the client can’t blindly trust the update server. The update server itself could be compromised, and its signing key could be stolen. So the client has the root built in at compile time, and after it gets the index, it verifies things itself:

  • Starting from its built-in root, it walks step by step to the newest root, and each step has to meet 2-of-3. Then it checks the index-keys list and confirms that the key that signed the index is on it;
  • It verifies the minisign signature of index.json and checks whether it has expired;
  • It checks the index’s serial number and doesn’t allow rolling back to an older index;
  • It uses the sha256 values in the index to check each version’s descriptor and Release Note, and then checks downloaded files against the sha256 in the descriptor.

If the update server is compromised or does something malicious, the client’s verification fails, and at worst you reject it. You just lose the convenience of updating automatically, but you can still download manually from the download page or from GitHub.

The Release Notes are separate now too. They’re no longer managed in GitHub Release but kept in the repository under release/notes/. A Release Note has to be signed with my OpenPGP key before it can be published, and its hash is written into the signed index. The links inside are controlled as well. Only https links to bktus.com and its subdomains, plus the GpgFrontend repository on GitHub, are allowed. They can’t link to unrelated, possibly malicious addresses, and links that don’t meet the rules are only shown as plain text.

As for automatic installation, the Nightly AppImage on Linux can already be installed by clicking “Install” in the update window. The downloaded file only replaces the old one if its size and sha256 match the descriptor. For stable releases, I’ll look later at how to do automatic installation safely, so it’s convenient for users and still secure.

One Set of Scripts: gfrelease

This model comes with a set of scripts, kept right in the GpgFrontend repository (scripts/gfrelease), and they are the concrete implementation of the whole model. The update server reuses these scripts directly to do its checks and handle metadata. I use the same scripts myself too, for example to update the root, update the index-keys list, and download stable builds to sign locally and upload back to staging. In other words, it’s not that the update server has one implementation and I have another. We both use the same implementation based on this model. The update server itself only has some peripheral parts, like the bits that keep it from going down, the Dockerfile, and a web page where I can see its current status at a glance. The core things, including the implementation of the model itself, the various checks and the OpenPGP verification, are all in the code repository, where everyone can see and review them.

As for the update server’s own repository, I wanted to open source it at first, but later I changed my mind. It involves some configuration of my own servers, and if someone knew that configuration, it might be easier for them to find other things after breaking in. The Dockerfile itself is fine, but the compose file has environment variables, and if some key or something like that got uploaded by accident, that would be a pain. It’s a very thin layer anyway, so I chose not to open source it.

So I promise that it uses these scripts from the repository, but does that promise count for anything? I can’t prove it myself. In theory I could stop using these scripts, but in practice I do use them. Still, a personal promise doesn’t count for much. In security software, there’s no such thing as “I say it’s secure, so you have to trust me unconditionally.” That just doesn’t hold. So you can review the root and the related public keys stored in the repository, check whether your client carries the public keys from the commit in the repository that you trust, and verify for yourself what the update server pushes to the buckets. Whether or not the update server has been compromised, you can see it.

What the Update Check Sends, and What It Doesn’t

I think this part needs to be explained on its own:

  • The update check is off by default. The setup wizard on first launch has an option to check for updates, and it’s unticked by default. Once you turn it on, it checks once a day. You can also check manually under Help > Check for Updates.
  • The client only sends HTTPS GET requests, to update.bktus.com and builds.bktus.com, the domains of the two buckets. It doesn’t talk to GitHub, and there’s no other extra server.
  • The requests contain no user ID and no machine ID, there’s no request body, and the User-Agent is a fixed, generic browser string. Information like the operating system version and the architecture is only used locally to pick the right file and is never sent out.
  • One thing to note: the last request is commits/<commit>, and that path contains the commit of your build. It’s used to confirm whether the build comes from the official repository. So the server side can tell which build you’re running, and from that which version, even though the version number itself isn’t sent.
  • As with any HTTPS request, the server can see your IP address. The buckets are on Cloudflare, and Cloudflare handles and keeps this request data under its own rules. I don’t collect or analyze any access data myself, and there’s no need for me to do that.

I live in the EU now, and under the EU legal framework, this has to be disclosed publicly. So I’ll publish a new version of the privacy policy later to bring it in line with the law, and explain in the documentation exactly what the update check accesses and what requests it sends.

How This Was Built

From the first version of the release model to the current implementation, you can see in the commit history that even with AI assistance, I went back and forth through quite a few rounds. First I came up with an idea and asked the AI to see if it could make a simple implementation first. Then I found that parts of the implementation didn’t follow software architecture principles, or had bugs, so I had it fix them. After the fixes I used it on Nightly, found it didn’t seem quite OK, and changed it again. Later I thought about what to do with other release lines or betas in the future, and changed it again. After the changes, I still had to have it check everything again for problems and whether the code was any good. For example, there was already something usable in one place, and it wrote another copy somewhere else, or wrote yet another function to calculate hashes. It needs to be made to check for these things itself.

I also read the code it commits myself, to see whether there are code smells and whether it matches my understanding of the model. Over several rounds, the AI might implement what you have in mind in a different way you don’t like, or simply misunderstand what you meant. You have to keep an eye on that yourself.

Afterword

Starting with 2.2.5, and including the current main line, the main version management has all moved to the new model and mechanism. The guidance files in the repository, the pipeline, the update server, the buckets, the client, plus my own computer, each have their own job, with high cohesion and low coupling, which I’d say also reflects my own experience. GitHub Release stays on as a mirror. I feel this mechanism is enough, and it probably won’t need big changes for a long time. There may be small fixes here and there.

For how to use it and the details, see these pages in the user manual: Version Numbering Policy, How Releases and Updates Work, how to verify downloaded files, and the download page.

That’s about it. This post mainly explains why I built the new version model and mechanism, and some of the basic ideas behind them. I dictated it, then had AI organize it, and after that I checked whether it matched what I think and whether it drifted from what I meant. It wasn’t generated in one click by having AI scan the repository. I don’t really like typing, and now that I can dictate, writing posts is much easier, so I don’t need to say everything at once either. If I have new ideas later, I’ll keep writing posts, and they’ll be linked from the Release Notes or the user manual, to help users understand what all these commits every day are actually updating, and what I’m thinking. If something is wrong or unclear, please let me know.