<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Open Source on Tech Philosopher</title><link>https://blog.bktus.com/en/tags/open-source/</link><description>Recent content in Open Source on Tech Philosopher</description><generator>Hugo</generator><language>en-US</language><lastBuildDate>Thu, 19 Nov 2026 21:06:01 +0100</lastBuildDate><atom:link href="https://blog.bktus.com/en/tags/open-source/index.xml" rel="self" type="application/rss+xml"/><item><title>GpgFrontend's New Version Model and Release Mechanism</title><link>https://blog.bktus.com/en/archives/651hln/</link><pubDate>Thu, 19 Nov 2026 21:06:01 +0100</pubDate><guid>https://blog.bktus.com/en/archives/651hln/</guid><description>&lt;p>Updates have always been one of GpgFrontend&amp;rsquo;s biggest pain points. From the very first versions, GpgFrontend relied on GitHub Release to publish versions and to check for updates, and it stayed that way for many years. Starting with 2.2.5, and including the current main line, the version numbering rules, the release process and the way the client checks for updates have all moved to a new model and mechanism. How to use it and how to verify things is already in the user manual. But why I did it this way, and what I was thinking at the time, doesn&amp;rsquo;t come across in the documentation.&lt;/p></description><content:encoded><![CDATA[<p>Updates have always been one of GpgFrontend&rsquo;s biggest pain points. From the very first versions, GpgFrontend relied on GitHub Release to publish versions and to check for updates, and it stayed that way for many years. Starting with 2.2.5, and including the current main line, the version numbering rules, the release process and the way the client checks for updates have all moved to a new model and mechanism. How to use it and how to verify things is already in the user manual. But why I did it this way, and what I was thinking at the time, doesn&rsquo;t come across in the documentation.</p>
<p>For security software like GpgFrontend, transparency is a basic requirement. And updates are directly about what actually ends up on the user&rsquo;s computer: where the client connects in the background and what it sends. Users have every reason to be suspicious about that. So I think it&rsquo;s necessary to explain it clearly, so that users know what the update side really looks like. If you want to know how GpgFrontend got here, you can read my <a href="https://blog.bktus.com/en/archives/u8hywl/">previous post</a>.</p>
<h2 id="many-years-of-github-release">Many Years of GitHub Release</h2>
<p>In the beginning, GitHub Release was actually pretty good, at least until a couple of years ago. One thing is that it gives you free distribution. Another is that its model is simple enough: a release is tied to a Git tag. When I released a version, I just created a tag, wrote a Release Note, and uploaded all the installers built by GitHub Actions in one go, and that was it.</p>
<p>Later, the project started moving faster, especially after I brought in AI coding assistance, and now agents too, and a lot of features were added. You don&rsquo;t always notice these features day to day, but when you really use them, you still find some problems. On the old tick-tock rhythm, I released a version every two months. A lot of the time, a problem I missed before a release would turn up one, two or three days after it, found by me or by someone else. At that point it was hard for me to release again right away, because the rhythm was one version every two months, and releasing in between was a hassle.</p>
<p>But on the other hand, releasing every day isn&rsquo;t good either. Most users care about stability and don&rsquo;t want to try new things. They might check for updates once every two months, or even less often. Anyway, most users don&rsquo;t need most of the new features. What they use is encryption, decryption, key generation and so on. Take email, for example. In the past it could basically only work with EML files, and I&rsquo;ll admit myself that it was pretty hard to use. Now the email module has been reworked quite a bit and can cover most everyday email use, but the people using it are still a minority.</p>
<p>So for most users, waiting two months probably doesn&rsquo;t feel like anything, because I check things before every release and the basic features don&rsquo;t break. But users who really do hit a problem have to wait two months to get a fixed version, and that&rsquo;s pretty uncomfortable. I can feel this myself. Still, when I find a problem I can just fix it, build it myself and use it, so anyway it doesn&rsquo;t affect me. But imagine you don&rsquo;t have the means to build it yourself, or you&rsquo;re not a technical user. Then you have to wait two months.</p>
<p>That&rsquo;s why I later introduced Nightly Builds, which follow my development line. They used to follow the develop branch, and now they follow the main branch. As soon as I commit and push to GitHub, the pipeline starts a build automatically. Nightly and the stable release are of course different. Whatever you say about it, the stable release is a set of features that have been verified, while the main line brings in a lot of new features. Most of the time these new features don&rsquo;t conflict with the stable ones, but sometimes I make architectural changes or optimizations, and at that point I can&rsquo;t yet verify that they won&rsquo;t affect stability. So Nightly uses a different profile from the stable release: it runs as a separate app, GpgFrontend Testing, with its own keys and settings.</p>
<p>As for how it was published, Nightly had only one <code>nightly</code> tag. When the pipeline ran, it pulled the latest commit on the branch and built it, then force-overwrote the existing <code>nightly</code> tag, used an action plugin to create the release, and filled in the Release Note template for the Nightly Build automatically. The stable release kept the usual <code>vX.Y.Z</code> tags. Nightly and the stable release lived side by side on GitHub Release like this for at least a year or two, and it felt fine. But Nightly had no numbering like 1, 2, 3, so users could only see the latest one.</p>
<h2 id="the-turning-point-22-and-multiple-release-lines">The Turning Point: 2.2 and Multiple Release Lines</h2>
<p>The real turning point was version 2.2. 2.2 introduced a second engine, so the GnuPG engine and the RPGP engine can be used at the same time. It was a fairly big architectural change. Back then, I needed time to prove that the changes that came with RPGP wouldn&rsquo;t affect existing users who mainly use GnuPG. So I kept the 2.1 line, the original single-engine release line, and kept updating it as usual, while the 2.2 line also kept moving forward.</p>
<p>At that point, 2.1 users would very likely want to stay on 2.1. For them, I could only push 2.1 updates first, and then recommend upgrading to 2.2 once 2.2 was stable. If you use GitHub Release for this, the only way is to make the client logic thicker. The client calls the GitHub Release API, works out which versions 2.2 has and which versions 2.1 has, gets the metadata of each version, and then calculates locally: does 2.1 have a new version, is 2.2 already stable or still in pre-release, has anything been withdrawn, and so on. Only then does it notify the user. It&rsquo;s doable, but it always felt pretty awkward to me.</p>
<p>GitHub&rsquo;s release model can express two release lines, but it&rsquo;s, how should I put it, missing some flexibility. For example, when you publish a release, you can mark it as latest, but that latest is the latest across all versions. For an older release line, say one kept around for users on older systems, its newest version isn&rsquo;t the newest version of the whole project.</p>
<p>Another thing is EOL (End of Life). Say I plan to end 2.1 on December 31 of some year. With GitHub Release, I can&rsquo;t just take all of 2.1 down, because all of it has to be archived and kept on record. So the only way is to hard-code it in the client: after December 31, 2.1 is EOL, and it recommends upgrading to 2.2. But the client is software installed on the user&rsquo;s computer. Once it&rsquo;s released, that value is very hard to change. If I want to extend support, or end it earlier, there&rsquo;s no way to do it unless the client reads the value remotely from somewhere.</p>
<p>Then there&rsquo;s the rate limit. GitHub&rsquo;s API limits how often you can call it. Once at my company, I downloaded GpgFrontend on my own computer and wanted to check for updates, and it turned out the whole company&rsquo;s exit IP had already hit the limit, so it simply didn&rsquo;t work. With a VPN it was often the same, unless you log in, and the client can&rsquo;t log in, which is pretty awkward. On top of that, it&rsquo;s tied too tightly to the platform: you can only distribute through GitHub. If I wanted my website to detect the user&rsquo;s platform automatically and offer a download link, that link could only point to GitHub. Downloads weren&rsquo;t very stable either, and with a VPN they could be pretty slow, so it wasn&rsquo;t a great experience. (As for the many people who don&rsquo;t want to touch anything GitHub-related since Microsoft bought it, that&rsquo;s a side note, not the main reason.)</p>
<p>Of course, all of this isn&rsquo;t completely impossible on GitHub Release. For example, you can write some agreed-upon data into the Release Note, have the client read it, and decide based on that. But one thing is that it&rsquo;s not a proper standard, and another is that it&rsquo;s fragile. It&rsquo;s like a kitchen knife: you can use it for almost anything, but in some special cases using it is just clumsy.</p>
<p>And precisely because the GitHub approach needed so much extra logic and was so easy to get wrong, I simply didn&rsquo;t add it. I kept the update check plugin very simple: it only talked to GitHub, read some simple metadata and made a decision. The result was that the plugin wasn&rsquo;t very useful, and it did almost no verification. How would the client know that a newly published version was signed? It&rsquo;s hard to conveniently get these signatures from GitHub, or metadata like the “root” behind the signatures. Actually, I rarely used this update plugin myself.</p>
<h2 id="the-new-version-numbering-rules">The New Version Numbering Rules</h2>
<p>Before getting to the new release mechanism, let me talk about the change in version numbers. Under the old rules, a change in the last number didn&rsquo;t mean the release was just a patch. It meant another step forward in a certain direction. For example, the 2.1 direction was about steadily improving all kinds of features on a single engine. 2.2 was multi-engine, and 2.2.1 and 2.2.2 just kept exploring in the multi-engine direction, which didn&rsquo;t mean those updates were small. Someone did raise this as an issue, and that&rsquo;s how I explained it back then. From 1.0 all the way to 2.2.4, that was how I thought about it.</p>
<p>Starting with 2.2.5, to fit the new release model, or rather to make it more sensible, the last number now really is just a patch:</p>
<ul>
<li>The first number is the “Generation”, a change like the one from 1.0 to 2.0. It only changes for a really major architectural change, or when the whole idea of the product changes. It has nothing to do with how many lines of code went in.</li>
<li>The second number is the feature release, which is what the “direction” used to be. 2.2 to 2.3, or 2.3 to 2.4, really means big changes, roughly every two months.</li>
<li>The third number is the fix release. 2.3.1 to 2.3.2, or 2.3.2 to 2.3.3, goes back to what it originally meant: mainly security updates and fixes. It may bring a few features, but as few as possible, unless it&rsquo;s really necessary.</li>
</ul>
<p>The benefit is that stable users can get fixes without needing new features. For example, a user might not want two engines but still want security fixes for the GnuPG engine, or might want to stay on 2.2, which works well for them, rather than move to 2.3. For me, it also makes it fairly easy to control the pace of each release line: I cherry-pick fixes from the main line onto each line, and then push them to the right users through the current update mechanism. Moving from 2.2 to 2.3 is only a recommendation. You can still stay on the 2.2 line and keep receiving patches.</p>
<p>But it&rsquo;s hard for me alone to maintain a lot of release lines. Usually it goes like this: I want to move from 2.2 to 2.3, but 2.3 brings in a lot of new things and doesn&rsquo;t settle down that quickly, so 2.2 definitely has to stay around for a while, and at that point there are two lines. Add the fairly important single-engine 2.1 line, and that&rsquo;s three. Three is the most I&rsquo;ll maintain. The current state is: 2.3 is the main line, 2.2 is stable, and 2.1 is legacy, which only receives security fixes until December 31, 2027. It&rsquo;s also the last series that builds with Qt 5. With the new mechanism, I can update 2.1 once in a while, update 2.2 at the usual pace, and keep pushing 2.3 forward. All of this state is written in <code>release/policy.toml</code> in the repository.</p>
<p>Also, the Nightly version numbers changed. Before, I wrote the version number, say 2.3.0, into the CMake project, so every Nightly Build was 2.3.0, no matter which commit it was built from. Users could see the commit in the build information, but it wasn&rsquo;t convenient to refer to. Now the Nightly version number is derived from git and looks like <code>2.3.0-alpha.138+g&lt;commit&gt;</code>. The number after <code>alpha</code> is the number of commits since the most recent base tag. This makes it easy for other people to refer to and easy for me to find. I don&rsquo;t need to dig through long commit messages and match them up in the repository anymore. When I see alpha.250, I roughly know which day&rsquo;s version it is. Users can also keep the last three or four Nightlies. If a new one doesn&rsquo;t work well, they can go back to the previous one for a while and update again after that stretch of development is over.</p>
<p>Besides stable releases and Nightly, there are also betas and RCs (Release Candidates). Say I&rsquo;m about to release 2.4, and the features on the main line feel about done, but it&rsquo;s not very stable yet, and I want users who are willing to try it to test it first. That&rsquo;s when there&rsquo;s an RC like <code>2.4.0-rc.1</code>. A beta is a test build of the next fix release of a line, with a version number like <code>2.3.2-alpha.3</code>. Betas and RCs are only offered to users who have ticked “Also offer beta versions” in the settings, and that option is off by default.</p>
<h2 id="the-release-mechanism-three-buckets-and-one-update-server">The Release Mechanism: Three Buckets and One Update Server</h2>
<p>With the background of the version model covered, let me talk about the release mechanism. I don&rsquo;t really want to use GitHub Release as the main release channel anymore. The biggest pain points are still the ones I mentioned before: one is that it&rsquo;s pretty slow, and another is the rate limit.</p>
<p>Actually, I&rsquo;ve always been fairly interested in object storage, and I felt something like AWS S3 should host this. A normal web page would also work, but in comparison, the client only needs a metadata file, say a JSON file, and doesn&rsquo;t need the JS and CSS that come with a web page. Also, a web server might let people scan its directories, which is a hassle. I don&rsquo;t want people to be able to grab everything at once. I&rsquo;d rather keep the metadata separate: the client first gets the metadata at the root, and when it needs something else, that metadata tells it where the next index is, and then it goes and fetches that. A lot of technical things can be done in different ways, but sometimes you just have an intuition about which one is better and which one feels more comfortable to me.</p>
<p>I originally wanted to use S3, but its outbound traffic is billed separately, and that&rsquo;s hard for me to control. What if this software gets hit by a DDoS attack and a few hundred TB of traffic gets pulled out at once? With the little money that comes from donations, a few hundred TB every day is not something I could pay for. Later I found that Cloudflare&rsquo;s R2 doesn&rsquo;t charge for distribution traffic, and I&rsquo;ve been using Cloudflare&rsquo;s CDN all along. It did have some incidents a while ago, but for something like software distribution, which doesn&rsquo;t need high availability, it&rsquo;s still OK. So I chose R2.</p>
<p>Then the question is what to store in the buckets, and who puts things into them. The principle I set for myself is that the new mechanism must free up my personal time and make releasing faster, especially for stable releases. In the past, to release a stable version, I had to download the installers one by one, sign them with my Authenticode certificate on my Windows computer, then do an OpenPGP signature on all of them, and then upload them one by one. It took two or three hours and was easy to get wrong. Sometimes I even got the file names wrong. I&rsquo;ve been using scripts since 2.0, and later I had AI write a script for me that downloads automatically from GitHub, asks me to confirm and enter a password for the certificate signature, then a password for the GPG signature, puts everything in one folder, and then slowly uploads it to GitHub Release. That was a lot faster. But I still had to wait for the pipeline to build, with one eye on the pipeline and the other on my own work. And where I live now, access to GitHub Release is pretty slow: the bandwidth isn&rsquo;t high and the VPN is throttled, which made it pretty painful. Another pain point was that Nightly never had a signature. There was no way I could download every build and sign it with my own GPG key, and putting my GPG private key into the pipeline was even more out of the question.</p>
<p>The current structure looks like this:</p>
<pre class="mermaid">
  flowchart LR
  CI[GitHub Actions] --&gt;|builds| Staging[(Staging bucket)]
  Staging --&gt;|stable builds| Me[Maintainer machine]
  Me --&gt;|signed| Staging
  Staging --&gt; Server[Release server]
  Repo[GpgFrontend repository] --&gt;|policy, trust, scripts| Server
  Server --&gt;|promote| Builds[(Builds bucket)]
  Server --&gt;|signed index| Update[(Update bucket)]
  Builds --&gt; Client[GpgFrontend]
  Update --&gt; Client
</pre>
<p>Let me go through them one by one:</p>
<ul>
<li>The pipeline: after a build finishes, it uploads the artifacts and <code>SHA256SUMS.txt</code> to the staging bucket. It also signs <code>SHA256SUMS.txt</code> with Sigstore (cosign), which proves that these files really were built by the <code>build.yml</code> pipeline of the GpgFrontend repository at the matching commit. This kind of signature doesn&rsquo;t need me to hand any private key to the pipeline, so Nightly finally has a signature.</li>
<li>The staging bucket: private and not open to users. It holds things that haven&rsquo;t been checked yet.</li>
<li>My computer: only involved for stable releases. I download the stable build from staging, sign the Windows installers locally with Authenticode using the certificate on my hardware token, sign with my OpenPGP key, and then upload everything back to staging. That certificate lives on a hardware token and can&rsquo;t be exported, so signing in the cloud is impossible, and it wouldn&rsquo;t be safe anyway.</li>
<li>The update server: it checks what&rsquo;s in staging, moves it to the builds bucket once the checks pass, and then updates and signs the index.</li>
<li>The builds bucket (<code>builds.bktus.com</code>): holds the installers, the descriptor of each version and the Release Notes. Users download from here.</li>
<li>The update bucket (<code>update.bktus.com</code>): holds the signed index <code>index.json</code>, plus trust-related files like the root and the index-keys.</li>
</ul>
<p>Why keep builds and update apart? One reason is not to put everything in one basket, so build artifacts and metadata are stored separately. Another is that it&rsquo;s easier to manage: one bucket only holds metadata and the other only holds installers, so they don&rsquo;t get mixed up. Add staging, and three buckets is about right.</p>
<p>The update server sits in the middle and acts as a checkpoint for users. For Nightly, beta and RC, it uses cosign to verify the Sigstore signature and confirms that the build came from <code>build.yml</code> and that both the ref and the commit match. Every file has to match the signed <code>SHA256SUMS.txt</code>, and there can&rsquo;t be any extra files. It derives the version number itself from git instead of trusting the version number the build claims. For stable releases, it also checks my OpenPGP signatures and the signed tag, as well as the Authenticode signer of the Windows installers. Besides that, it reads the guidance file <code>release/policy.toml</code> in the repository, things like when 2.1 reaches EOL and what state 2.2 and 2.3 are in, and generates metadata that&rsquo;s easy for the client to read, which is the index.</p>
<p>The update server is important, because it decides what the client can see. But it&rsquo;s not the most critical link. It doesn&rsquo;t take requests from clients directly, so it doesn&rsquo;t need high concurrency or DDoS protection. It just moves things from the unverified place to the verified place and then updates the related metadata. High availability, high concurrency and DDoS protection are left to Cloudflare&rsquo;s infrastructure. Writing code myself specifically to handle these would be a huge project, and there&rsquo;s no need for it. So the load sits on the buckets, and management and admission sit on the update server. And the client does its own verification too. It treats the update server with suspicion as well, which I&rsquo;ll get to later.</p>
<h2 id="the-signing-root-2-of-3-and-minisign">The Signing Root: 2-of-3 and minisign</h2>
<p>Another question is the signing root. A signing key can&rsquo;t go without rotation forever. I can&rsquo;t use the same key for 30 years. So how do I manage it? I didn&rsquo;t use CA certificates, because that would actually be more complicated. I know X.509 is very flexible and you can add all kinds of fields, but managing it already takes a lot of tools, OpenSSL and so on. It&rsquo;s too flexible and has too many features. This is just a software update. There&rsquo;s no need to do it like Apple: set up a CA root, self-sign it, hold some kind of ceremony, use the CA certificate to sign an intermediate certificate, and then use the intermediate certificate to sign other things. It can be done, but there&rsquo;s no need. Day to day, 2-of-3 plus minisign is already fully enough to handle all of this.</p>
<p>Specifically:</p>
<ul>
<li>The root is made up of 3 keys. As long as 2 of them sign, the next version of the root can take over. The root is valid for two years, which means I update the root roughly every two years, and a new root has to be signed by enough keys from both the old root and the new root.</li>
<li>The root doesn&rsquo;t sign the update server&rsquo;s key directly. It names another offline key, which isn&rsquo;t allowed to be one of the root keys, and that key is only used to sign an index-keys list. This list is valid for one year and states which key the update server may use to sign the index.</li>
<li>The update server only has the one minisign key that signs the index. It has no root key, and none of my OpenPGP keys or my Authenticode certificate. The index is valid for 14 days, and the server re-signs it every day.</li>
</ul>
<p>So the whole chain is: root → offline key → index-keys list → index key on the update server → <code>index.json</code>. If something goes wrong with the update server&rsquo;s key, or if I just feel like changing it once in a while, I only need to sign a new list with the offline key, without touching the root.</p>
<p>There&rsquo;s no way I would write all this JSON by hand, or generate three keys manually and configure them one by one. So I wrote a set of scripts that generate everything automatically according to the design of this mechanism, and I only need to enter passwords. What they generate goes into the project repository under <code>release/root/</code> and <code>release/index-keys/</code>, next to more readable files meant for humans, that is, for me or for other people reviewing the code. The three keys in the current root are <code>4E179325A34DCF45</code>, <code>27458E01B54EAF5D</code> and <code>4D875241843652E8</code>.</p>
<p>Why minisign instead of OpenPGP? GpgFrontend is itself an OpenPGP tool, so this is a natural question. The problem is that OpenPGP needs a context to be prepared before you can use it, and key management is fairly complex. It&rsquo;s also usually tied to a person&rsquo;s identity, so you can&rsquo;t just casually generate a new key, and using subkeys makes it even more complex. At the end of the day, OpenPGP is mainly for individual users in all kinds of situations. For something read by machines and verified by the client itself, it&rsquo;s best to depend on as little context as possible. minisign verification can be done with just libsodium. There&rsquo;s no need to pull in all of GnuPG or RPGP, or to have GpgFrontend&rsquo;s own context load anything. Something lightweight does the job. And here I only need verification, and only the key itself matters. There&rsquo;s no need to look at a User ID like with a personal identity. So in the end I went with minisign.</p>
<p>But the bottom line is still there. Stable releases are still signed in the end with my offline OpenPGP key. No matter how much gets automated, every stable release needs me to personally confirm the version, type in the password by hand and sign with the key on my hardware token. This OpenPGP key is tied to my identity and is trusted by other people. Its fingerprint is <code>12F7E8858CF15BEC9975FF3C5CA3DA246843FD03</code>. This is the last line of defense, and going back to what this project is about, I think it&rsquo;s enough.</p>
<h2 id="the-client-not-blindly-trusting-the-update-server">The Client: Not Blindly Trusting the Update Server</h2>
<p>With the new mechanism, the client can have less logic of its own and rely more on the metadata from the update server: which release lines there are, what state each line is in, which betas are available and which version to recommend. The update server can adjust these for security or other reasons. If version numbers or release lines change later, it can also guide the client fairly easily, without old clients breaking as soon as the rules change.</p>
<p>But the client can&rsquo;t blindly trust the update server. The update server itself could be compromised, and its signing key could be stolen. So the client has the root built in at compile time, and after it gets the index, it verifies things itself:</p>
<ul>
<li>Starting from its built-in root, it walks step by step to the newest root, and each step has to meet 2-of-3. Then it checks the index-keys list and confirms that the key that signed the index is on it;</li>
<li>It verifies the minisign signature of <code>index.json</code> and checks whether it has expired;</li>
<li>It checks the index&rsquo;s serial number and doesn&rsquo;t allow rolling back to an older index;</li>
<li>It uses the sha256 values in the index to check each version&rsquo;s descriptor and Release Note, and then checks downloaded files against the sha256 in the descriptor.</li>
</ul>
<p>If the update server is compromised or does something malicious, the client&rsquo;s verification fails, and at worst you reject it. You just lose the convenience of updating automatically, but you can still download manually from the download page or from GitHub.</p>
<p>The Release Notes are separate now too. They&rsquo;re no longer managed in GitHub Release but kept in the repository under <code>release/notes/</code>. A Release Note has to be signed with my OpenPGP key before it can be published, and its hash is written into the signed index. The links inside are controlled as well. Only https links to <code>bktus.com</code> and its subdomains, plus the GpgFrontend repository on GitHub, are allowed. They can&rsquo;t link to unrelated, possibly malicious addresses, and links that don&rsquo;t meet the rules are only shown as plain text.</p>
<p>As for automatic installation, the Nightly AppImage on Linux can already be installed by clicking “Install” in the update window. The downloaded file only replaces the old one if its size and sha256 match the descriptor. For stable releases, I&rsquo;ll look later at how to do automatic installation safely, so it&rsquo;s convenient for users and still secure.</p>
<h2 id="one-set-of-scripts-gfrelease">One Set of Scripts: gfrelease</h2>
<p>This model comes with a set of scripts, kept right in the GpgFrontend repository (<code>scripts/gfrelease</code>), and they are the concrete implementation of the whole model. The update server reuses these scripts directly to do its checks and handle metadata. I use the same scripts myself too, for example to update the root, update the index-keys list, and download stable builds to sign locally and upload back to staging. In other words, it&rsquo;s not that the update server has one implementation and I have another. We both use the same implementation based on this model. The update server itself only has some peripheral parts, like the bits that keep it from going down, the Dockerfile, and a web page where I can see its current status at a glance. The core things, including the implementation of the model itself, the various checks and the OpenPGP verification, are all in the code repository, where everyone can see and review them.</p>
<p>As for the update server&rsquo;s own repository, I wanted to open source it at first, but later I changed my mind. It involves some configuration of my own servers, and if someone knew that configuration, it might be easier for them to find other things after breaking in. The Dockerfile itself is fine, but the compose file has environment variables, and if some key or something like that got uploaded by accident, that would be a pain. It&rsquo;s a very thin layer anyway, so I chose not to open source it.</p>
<p>So I promise that it uses these scripts from the repository, but does that promise count for anything? I can&rsquo;t prove it myself. In theory I could stop using these scripts, but in practice I do use them. Still, a personal promise doesn&rsquo;t count for much. In security software, there&rsquo;s no such thing as “I say it&rsquo;s secure, so you have to trust me unconditionally.” That just doesn&rsquo;t hold. So you can review the root and the related public keys stored in the repository, check whether your client carries the public keys from the commit in the repository that you trust, and verify for yourself what the update server pushes to the buckets. Whether or not the update server has been compromised, you can see it.</p>
<h2 id="what-the-update-check-sends-and-what-it-doesnt">What the Update Check Sends, and What It Doesn&rsquo;t</h2>
<p>I think this part needs to be explained on its own:</p>
<ul>
<li>The update check is off by default. The setup wizard on first launch has an option to check for updates, and it&rsquo;s unticked by default. Once you turn it on, it checks once a day. You can also check manually under Help &gt; Check for Updates.</li>
<li>The client only sends HTTPS GET requests, to <code>update.bktus.com</code> and <code>builds.bktus.com</code>, the domains of the two buckets. It doesn&rsquo;t talk to GitHub, and there&rsquo;s no other extra server.</li>
<li>The requests contain no user ID and no machine ID, there&rsquo;s no request body, and the User-Agent is a fixed, generic browser string. Information like the operating system version and the architecture is only used locally to pick the right file and is never sent out.</li>
<li>One thing to note: the last request is <code>commits/&lt;commit&gt;</code>, and that path contains the commit of your build. It&rsquo;s used to confirm whether the build comes from the official repository. So the server side can tell which build you&rsquo;re running, and from that which version, even though the version number itself isn&rsquo;t sent.</li>
<li>As with any HTTPS request, the server can see your IP address. The buckets are on Cloudflare, and Cloudflare handles and keeps this request data under its own rules. I don&rsquo;t collect or analyze any access data myself, and there&rsquo;s no need for me to do that.</li>
</ul>
<p>I live in the EU now, and under the EU legal framework, this has to be disclosed publicly. So I&rsquo;ll publish a new version of the privacy policy later to bring it in line with the law, and explain in the documentation exactly what the update check accesses and what requests it sends.</p>
<h2 id="how-this-was-built">How This Was Built</h2>
<p>From the first version of the release model to the current implementation, you can see in the commit history that even with AI assistance, I went back and forth through quite a few rounds. First I came up with an idea and asked the AI to see if it could make a simple implementation first. Then I found that parts of the implementation didn&rsquo;t follow software architecture principles, or had bugs, so I had it fix them. After the fixes I used it on Nightly, found it didn&rsquo;t seem quite OK, and changed it again. Later I thought about what to do with other release lines or betas in the future, and changed it again. After the changes, I still had to have it check everything again for problems and whether the code was any good. For example, there was already something usable in one place, and it wrote another copy somewhere else, or wrote yet another function to calculate hashes. It needs to be made to check for these things itself.</p>
<p>I also read the code it commits myself, to see whether there are code smells and whether it matches my understanding of the model. Over several rounds, the AI might implement what you have in mind in a different way you don&rsquo;t like, or simply misunderstand what you meant. You have to keep an eye on that yourself.</p>
<h2 id="afterword">Afterword</h2>
<p>Starting with 2.2.5, and including the current main line, the main version management has all moved to the new model and mechanism. The guidance files in the repository, the pipeline, the update server, the buckets, the client, plus my own computer, each have their own job, with high cohesion and low coupling, which I&rsquo;d say also reflects my own experience. GitHub Release stays on as a mirror. I feel this mechanism is enough, and it probably won&rsquo;t need big changes for a long time. There may be small fixes here and there.</p>
<p>For how to use it and the details, see these pages in the user manual: <a href="https://gpgfrontend.bktus.com/appendix/version-policy/">Version Numbering Policy</a>, <a href="https://gpgfrontend.bktus.com/appendix/release-updates/">How Releases and Updates Work</a>, <a href="https://gpgfrontend.bktus.com/appendix/code-binary-verify/">how to verify downloaded files</a>, and the <a href="https://gpgfrontend.bktus.com/overview/downloads/">download page</a>.</p>
<p>That&rsquo;s about it. This post mainly explains why I built the new version model and mechanism, and some of the basic ideas behind them. I dictated it, then had AI organize it, and after that I checked whether it matched what I think and whether it drifted from what I meant. It wasn&rsquo;t generated in one click by having AI scan the repository. I don&rsquo;t really like typing, and now that I can dictate, writing posts is much easier, so I don&rsquo;t need to say everything at once either. If I have new ideas later, I&rsquo;ll keep writing posts, and they&rsquo;ll be linked from the Release Notes or the user manual, to help users understand what all these commits every day are actually updating, and what I&rsquo;m thinking. If something is wrong or unclear, please let me know.</p>
]]></content:encoded></item><item><title>The Past and Present of GpgFrontend: My Journey with an Open-Source Encryption Tool</title><link>https://blog.bktus.com/en/archives/u8hywl/</link><pubDate>Wed, 12 Nov 2025 23:24:37 +0100</pubDate><guid>https://blog.bktus.com/en/archives/u8hywl/</guid><description>&lt;h2 id="how-it-started-running-into-gnupg-gpg">How It Started: Running into GnuPG (GPG)&lt;/h2>
&lt;p>I first came across GPG around April 2021. Back then I was just a student who was curious about everything in the computer world, and my understanding of cryptography didn&amp;rsquo;t go much further than “a password is an encrypted string”. Then I read about how asymmetric encryption works: anyone can use a public key to encrypt a message, but only the person who holds the private key can decrypt it. That really amazed me. I liked this kind of “mathematical trust”. It&amp;rsquo;s elegant, rigorous and irreversible, and at that moment I felt it was just as beautiful as any piece of art. So I started learning the GPG command line, from generating keys to verifying signatures, trying things again and again to understand the logic behind them.&lt;/p></description><content:encoded><![CDATA[<h2 id="how-it-started-running-into-gnupg-gpg">How It Started: Running into GnuPG (GPG)</h2>
<p>I first came across GPG around April 2021. Back then I was just a student who was curious about everything in the computer world, and my understanding of cryptography didn&rsquo;t go much further than “a password is an encrypted string”. Then I read about how asymmetric encryption works: anyone can use a public key to encrypt a message, but only the person who holds the private key can decrypt it. That really amazed me. I liked this kind of “mathematical trust”. It&rsquo;s elegant, rigorous and irreversible, and at that moment I felt it was just as beautiful as any piece of art. So I started learning the GPG command line, from generating keys to verifying signatures, trying things again and again to understand the logic behind them.</p>
<p>But I also found a problem: the command line is really hard to get into. Sometimes I just wanted to do something simple quickly, and I still had to type a long string of commands, which honestly annoyed me. For non-technical users, the GPG command line is like an obscure wasteland, and it stops almost everyone who wants to try it. So I started thinking: what if GPG had a simple, easy-to-use graphical interface, so it felt like a crypto machine you can actually see and touch? If everyone could easily have a small, durable “crypto machine”, maybe more people would be willing to try it. At the time this idea seemed pretty small, but it was what started the next several years of my technical work.</p>
<blockquote>
<p>The term “crypto machine” immediately evokes a sense of reliability,
something like the mechanical encryption devices of the past, such as the
Enigma. It’s not just technology, but also a belief in security. Such a device
doesn’t need endless features. It just needs to be <strong>precise and dependable</strong>.
When a user presses a button, the result must be <strong>predictable and safe</strong>. That
pursuit of control and predictability is the soul of a crypto machine.</p></blockquote>
<p>I started looking for existing solutions. There were already some GUI tools back then, like Kleopatra. But a lot of them had complicated interfaces and too many features, and some even brought in the X.509 certificate system, which put me off right away. I wanted something lighter and simpler. Later I found a project called <a href="https://git.bktus.com/gpgfrontend/gpg4usb/">gpg4usb</a>, which was very close to what I had in mind: lightweight, portable and cross-platform. Unfortunately, it had stopped being updated long ago and only supported GPG 1.4. That meant compatibility problems with modern systems were almost unavoidable, and it couldn&rsquo;t support subkeys either.</p>
<p>So I started thinking: could I rewrite part of it and bring it back to life? Around May, I spent about half a month in my dorm reading its source code, figuring out how to call GPG&rsquo;s interfaces, and learning about the Qt framework and cross-platform compatibility. It was a lonely process, but a really fulfilling one. Every time I got a new feature working, I felt a sense of achievement that&rsquo;s hard to describe. When I finally got it running with GPG 2 and fixed quite a few of the original bugs, I decided to give the refreshed project a new name: <a href="https://gpgfrontend.bktus.com/">GpgFrontend</a>. It&rsquo;s a graphical frontend for GPG, and it was also the first complete open-source project of my life.</p>
<p><img alt="GpgFrontend" loading="lazy" src="https://img.bktus.com/P7a8YV8G.webp"></p>
<h2 id="going-public-from-dorm-room-code-to-github">Going Public: From Dorm Room Code to GitHub</h2>
<p>After I uploaded the first version to GitHub, I didn&rsquo;t expect much attention. After all, this is a very niche area, and GPG itself isn&rsquo;t something most people use. But a few days later, I got my first issue. A user said the program didn&rsquo;t run on a certain operating system. They provided logs and even suggested a fix. That was the first time I felt something that made me want to keep putting time into this project.</p>
<p>After that, more and more people started using GpgFrontend. They reported problems, made suggestions and even contributed translations. Every piece of feedback showed me that this project was no longer just my own tech toy. It was a tool people actually used. From then on, my understanding of open source changed completely. To me, it&rsquo;s not just sharing code anymore. It&rsquo;s a whole ecosystem that keeps getting feedback and keeps changing.</p>
<p>Over the next two or three years, I gradually understood that GpgFrontend can&rsquo;t be, and doesn&rsquo;t need to be, an all-in-one tool. Its purpose is very clear: to help users encrypt, decrypt, sign and verify more easily. Its value isn&rsquo;t in the number of features, it&rsquo;s in being easy to use and reliable. So now I clearly define it as a <strong>lightweight, secure crypto machine</strong>, which is exactly what I wanted when I first started using GPG.</p>
<p>For experienced users who know the command line, it&rsquo;s a tool that saves time. For beginners, it&rsquo;s a bridge into the world of cryptography. So in the UI, I try to keep things simple, so that every core operation is clear at a glance. Under the hood, every action maps directly to a native GPG command, to keep it stable and compatible. Later I realized that this “small but focused” positioning actually became what makes GpgFrontend different. While other projects go after more features, I chose to keep it focused.</p>
<h2 id="growing-using-the-project-as-my-testing-ground">Growing: Using the Project as My Testing Ground</h2>
<p>As the versions went on, GpgFrontend became my training ground for learning new technology. I kept trying to bring new ideas I picked up at work and in my studies into the project. I studied Chromium&rsquo;s multithreading architecture and brought its ideas of thread scheduling and task isolation into GpgFrontend, so the interface no longer freezes during encryption, and race conditions and other common multithreading bugs are reduced as much as possible. I designed a plugin system and an SDK, so third parties can extend features at runtime. I improved the build scripts and added CI/CD, so packaging for different platforms is more automated and more reliable. Also, through the cross-platform support, I learned about the many implementation differences between Windows, macOS and Linux, and how packaging works on different operating systems and distribution platforms.</p>
<p>These experiments made the tool itself better, and they also gave me real engineering experience: how to design an architecture that can be maintained, how to balance stability and new ideas, and how to refactor without breaking old features. Step by step, GpgFrontend grew from a simple experiment into an open-source tool with a reasonable structure that people can rely on.</p>
<p>When I was taking a network security course, the professor talked about why data security and key management matter so much. Suddenly I realized that some of the things I had implemented early in the project had hidden risks. For example, the key derivation algorithm was too simple, stored data had no key rotation, and some memory wasn&rsquo;t protected. So I started a full security refactoring.</p>
<p>In <a href="https://github.com/saturneric/GpgFrontend/releases/tag/v2.1.9">version 2.1.9</a>, I added security levels, startup self-checks, AES-GCM encryption mode, key rotation and secure memory. I also redesigned the program&rsquo;s data structures, so that temporary secrets in memory are wiped as safely as possible after they&rsquo;re used. These changes happen behind the scenes, but they made GpgFrontend noticeably more secure and reliable. As I&rsquo;ve always believed, security isn&rsquo;t a feature, it&rsquo;s an attitude. Only when security is built in from the design stage can a tool keep people&rsquo;s trust over time.</p>
<p>Over the years of maintaining the project, I&rsquo;ve gradually felt a kind of satisfaction that goes beyond the technology. GpgFrontend isn&rsquo;t a popular project, but it has really helped some real users. Some people use it to verify the signatures of software packages, some use it to handle secure communication quickly at work, and someone told me it was their first way into learning GPG. I don&rsquo;t track users, and I don&rsquo;t collect download numbers from each platform either. Honestly, I don&rsquo;t know how many people are using it. But I like not knowing. It makes me feel that GpgFrontend is a pure contribution. To put it more concretely, it&rsquo;s a small crypto machine that&rsquo;s always within reach, can go anywhere with you, and quietly does its job wherever people need it.</p>
<h2 id="looking-ahead-stable-focused-and-in-it-for-the-long-term">Looking Ahead: Stable, Focused and in It for the Long Term</h2>
<p>Over the next few years, I won&rsquo;t make big changes to GpgFrontend. Updates will continue, but they&rsquo;ll mostly be about a better user experience, more stability, bug fixes and small optimizations. I&rsquo;d rather keep it light and under control. The software industry today is full of fast iteration, and I actually hope GpgFrontend can be an exception: a steady, reliable tool you can trust. Maybe it will never be popular or make the tech headlines. But as long as it keeps helping the people who really need it, I think it&rsquo;s all worth it.</p>
<p>In recent years, I&rsquo;ve been paying close attention to how Rust is growing in the security field. I&rsquo;ve looked into a few Rust-based alternatives to the GPG libraries, like RPGP and Sequoia. They have better memory safety and concurrency management, and they support some of the newest OpenPGP features. I think this is clearly the direction things are going. But I also have to face reality: the Rust ecosystem isn&rsquo;t fully mature yet, and its compatibility with the existing GPG ecosystem still needs to be tested.</p>
<p>So I plan to set up a separate branch in the future, plug in Rust-based crypto libraries there as an experiment, and wrap them with a new key database layer. This kind of gradual migration lets me find out whether the new technology works without breaking the stability of the main line. I&rsquo;ve always believed that steady updates matter more than chasing new things. After all, the value of a security tool isn&rsquo;t in being new, it&rsquo;s in being trusted. And building that kind of trust may take something like ten years.</p>
]]></content:encoded></item><item><title>Running a “Vanilla” Mainline Kernel on Raspberry Pi 5: Full Workflow of Trimming, Cross-Compilation, and Deployment</title><link>https://blog.bktus.com/en/archives/czh2is/</link><pubDate>Sat, 14 Jun 2025 01:09:13 +0200</pubDate><guid>https://blog.bktus.com/en/archives/czh2is/</guid><description>&lt;h2 id="summary">Summary&lt;/h2>
&lt;p>This post records how I turned the Raspberry Pi 5 into a minimal development and testing platform for the mainline Linux kernel (the vanilla kernel). It covers how I trimmed the repository down, a custom kernel config, cross-compilation and automated deployment scripts, a trick for removing overlays, and what I’ve learned from keeping it in sync with the mainline community over time. It should be useful for geeks who follow Linux mainline, ARM64 support, kernel development or upstream contributions, or who just like tinkering with the kernel.&lt;/p></description><content:encoded><![CDATA[<h2 id="summary">Summary</h2>
<p>This post records how I turned the Raspberry Pi 5 into a minimal development and testing platform for the mainline Linux kernel (the vanilla kernel). It covers how I trimmed the repository down, a custom kernel config, cross-compilation and automated deployment scripts, a trick for removing overlays, and what I’ve learned from keeping it in sync with the mainline community over time. It should be useful for geeks who follow Linux mainline, ARM64 support, kernel development or upstream contributions, or who just like tinkering with the kernel.</p>
<h2 id="background-and-motivation">Background and Motivation</h2>
<p>Recently I’ve been studying the theory and architecture of the Linux kernel in depth, and I’m getting ready to properly move into OS kernel work. So I bought a Raspberry Pi 5 as a development board to learn and practice kernel development. But as soon as I started, I found that the Pi 5 can’t run the mainline Linux kernel directly. It can only rely on the <a href="https://github.com/raspberrypi/linux">kernel repository maintained by the Raspberry Pi team</a>. At the time, the latest branch was rpi-6.14.y (now it’s rpi-6.16.y). The repository is maintained by “dynamically pulling upstream mainline code + rebase + lots of custom patches”, which means tens of thousands of lines of changes and a history that keeps getting rewritten.</p>
<p>These custom patches really are important for the official kernel to run stably on the Raspberry Pi with every feature working. But for a developer like me, who <strong>just wants to study mainline kernel features</strong> and cares about the kernel itself, these patches turn into a burden:</p>
<ul>
<li>Syncing with mainline is hard, and merges often run into conflicts.</li>
<li>The version history is rewritten often, so it’s hard to trace back and compare.</li>
<li>Testing new mainline features or giving feedback becomes less straightforward.</li>
</ul>
<p>I looked at x86 development boards and found that most of them can run the mainline kernel directly, but they’re expensive and nowhere near as cost-effective as the Raspberry Pi 5. So I started thinking about how to get the Pi 5 to run the mainline kernel smoothly too. That way I could try out and test the latest kernel features at any time, and it would be easier to report problems and contribute patches for the ARM64 platform. My goal was clear: with as few patches as possible, and staying as close to upstream as possible, make the Raspberry Pi 5 an ideal ARM64 platform for my own mainline kernel experiments and learning.</p>
<h2 id="repository-structure-and-scope-of-changes">Repository Structure and Scope of Changes</h2>
<p>My approach is really simple and goes straight to the point: keep only the drivers and patches that are truly needed, and follow mainline closely for everything else. I only patch the drivers/ directory and focus on core hardware support, mainly:</p>
<ul>
<li>
<p><strong>SD card driver</strong>: makes sure the Raspberry Pi boots normally and the kernel can load the root filesystem. The boot firmware does part of the initialization itself, but the mainline kernel still needs the driver to mount the root filesystem.</p>
</li>
<li>
<p><strong>Wired network card (Ethernet)</strong>: makes it easy to log in remotely over SSH, download new kernel images, and do remote development and debugging. It’s one of the most important I/O channels during kernel development.</p>
</li>
<li>
<p><strong>UART serial port</strong>: the output window for kernel debugging and early print. I can watch the kernel log directly and tell how booting and running are going.</p>
</li>
<li>
<p><strong>USB controller</strong>: for mounting external devices or occasionally moving data offline, which covers more test scenarios.</p>
</li>
<li>
<p><strong>Basic GPIO, power management</strong> and other parts closely tied to the RPi5 hardware.</p>
</li>
<li>
<p>Remove support for subsystems from the official kernel that aren’t needed for kernel development, such as <strong>Wi-Fi, Bluetooth, GPU (VideoCore), HDMI and audio/video acceleration</strong>. They don’t mean much for studying and testing the kernel itself.</p>
</li>
</ul>
<p><strong>Maintain the Raspberry Pi 5 DTS (Device Tree) configuration under arch/arm64/boot/dts/</strong></p>
<ul>
<li>The focus is to make sure the RPi5’s hardware resources, CPU configuration and peripheral mappings are recognized correctly at boot, so the mainline kernel runs smoothly.</li>
</ul>
<p><strong>Keep everything else 1:1 in sync with Linus’s mainline</strong></p>
<ul>
<li>Apart from the few patches above, the rest of the repository stays exactly the same as the <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git">official mainline repository</a>.</li>
<li>Regularly (say, every month or every two weeks) pull the latest code from mainline. Then I only need to apply a handful of patches and resolve a few small conflicts.</li>
</ul>
<p>With this approach, the repository stays minimal and easy to maintain, and it also lays the groundwork for slimming down the config and keeping up with mainline quickly later on.</p>
<h2 id="example-of-a-custom-kernel-config">Example of a Custom Kernel Config</h2>
<p>During the actual deployment, I prepared my own .config for the Raspberry Pi 5 mainline kernel. The goal was to cover my own development, testing and service needs while <strong>cutting out unrelated features as much as possible and keeping build size and build time to a minimum</strong>. Getting there took a lot of trial and error and tuning.</p>
<h3 id="phase-one-trim-everything-keep-only-the-core">Phase One: Trim Everything, Keep Only the Core</h3>
<p>The first version of the .config turned off almost every driver and feature module I didn’t think I needed, including:</p>
<ul>
<li><strong>All network card drivers except the onboard one</strong> (I only kept the driver for the Raspberry Pi 5’s onboard Cadence network card)</li>
<li><strong>Virtualization support</strong> (KVM and so on)</li>
<li><strong>The graphics subsystem, sound cards, HDMI, multimedia acceleration and so on</strong></li>
<li><strong>All network file systems</strong></li>
<li><strong>All file systems except ext4 and the SD card (mmcblk)</strong></li>
<li><strong>Some crypto algorithms and kernel mechanisms</strong> related to the modules above</li>
<li>I even <strong>turned off cgroup</strong></li>
</ul>
<p>After a few rounds of debugging and real testing, the final config may not be the “most minimal” one in theory. But it already cut the kernel build time a lot, and the kernel boots stably and handles the basic functions I need.</p>
<h3 id="phase-two-practical-extensions-for-docker-workloads-and-everyday-services">Phase Two: Practical Extensions for Docker Workloads and Everyday Services</h3>
<p>As my kernel experiments went further, I realized that:</p>
<ul>
<li>I needed Docker to run some real workloads and check the kernel’s stability and compatibility;</li>
<li>I needed to support everyday things like a LAN file server (NAS);</li>
<li>I also wanted another Raspberry Pi to run my self-built kernel long term, hosting services that aren’t so critical.</li>
</ul>
<p>So the newer config files gradually <strong>added back some features</strong>, such as:</p>
<ul>
<li><strong>cgroup, iptables, fuse, overlayfs, wireguard, macvlan, NFS, Samba</strong> and other common kernel modules (to support Docker and file services)</li>
<li>Most of the new features are built as modules and loaded on demand, which further improves boot and runtime efficiency</li>
</ul>
<p>This step made the build a little slower, but the config is still very lean compared with the mainline default, and it fits what I actually use and test much better.</p>
<p><strong>The config files are hosted at:</strong> <a href="https://git.bktus.com/Linux/kernel-configs/">https://git.bktus.com/Linux/kernel-configs/</a></p>
<p>Some key config files, in timeline order:</p>
<ul>
<li>eric-rpi-v1_defconfig</li>
<li>eric-vanilla-c-6.13_defconfig</li>
<li>eric-vanilla-c-6.14_defconfig</li>
<li>eric-vanilla-c-6.15_defconfig</li>
<li>eric-vanilla-v1_defconfig</li>
<li>eric-vanilla-v2_defconfig</li>
<li>eric-vanilla-v3_defconfig</li>
</ul>
<p>From these config files, you can see how I went from extreme trimming to gradually adding things back as I debugged and tuned.</p>
<h2 id="advantages-and-use-cases">Advantages and Use Cases</h2>
<h3 id="close-to-mainline-quick-to-try-and-test-new-features">Close to Mainline, Quick to Try and Test New Features</h3>
<p>The biggest benefit of keeping the patches minimal is that you can take the latest patches from the linux-next branch, or even new features from Linus’s merge window, and run them on the RPi5 at almost “zero cost”. Whether it’s a new scheduler, new file system features, improvements to the virtualization and security subsystems (like KVM, eBPF and LSM), or Zstd compression and new memory management mechanisms, as soon as something changes in mainline, you can <strong>test it on real hardware right away, check its compatibility and performance, and report problems and suggestions to the upstream ARM64 community in time</strong>.</p>
<p>More importantly, <strong>by running a “vanilla” kernel, you can see and understand how the mainline kernel itself is implemented, as faithfully as possible</strong>. You avoid the “side effects” introduced by the Raspberry Pi vendor patches, and you don’t have to worry about mistaking a problem in a vendor patch for a mainline bug.</p>
<p>I maintain the patches with merge rather than rebase, so the repository history is cleaner and the differences between mainline and my local changes are clear at a glance. Almost every mainline merge goes through “instantly”, so you can always keep up with the latest kernel development. For example, you can pull linux-next for testing at any time, or benchmark new kernel features, and it really feels like you’re taking part.</p>
<p>This approach is really valuable for kernel developers, upstream enthusiasts and even “hardcore” hackers. The bugs you run into and the patches you submit are real problems in a mainline environment, <strong>so you can take them straight to the kernel community.</strong></p>
<h3 id="more-efficient-maintenance-and-development">More Efficient Maintenance and Development</h3>
<ul>
<li><strong>Very few patches, very few conflicts</strong>: the repository only covers the hardware support the RPi5 really needs. For each upstream merge, you only have to watch drivers/net/arm/…, drivers/usb/…, drivers/bus/… and arch/arm64/boot/dts/broadcom/…, so conflicts are very limited and there’s little maintenance work.</li>
<li><strong>Debugging and rolling back are really easy</strong>: if mainline introduces a regression or something odd, you only need to check a handful of patches to roll back or adjust. There’s no need to look for a needle in a haystack across thousands of lines of vendor patches.</li>
<li><strong>Easier to contribute upstream</strong>: your driver patches and bug fixes are very “clean”. You can turn them straight into patches and send them to the kernel mailing list, where they’re easy for the community to review and won’t get stuck because of interference from vendor patches.</li>
</ul>
<h3 id="an-ideal-platform-for-development-testing-and-learning">An Ideal Platform for Development, Testing and Learning</h3>
<ul>
<li><strong>Exploring and testing new features</strong>: the RPi5 is a low-cost, mainstream ARM64 development board, so you can easily test linux-next or any mainline branch on it and give test results and feedback to the upstream community.</li>
<li><strong>Performance tuning and driver testing</strong>: you can run performance benchmarks, optimize drivers, do power experiments and so on, and see how new mainline features really perform on ARM64.</li>
<li><strong>A teaching and self-study environment</strong>: for people learning Linux drivers, device trees and HAL, a minimal environment cuts down “environmental noise” as much as possible, so they can focus on understanding the mainline driver framework and hardware abstraction.</li>
</ul>
<p>The official repository has a clear advantage in driver coverage and vendor support, and it suits users who want an “out-of-the-box, full multimedia experience”. My mainline repository focuses on “staying close to upstream, simple maintenance and easy development”. If you just want to watch videos, play games or run a GUI desktop on the Raspberry Pi 5, the official repository is probably more convenient. But for kernel engineers, developers and people who love to experiment, a “lightweight mainline repository” lets you focus more on the kernel itself.</p>
<h2 id="how-to-use-and-contribute">How to Use and Contribute</h2>
<p>In theory, the Raspberry Pi 5 is a lot faster than the Raspberry Pi 4, and it can compile its own kernel just fine. The experience isn’t bad either (with more than 4GB of RAM). But if you need to do kernel or kernel module development and turn on things like code indexing, it doesn’t have enough power.</p>
<p>I also tried adding the official SSD Kit (512GB). That solved the I/O throughput problem, but in the end I still hit a CPU bottleneck, because it only has 4 cores. So if you just need a working, up-to-date kernel and don’t compile often, you can build and install it on the Raspberry Pi itself. But if you need to go further with development and debugging, I still strongly recommend cross-compilation. That means compiling the kernel and kernel modules on a PC or workstation (usually x86), and then copying them to the Raspberry Pi over the network (or with the SD card). For example, my workstation has 12 cores, and builds take several times less time on it (compared with the RPi5’s 4 cores). So below, I’ll use my own cross-compilation setup as the example.</p>
<p>Before that, you need to install an ARM64 cross-compilation environment. I suggest installing ccache to cut the time of repeated builds even more. If you don’t want to use ccache, remove the ccache part from all the commands in the following sections.</p>
<pre tabindex="0"><code>sudo apt install bc bison flex libssl-dev make libc6-dev libncurses5-dev
sudo apt install crossbuild-essential-arm64
sudo apt install ccache
</code></pre><h3 id="clone-the-repositories-and-check-out-the-latest-branch">Clone the Repositories and Check Out the Latest Branch</h3>
<p>My repository is on my own private Git server, but you can use my mirror on GitHub. I regularly track Linus’s mainline repository and update the main branch.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>git clone https://git.bktus.com/Linux/kernel-configs.git
</span></span><span style="display:flex;"><span>git clone https://github.com/saturneric/linux <span style="color:#f92672">(</span>recommand<span style="color:#f92672">)</span> or https://git.bktus.com/linux/kernel.git
</span></span><span style="display:flex;"><span>cd linux
</span></span><span style="display:flex;"><span>git checkout main
</span></span></code></pre></div><p>Note: the first command clones the <a href="https://git.bktus.com/Linux/kernel-configs/">kernel config repository</a> I mentioned earlier. So <strong>don’t use the config files provided by the Raspberry Pi team in the later steps</strong> (they turn on lots of features), because that will most likely give you a compiled kernel that doesn’t boot.</p>
<p>Of the two, <a href="https://github.com/saturneric/linux">https://github.com/saturneric/linux</a> is on GitHub and should be the fastest in theory, so if you don’t have any special needs, try to clone from there. <a href="https://git.bktus.com/linux/kernel.git">https://git.bktus.com/linux/kernel.git</a> is the repository hosted on my own server, and the GitHub repository above is a mirror of it.</p>
<h3 id="basic-configuration">Basic Configuration</h3>
<p>You need to pick a config file from the kernel-configs repository that fits the current version, rename it to <code>.config</code> and copy it into the root of the kernel repository.</p>
<pre tabindex="0"><code>cp ../kernel-configs/eric-vanilla-c-6.16_defconfig .config
</code></pre><p>You can trim the kernel further on top of my config, or add features.</p>
<pre tabindex="0"><code>make ARCH=arm64 CROSS_COMPILE=&#34;ccache aarch64-linux-gnu-&#34; menuconfig
</code></pre><h3 id="build">Build</h3>
<p>Once the cross-compilation environment is ready, you can build right away. How long it takes depends on your config and your machine. For me, it takes about 10 minutes.</p>
<pre tabindex="0"><code>make ARCH=arm64 CROSS_COMPILE=&#34;ccache aarch64-linux-gnu-&#34; -j$(nproc)
</code></pre><h3 id="copy-the-kernel-and-kernel-modules-to-the-raspberry-pi">Copy the Kernel and Kernel Modules to the Raspberry Pi</h3>
<p>You can use the script I provide to quickly transfer the kernel image and kernel modules you just built to the home directory of the default user on the Raspberry Pi. I strongly recommend not giving the kernel image and device tree the same names as the Raspberry Pi 5’s default kernel. Use names like <code>kernel-vanilla.c.img</code> and <code>bcm2712d0-rpi-5-b.vanilla.c.dtb</code> instead, so even if something goes wrong, we can quickly switch back to the original working kernel or device tree through the config file. Also, the rsync in the script needs passwordless SSH, so I suggest setting up your public key beforehand.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#75715e">#!/bin/bash
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># ====== Configurable parameters (change them to fit your setup) ======</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Local directory for the kernel build artifacts</span>
</span></span><span style="display:flex;"><span>LOCAL_KERNEL_ARTIFACTS<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;./kernel-artifacts&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># SSH connection info for the target Raspberry Pi (e.g. pi@192.168.1.100)</span>
</span></span><span style="display:flex;"><span>REMOTE_HOST<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;pi@192.168.1.100&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Target directory on the Raspberry Pi (an absolute path is recommended, e.g. /home/pi/kernel)</span>
</span></span><span style="display:flex;"><span>REMOTE_KERNEL_ARTIFACTS<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;/home/pi/kernel&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Target kernel file name (customizable, default kernel_2712.img)</span>
</span></span><span style="display:flex;"><span>KERNEL_IMG<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;kernel-vanilla.c.img&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Device tree file name (customizable)</span>
</span></span><span style="display:flex;"><span>DTB_FILE<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;bcm2712d0-rpi-5-b.vanilla.c.dtb&#34;</span>
</span></span><span style="display:flex;"><span>DTB_SOURCE<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;arch/arm64/boot/dts/broadcom/bcm2712d0-rpi-5-b.dtb&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># ==============================================</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>set -e
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; Cleaning and preparing the local directory &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span>rm -rf <span style="color:#e6db74">&#34;</span>$LOCAL_KERNEL_ARTIFACTS<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>mkdir -p <span style="color:#e6db74">&#34;</span>$LOCAL_KERNEL_ARTIFACTS<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; Installing kernel modules into the local output directory &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span>make ARCH<span style="color:#f92672">=</span>arm64 CROSS_COMPILE<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;aarch64-linux-gnu-&#34;</span> INSTALL_MOD_PATH<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span>$LOCAL_KERNEL_ARTIFACTS<span style="color:#e6db74">&#34;</span> modules_install
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; Copying the kernel image and device tree &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span>cp -v -f arch/arm64/boot/Image.gz         <span style="color:#e6db74">&#34;</span>$LOCAL_KERNEL_ARTIFACTS<span style="color:#e6db74">/</span>$KERNEL_IMG<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>cp -v -f <span style="color:#e6db74">&#34;</span>$DTB_SOURCE<span style="color:#e6db74">&#34;</span>                    <span style="color:#e6db74">&#34;</span>$LOCAL_KERNEL_ARTIFACTS<span style="color:#e6db74">/</span>$DTB_FILE<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; Uploading the kernel artifacts to the Raspberry Pi &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span>ssh <span style="color:#e6db74">&#34;</span>$REMOTE_HOST<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#34;sudo mkdir -p </span>$REMOTE_KERNEL_ARTIFACTS<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>rsync -avz --progress --delete <span style="color:#e6db74">&#34;</span>$LOCAL_KERNEL_ARTIFACTS<span style="color:#e6db74">/&#34;</span> <span style="color:#e6db74">&#34;</span>$REMOTE_HOST<span style="color:#e6db74">:</span>$REMOTE_KERNEL_ARTIFACTS<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; Upload complete. You can now log in to the Raspberry Pi and configure/switch the boot kernel &lt;&lt;&lt;&#34;</span>
</span></span></code></pre></div><p>So after the script finishes, all the files you’re about to install are already on the Raspberry Pi. But note that they’ve only been copied over. The new kernel isn’t actually in use yet.</p>
<h3 id="install-the-kernel-and-kernel-modules-on-the-raspberry-pi">Install the Kernel and Kernel Modules on the Raspberry Pi</h3>
<p>With the script below, you can easily install the files you just copied to the Raspberry Pi (device tree, kernel and kernel modules) to where they should go. <strong>Note: this script touches several critical system directories, so double-check the parameters to avoid deleting or overwriting important files by mistake.</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#75715e">#!/bin/bash
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#75715e"># ==============================================</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># RPi5 mainline kernel local deployment/flashing script</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Installs the kernel image, device tree and kernel modules into /boot and /lib/modules</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Make sure you run it on the Raspberry Pi with sudo privileges</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># ==============================================</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>set -e
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># ====== Configurable parameters (change them to fit your setup) ======</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Directory holding the uploaded artifacts (the ones just synced over with rsync)</span>
</span></span><span style="display:flex;"><span>REMOTE_KERNEL_ARTIFACTS<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;/home/pi/kernel&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Kernel image name (keep it the same as in the upload script)</span>
</span></span><span style="display:flex;"><span>KERNEL_IMG<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;kernel-vanilla.c.img&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Device tree file name</span>
</span></span><span style="display:flex;"><span>DTB_FILE<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;bcm2712d0-rpi-5-b.vanilla.c.dtb&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Location of the kernel image and device tree</span>
</span></span><span style="display:flex;"><span>BOOT_FIRMWARE_PATH<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;/boot/firmware&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Actual install directory for kernel modules (target)</span>
</span></span><span style="display:flex;"><span>LIB_MODULES_PATH<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;/lib/modules&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># ==============================================</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; 1. Installing the kernel image and device tree &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span>sudo cp -v <span style="color:#e6db74">&#34;</span>$REMOTE_KERNEL_ARTIFACTS<span style="color:#e6db74">/</span>$KERNEL_IMG<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#34;</span>$BOOT_FIRMWARE_PATH<span style="color:#e6db74">/</span>$KERNEL_IMG<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>sudo cp -v <span style="color:#e6db74">&#34;</span>$REMOTE_KERNEL_ARTIFACTS<span style="color:#e6db74">/</span>$DTB_FILE<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#34;</span>$BOOT_FIRMWARE_PATH<span style="color:#e6db74">/</span>$DTB_FILE<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; 2. Setting permissions for the image and DTB &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span>sudo chown root:root <span style="color:#e6db74">&#34;</span>$BOOT_FIRMWARE_PATH<span style="color:#e6db74">/</span>$KERNEL_IMG<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#34;</span>$BOOT_FIRMWARE_PATH<span style="color:#e6db74">/</span>$DTB_FILE<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>sudo chmod <span style="color:#ae81ff">644</span> <span style="color:#e6db74">&#34;</span>$BOOT_FIRMWARE_PATH<span style="color:#e6db74">/</span>$KERNEL_IMG<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#34;</span>$BOOT_FIRMWARE_PATH<span style="color:#e6db74">/</span>$DTB_FILE<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; 3. Installing kernel modules &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Modules with the same version number are overwritten automatically</span>
</span></span><span style="display:flex;"><span>sudo rsync -av --progress <span style="color:#e6db74">&#34;</span>$REMOTE_KERNEL_ARTIFACTS<span style="color:#e6db74">/lib/modules/&#34;</span> <span style="color:#e6db74">&#34;</span>$LIB_MODULES_PATH<span style="color:#e6db74">/&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; 4. Setting permissions for kernel modules &lt;&lt;&lt;&#34;</span>
</span></span><span style="display:flex;"><span>sudo chown -R root:root <span style="color:#e6db74">&#34;</span>$LIB_MODULES_PATH<span style="color:#e6db74">/&#34;</span>
</span></span><span style="display:flex;"><span>find <span style="color:#e6db74">&#34;</span>$LIB_MODULES_PATH<span style="color:#e6db74">/&#34;</span> -type d -exec sudo chmod <span style="color:#ae81ff">755</span> <span style="color:#f92672">{}</span> <span style="color:#ae81ff">\;</span>
</span></span><span style="display:flex;"><span>find <span style="color:#e6db74">&#34;</span>$LIB_MODULES_PATH<span style="color:#e6db74">/&#34;</span> -type f -exec sudo chmod <span style="color:#ae81ff">644</span> <span style="color:#f92672">{}</span> <span style="color:#ae81ff">\;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;&gt;&gt;&gt; Kernel and modules deployed. If needed, update /boot/config.txt to point to the new kernel &lt;&lt;&lt;&#34;</span>
</span></span></code></pre></div><h3 id="configure-the-raspberry-pi-firmware-to-boot-the-new-kernel">Configure the Raspberry Pi Firmware to Boot the New Kernel</h3>
<p>You need to edit <code>/boot/firmware/config.txt</code> so that the Raspberry Pi firmware boots the new kernel and passes it the matching device tree to use. Mainly, you change or add the two lines below.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-ini" data-lang="ini"><span style="display:flex;"><span><span style="color:#75715e"># DTB</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">device_tree</span><span style="color:#f92672">=</span><span style="color:#e6db74">bcm2712d0-rpi-5-b.vanilla.c.dtb</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Kernel</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">kernel</span><span style="color:#f92672">=</span><span style="color:#e6db74">kernel-vanilla.c.img</span>
</span></span></code></pre></div><h2 id="removing-the-raspberry-pis-dts-overlay-feature-and-avoiding-common-boot-problems">Removing the Raspberry Pi’s DTS Overlay Feature (and Avoiding Common Boot Problems)</h2>
<p>The Raspberry Pi team added a DTS overlay mechanism to their platform. When the main device tree is loaded, this design lets extra overlay fragments be stacked on top of it dynamically, depending on what the firmware or the user needs. That way, peripherals, parameters or features can be turned on or adjusted flexibly. For example, if you add a new module or feature to your Raspberry Pi, you only need to configure an overlay to activate it, which is pretty convenient.</p>
<p>But this mechanism is currently <strong>not supported by the mainline Linux kernel</strong>.</p>
<p>If overlay settings are enabled in <code>/boot/firmware/config.txt</code> and the overlay files don’t match the device tree (DTB) version of the mainline kernel, it’s very easy to end up with boot failures, devices that aren’t recognized and similar problems. For kernel developers who want a minimal, pure mainline kernel experience, this often causes more trouble than it saves.</p>
<h3 id="my-approach">My Approach</h3>
<p>For me personally, DTS overlays are very flexible, but the mainline kernel doesn’t support them natively, and they get in the way of kernel debugging and a clean environment. So I chose to <strong>remove the overlay feature completely</strong> and keep only the main device tree file. The direct side effect is that <strong>some hardware parameters (like CPU boot parameters) are normally injected through overlays, so just removing overlays will stop the Raspberry Pi from booting.</strong></p>
<p><strong>The fix</strong>: I got around this by writing the required CPU parameters directly into the main device tree file. Actually, the official device tree file bcm2712d0-rpi-5-b.dtb is made for the no-overlay case. As long as you use it as the main device tree, the system boots normally without any overlay.</p>
<h3 id="how-to-remove-the-overlay-config-step-by-step">How to Remove the Overlay Config (Step by Step)</h3>
<p>Edit /boot/firmware/config.txt, make sure only device_tree is set, and remove or comment out all overlay settings (such as the dtoverlay= lines). Here’s an example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-ini" data-lang="ini"><span style="display:flex;"><span><span style="color:#75715e"># Specify the main device tree file (bcm2712d0-rpi-5-b.vanilla.c.dtb is recommended)</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">device_tree</span><span style="color:#f92672">=</span><span style="color:#e6db74">bcm2712d0-rpi-5-b.vanilla.c.dtb</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Don&#39;t load any overlay</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#dtoverlay=xxx      # Comment out or delete all of these</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># It&#39;s best to also turn off the other overlay auto-detection features</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">camera_auto_detect</span><span style="color:#f92672">=</span><span style="color:#e6db74">0</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">display_auto_detect</span><span style="color:#f92672">=</span><span style="color:#e6db74">0</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Kernel image to boot</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">kernel</span><span style="color:#f92672">=</span><span style="color:#e6db74">kernel-vanilla.c.img</span>
</span></span></code></pre></div><h3 id="check-other-overlay-related-settings">Check Other Overlay-Related Settings</h3>
<ul>
<li>I recommend turning off camera_auto_detect and display_auto_detect.</li>
<li>Make sure no dtoverlay= entry is enabled.</li>
</ul>
<h3 id="reboot-the-raspberry-pi">Reboot the Raspberry Pi</h3>
<p>Once the changes above are in place, reboot the device. The system will load the main device tree file directly and <strong>won’t apply any overlay anymore</strong>, which keeps the mainline kernel as clean and controllable as possible. If everything works fine afterward, you can remove the <code>/boot/firmware/overlays</code> directory to save space. This also makes sure these overlays can never be loaded at boot again, no matter what, so they can’t pollute the environment.</p>
<h3 id="recommended-minimal-configtxt-template">Recommended Minimal config.txt Template</h3>
<pre tabindex="0"><code># Only specify the main device tree and the mainline kernel image
device_tree=bcm2712d0-rpi-5-b.vanilla.c.dtb
kernel=kernel-vanilla.c.img

# Turn off overlay auto-detection
camera_auto_detect=0
display_auto_detect=0

# Other common parameters can stay
disable_fw_kms_setup=1
arm_64bit=1
disable_overscan=1
arm_boost=1
</code></pre><h2 id="submitting-patches-or-issues">Submitting Patches or Issues</h2>
<ul>
<li>If you run into any problems while following this post, or find a step that keeps the process from working, feel free to let me know anytime.</li>
<li>If you fix a compatibility problem or find a bug while using it, you’re also welcome to open an issue directly, so we can discuss it with other users and improve the project together.</li>
<li>If you add hardware support, make performance improvements or port new mainline patches on top of this repository, I suggest generating standard patches with scripts/format-patch.sh and then sending them straight to the upstream Linux kernel mailing list, so the community can take in the improvements.</li>
</ul>
<p>GitHub repository: <a href="https://github.com/saturneric/linux">https://github.com/saturneric/linux</a></p>
<h2 id="long-term-maintenance-notes">Long-Term Maintenance Notes</h2>
<p>Neither this repository nor this post is a one-off “pop-up project”. Since I’ll be studying the Linux kernel and operating system theory for a long time, I’ll keep following every major mainline kernel release (including the rc1 … rcX stages) and merge the key changes and adaptations into the main branch in time. Before each mainline sync, I first build it on my workstation (Debian Testing) and then test it on a real Raspberry Pi 5, to make sure the repository always works and the process runs smoothly.</p>
<p>If you’re also doing mainline kernel development on a Raspberry Pi, or you run into compatibility or adaptation problems, feel free to leave a comment. I also hope you’ll join the mainline kernel community and help the ARM64 platform keep getting better.</p>
]]></content:encoded></item></channel></rss>