Coming soon 18:18 Recorded at Open Source Summit Europe, Prague

Why does Linux kernel development still run on email?

How the kernel is built by email

Konstantin Ryabitsev of the Linux Foundation on why kernel development still runs on email, b4, distributed trust, and the cost of AI-written junk patches.

Not filmed yet.

Konstantin Ryabitsev has run the infrastructure behind kernel.org at the Linux Foundation since 2011, and wrote b4, the tool many kernel maintainers use to work with patches sent by email. Before that he worked in open source at Duke Physics, where early Fedora releases were built.

Follow the show and the episode arrives the morning it's published.

In brief

  1. Kernel developers value a workflow that no single company controls; many maintainers oppose moving to a proprietary platform such as GitHub.
  2. Git exists because of BitKeeper: when its access terms changed, Linus Torvalds wrote Git.
  3. After the 2011 break-in, kernel.org was down for almost two months, and kernel development carried on without it.
  4. Email is still the only distributed medium they have; replacing it would need a flag day that a community this size cannot manage.
  5. b4 grabs patches from mailing lists for maintainers, sends series in the format maintainers expect, and is gaining review and tracking features.
  6. Trust is built over years with PGP keys; a keyring of about 600 maintainer keys lets Linus verify where a pull request came from.
  7. Work on lore.kernel.org aims at local copies and failover mirrors, so developers can keep working if it is knocked offline.
  8. AI-written patches are like work from interns: sometimes great, sometimes junk, always in need of review, and junk adds to maintainers' decision fatigue.

Chapters

  1. 0:00 Kernel development by email
  2. 2:10 From BitKeeper to Git
  3. 3:13 The 2011 kernel.org break-in
  4. 4:16 No flag day: email stays
  5. 5:18 b4, for maintainers
  6. 7:32 Distributed trust
  7. 9:48 Advice for newcomers
  8. 11:53 AI patches, like interns
  9. 14:04 The cost of junk patches

Why this conversation

Every company that ships Linux depends on a development process that still runs on email and patches. To anyone used to pull requests in a web interface that looks old-fashioned. It is a deliberate choice, and the person who looks after the infrastructure behind it can explain why. At Open Source Summit Europe in Prague the host sat down with Konstantin Ryabitsev, who has run kernel.org’s infrastructure at the Linux Foundation since 2011 and wrote b4, the tool many kernel maintainers use to handle patches.

The host also published the conversation as a written interview. The story of the week it was filmed is in Open Source Summit Europe 2026: who we met.

Why email, still

The kernel community has worked this way since the early nineties. The people in it are efficient with their tools, and they like that the process is decentralised and that no single company controls it. A common question is why the kernel does not just move to GitHub. Many maintainers oppose it: GitHub is a commercial company, and although it runs Git, everything beneath the push is proprietary. It is friendly to developers now. Nobody knows whether it will stay that way.

There is history behind that caution. Before the early 2000s the kernel had no version control: Linus Torvalds received patches and published tarballs. CVS and Subversion needed a central server, which the maintainers did not like. BitKeeper came closer and had good ideas, but it was a proprietary service people came to depend on. When its owners changed how maintainers could access it, the community decided to stop using it, and Linus came back with Git.

The second lesson came in 2011, before his time, when the kernel infrastructure was broken into. kernel.org was down for almost two months while it was rebuilt from scratch, and kernel development continued anyway. He wants to keep it that way, because kernel.org can be brought down by an adversary, by a mistake, or by AI crawlers consuming all the bandwidth they can. Email is still the only platform that gives them that kind of distributed environment. Replacing it would take a flag day, when everyone stops using email and starts using something else, and with a user base this large that is very hard to do. The main kernel mailing list alone sees on the order of 30,000 patches a month, around a thousand a day, with ebbs and flows around the merge window.

b4 and the tools around it

So his approach is to make email and patches easier to work with. b4 started as a way for maintainers to grab patches from the mailing lists and prepare them to apply to their Git tree. It grew to send patches too, preparing a series and formatting it the way maintainers like to see it, which helps newcomers. The latest work lets maintainers review series, send quick replies, integrate agents if they want to, and track several series and versions at once. None of it is imposed. Greg Kroah-Hartman, for example, has an efficient workflow of his own, and Konstantin has no wish to tell him to change it. b4 is for those who do not yet have one.

He is also working on lore.kernel.org, where all the patches and mailing lists are aggregated and searchable. It is a central platform, so the aim is to make it easy to keep local copies, say the last three months, kept up to date, and to add failover mirrors, including a hidden service known to maintainers, so work can continue if the main one is knocked out.

Trust without a centre

How do you know a patch comes from who it claims? “Well, we don’t really,” he says. There is patch attestation, using keys tied to email, but trust is mostly built over time: the same person sending patches with the same key for five years, keys verified at conferences, a lot of PGP. Distributed trust is not a solved problem. It is well established between major maintainers, though. Pull requests to Linus are signed, and a keyring of about 600 keys, which he maintains, lets Linus verify them with nothing more than his own keyring.

AI patches and maintainer fatigue

For newcomers he does not claim to know how AI will play out. The whole AI system is heavily invested in and subsidised, so what is cheap and easy today may not be tomorrow. Do not abandon your tech skills, and keep hacking if it is fun.

He uses AI-generated code himself. Working with agents, he says, is like working with contractors or interns: sometimes great, sometimes not, and you still need to judge it. He would not put an agent’s code into any environment without review, “because the agent doesn’t care.”

The problem for the kernel is volume. People submit AI-written patches to get “Linux kernel contributor” on a CV: large, well-formatted changes across five or ten subsystems, never tested, from authors who will not be around when problems come up. Each one can cost several maintainers half an hour before it gets a NACK. “Maintainers is the hardest part to scale,” he says, and constant critical decisions, which he calls decision fatigue, are why they burn out. The topic was due at the Maintainers Summit the next day. He does not want to ban AI patches, as some projects have done for scalability reasons; a contributor who reviewed the code and works with maintainers is welcome. Finding the middle ground is still open.

Getting involved

Ideas for tools that should exist, or contributions to existing ones, go to the tools mailing list, tools@kernel.org.

Mentioned in this episode

About Konstantin Ryabitsev

Konstantin Ryabitsev has run the infrastructure behind kernel.org at the Linux Foundation since 2011, and wrote b4, the tool many kernel maintainers use to work with patches sent by email. Before that he worked in open source at Duke Physics, where early Fedora releases were built.

Running this in production? helps platform teams take it from prototype to reliable production. See how he works

Built something the rest of us run on? Being a guest is free. Take the other chair

More episodes

All episodes →
  1. Episode 42 · Coming

    Arnaldo Carvalho de Melo: Profiling data for the compiler

  2. Episode 44 · Coming

    Daniel Schäfer: A laptop you upgrade, not replace

  3. Episode 14 · Coming

    Greg Kroah-Hartman: Inside the stable kernel

  4. Episode 54 · Coming

    Alex Scheel: OpenBao 2.7: post-quantum and HSM keys

Every episode, plus the notes that didn't fit in ten minutes.

One email a week: the episode, three takeaways, and what to read or try next. No sponsor blasts.

    You can unsubscribe with one click.