Twenty years of pahole

Why this conversation

Anyone who runs BPF-based tools in production depends on a piece of software most of them have never heard of. pahole started as a way to look inside data structures and pack them better. Twenty years on, every kernel build runs it, because BPF needs the type information it produces. Arnaldo Carvalho de Melo of Red Hat, who maintains pahole and the perf tooling, recorded with the host on the Wednesday afternoon of Open Source Summit Europe in Prague, and was still wearing the lavalier mic when the photo in Open Source Summit Europe 2026: who we met was taken.

This is the first part of the conversation; the second part has a page of its own.

Twenty years of kernel work from Brazil

Arnaldo has always worked from Brazil. Before Red Hat he had a company there that sold Linux and made a Linux distribution, and a number of kernel developers came out of it. He names some of the people around that time: Rik van Riel, who worked on memory management in the 2.4 to 2.5 kernel era, Marcelo Tosatti, a young Brazilian who maintained the kernel for a while, and Harald Welte from Germany, who worked on GPL enforcement and the iptables subsystem.

At Red Hat he started in networking, reorganising the structures behind the kernel’s network protocols: TCP, IPv4, IPv6. He then joined the team with Ingo Molnar, Thomas Gleixner, Peter Zijlstra and Steven Rostedt, working on productising real-time. Along the way Ingo asked whether he would like to contribute to perf’s user space, and he has maintained it since 2009 or thereabouts.

How pahole became part of BPF

The BPF connection goes back to 2015. Alexei and Daniel, the main BPF developers at the time, approached him after David Miller told them his tooling already handled DWARF and CTF, the compact type format used by DTrace on Solaris. They looked at the code, saw a CTF loader and an encoder that produced CTF from DWARF, and designed BTF around that.

BTF is what makes compile once, run everywhere possible. A BPF program built against one kernel may run on another where the layout of a data structure it uses has changed, so its accesses have to be relocated. Because of that, every kernel that is built now uses pahole to convert DWARF into BTF. The compilers, Clang and GCC, only generate BTF when they build BPF code itself, not for x86-64, ARM or any other native target. The compiler is the right place for it eventually, Arnaldo says, but BPF evolves fast, and waiting for the compilers would slow it down.

The original purpose has not gone away. pahole was written to reduce memory use and make better use of the cache hierarchy. Saving 10, 12 or 32 bytes sounds small, but “it’s 32 bytes per connection,” he says: a server can hold thousands of connections, and a first-level cache can be as small as 64 kilobytes. Using that scarce space well is what lets a machine serve more connections and process packets faster.

Profiling data, not code

Two pieces of current work stand out. The first is extending BTF to Rust. Rust is now used for components inside the kernel, and people want the same tracing they have for C, with tools such as bpftrace. But Rust has concepts BTF was never designed to express, such as traits and enums that carry functions.

The second is in perf. Instead of profiling code, perf can now profile data structures. It samples loads and stores to find false sharing, where one CPU writes to a cache line that another is reading, forcing the cache to be refreshed from memory. NUMA machines make this worse, and so do chips built from chiplets that each share their own cache. The same sampling also finds fields that are used together: looking at accesses to a structure such as struct tcp_sock across a short window, perf can show that two fields on separate cache lines should sit on one, so one cache line is fetched instead of two.

This matters most where large amounts of data move, he says, such as AI inference across 200-gigabit network ports, where it shows up as more tokens per second. The host described his own experience with RDMA, NFS and GPUs reading directly from storage; Arnaldo’s answer was that every piece of hardware has its own characteristics and its own counters, and that you need tools like perf to observe it before acting on the numbers. In those extreme cases every 1 or 2 per cent counts, and the machines are too expensive not to use fully.

Keeping the gains

The last part of the conversation is about why measurement has to continue. A structure tuned with these techniques can lose its performance when someone adds a field in the wrong place the next day. “It’s difficult to optimize something, it’s easy to pessimize,” he says. That is the case for continuous profiling in production: sampling all the time, increasingly with inference to analyse the data, so that a regression or a change in the workload, such as prompts getting bigger, can be connected to its cause.