Generating Kernel Headers For BPF Kprobes

Recently I've been trying to use BCC for some projects that involve kernel tracing using kprobes and eBPF. The way BCC works is that you write your BPF programs in C, and then BCC translates your C source code to a BPF program using LLVM. To write any useful program this way, BCC needs a copy of the kernel headers for the kernel the program is targeting. This is required because most BPF programs using kprobes need to know the exact struct definitions for internal kernel data structures, and these structs aren't part of the kernel uapi (i.e. the headers typically installed to /usr/include/linux).

If you're using an RPM Linux distro these headers are provided by the kernel-devel package, and on Debian/Ubuntu the headers are provided by a package named linux-headers-$(uname -r). But if you're in a different environment (in my case Buildroot) it's not really clear how to get these headers, as they're not actually installed by any make target in the kernel. The headers_install make target you might expect to generate these headers actually just installs the uapi headers, which are not the headers you need. Likewise, the Buildroot kernel-headers package installs these uapi headers and not the headers needed by BCC.

I wasted more time than I should have trying to figure out how this is supposed to work, as I couldn't find it documented anywhere. Hopefully this post saves someone else a lot of time trying to figure out what to do.

The secret can be found in the kernel's scripts/package/builddeb shell script, which is a helper script provided for building deb packages. Specifically, there's a function in this script called deploy_kernel_headers. It contains some code to generate a file named debian/hdrsrcfiles like this:

(
    cd $srctree
    find . arch/$SRCARCH -maxdepth 1 -name Makefile\*
    find include scripts -type f -o -type l
    find arch/$SRCARCH -name module.lds -o -name Kbuild.platforms -o -name Platform
    find $(find arch/$SRCARCH -name include -o -name scripts -type d) -type f
) > debian/hdrsrcfiles

If you build an archive with all of these files you'll have everything you need. This actually includes some extra stuff like Makefiles that aren't needed by BCC (but are useful for doing things like writing kernel modules). If you're just trying to use BCC with kprobes you only need the .h files. BCC also requires generated headers that you'll find in the directories include/generated and arch/x86/include/generated (assuming x86 arch), so make sure you have these. I'm not sure which kernel make target builds these, but if you just do a regular kernel build they'll be there.

You can instruct BCC where to find the headers by providing the environment variable BCC_KERNEL_SOURCE with the path that contains your header files. This should be a directory that contains two directories, one named include and another named arch.