Recently I've been trying to use BCC for some
projects that involve kernel tracing using kprobes and eBPF. The way BCC works
is that you write your BPF programs in C, and then BCC translates your C source
code to a BPF program using LLVM. To write any useful program this way, BCC
needs a copy of the kernel headers for the kernel the program is targeting. This
is required because most BPF programs using kprobes need to know the exact
struct definitions for internal kernel data structures, and these structs aren't
part of the kernel uapi (i.e. the headers
typically installed to
If you're using an RPM Linux distro these headers are provided by the
kernel-devel package, and on Debian/Ubuntu the headers are provided by a
linux-headers-$(uname -r). But if you're in a different
environment (in my case Buildroot) it's not really
clear how to get these headers, as they're not actually installed by any make
target in the kernel. The
headers_install make target you might expect to
generate these headers actually just installs the uapi headers, which are not
the headers you need. Likewise, the Buildroot
kernel-headers package installs
these uapi headers and not the headers needed by BCC.
I wasted more time than I should have trying to figure out how this is supposed to work, as I couldn't find it documented anywhere. Hopefully this post saves someone else a lot of time trying to figure out what to do.
The secret can be found in the kernel's
shell script, which is a helper script provided for building deb packages.
Specifically, there's a function in this script called
It contains some code to generate a file named
debian/hdrsrcfiles like this:
( cd $srctree find . arch/$SRCARCH -maxdepth 1 -name Makefile\* find include scripts -type f -o -type l find arch/$SRCARCH -name module.lds -o -name Kbuild.platforms -o -name Platform find $(find arch/$SRCARCH -name include -o -name scripts -type d) -type f ) > debian/hdrsrcfiles
If you build an archive with all of these files you'll have everything you need.
This actually includes some extra stuff like Makefiles that aren't needed by BCC
(but are useful for doing things like writing kernel modules). If you're just
trying to use BCC with kprobes you only need the
.h files. BCC also requires
generated headers that you'll find in the directories
arch/x86/include/generated (assuming x86 arch), so make sure you have these.
I'm not sure which kernel make target builds these, but if you just do a regular
kernel build they'll be there.
You can instruct BCC where to find the headers by providing the environment
BCC_KERNEL_SOURCE with the path that contains your header files. This
should be a directory that contains two directories, one named