Ptrace (continued)

I made some significant improvements to my userspace ptrace prober today. The new way it works is pretty interesting.

There are essentially two major changes. The first is that instead of single stepping through the program, I am now using the x86 TRAP facility (a.k.a. INT 3) to more efficiently pause at the right point. This is the same mechanism that a debugger like GDB uses to implement break points. The idea is pretty simple. You insert the value 0xcc at the instruction that you want to break on. Then when the program hits that instruction the kernel will deliver SIGTRAP to the process. From the tracer process you can use wait(), waitpid(), or waitid() to wait until the child is delivered SIGTRAP. At that point you can do whatever you need to do.

The reason that using a trap is more efficient is because in single stepping mode the process gets interrupted after literally every x86 instruction it executes, which imposes a significant overhead as you might imagine. When using a trap there is essentially no overhead, the only overhead is when you actually process the trap.

The second problem that I had was that I was encoding the CALL instruction to fprintf, and the format string data, directly into the code segment of the executable at the current instruction pointer. This actually works fine, but there's one caveat. If you had run my tracer exactly when the tracee was in the middle of executing fprintf() (or a routine called by fprintf()) then the fprintf() code would actually be corrupted and the program would crash.

Here's how the new tracer works:

This new tracer is both faster and more correct than the previous implementation. As before, you can find the code on GitHub at eklitzke/ptrace-call-userspace.