GNOME Terminal Server

If you are using a Linux desktop then there's a pretty decent chance that the terminal emulator you're using is gnome-terminal, the GNOME terminal emulator.

I have a bad habit of opening way too many GNOME terminal emulator windows on my deskop. At the end of the day I'll easily hit a dozen open gnome-terminal windows. As a consequence of this I have noticed a few really interesting things about how gnome-terminal works.

First: while there is a command called gnome-terminal, you won't see any instances of gnome-terminal in your process listing. Instead you'll see a single process called gnome-terminal-server that is bound to the same TTY as all your other X11 processes.

Second: the gnome-terminal-server instance is multi-threaded, but not thread-per-window. On my machine it always uses four threads even though I may have more (or fewer) than four terminal instances.

Third: the gnome-terminal-server process can get "blocked" and therefore prevent terminal input or output on other terminals!

I will explain what I mean. I have previously written about how I use autofs to develop with NFS and virtual machines. Lately I have had an issue where the first time I try to access the autofs system (which has fully booted at that point) sometimes the initial I/O commands in the mounted directory get "blocked". What I mean by "blocked" is that any I/O over the autofs directory blocks for a few seconds when I first try to access it---and then I'll see all the processes finishing their I/O at once. I have written another article about the "uninterruptible sleep" state on Unix, and in that article I specifically discussed how historically that program state is associated with NFS.

I haven't debugged the issue yet. It's probably something I can work out by looking through the appropriate log files on the NFS server (or enabling the right verbosity of something to get the logging to happen). Since it only happens once per boot and blocks for a brief time (usually less than a minute) I haven't devoted my time to tracking down the root issue yet. My theory is that it's related to my PS1 prompt variable having overly complex git state tracking---I run git status at least once every time a prompt needs to be rendered, and the git status command has to do a ton of networked I/O operation to read the .git/ directory and its subdirectories, etc. This is just a hunch though.

Today I rebooted my machine and in the first gnome-terminal emulator instance that I opened I tried to cd into a git directoy within the autofs system. This caused my system to pause, as I have previously observed.

What was new today was that when I launched new terminals they were all blocked as well! I tried to open multiple terminals to diagnose the issue as it was happening, but since I didn't have any terminals open yet (this being the first terminal emulator process) I wasn't able to actually really debug anything with gdb or strace. After a minute or so the initial window's cd completed and immediately all of the other windows were rendered as well.

What does this mean?

Well, first of all it's interesting that gnome-terminal was designed like this, i.e. without a thread/process per window. It's easy to see why someone might have tried to make this "optimization": it reduces memory footprint to multiplex multiple terminals at once.

It also means that internally within gnome-terminal there's some kind of mutex that it's possible to hold while entering the uinterruptible I/O state. This is a bad practice: because certain I/O operations can block for a long time (potentially forever), one should not hold any mutexes while doing an uninterruptible I/O operation.

Postscript

My blog has quieted down recently---it's been nearly a month since my last post, which is one of the longest periods since I restarted blogging recently.

Do not fear, intrepid reader! I have not given up blogging.

Rather, I want to start producing higher quality posts in general. That means fewer ad-hoc posts like this one, and more posts that are systemic narratives of interesting systems.

The next series of posts that I hope to publish involves telling a very long multi-article story about the free software movement, what it means to me personally, and how I have seen it change over the years. I want to talk about the various actors in the free software movement, especially "corporate" influence on free software. I also want to write about the x86/Python/weird low level ptrace stuff I've been doing recently.

Doing this involves a lot of work:

In other words, to get better content I need a better copy editing system. I plan to make some behind-the-scenes changes to how I write blog posts so that I can (hopefullly) write new content in Google Drive and then have it rendered the same as currently. That is a lot of code that I have to write still.

In other words, I hope to re-launch my blog with higher quality content soon. Stay tuned.