If you are using a Linux desktop then there's a pretty decent chance that the
terminal emulator you're using is
gnome-terminal, the GNOME terminal emulator.
I have a bad habit of opening way too many GNOME terminal emulator windows on my
deskop. At the end of the day I'll easily hit a dozen open
windows. As a consequence of this I have noticed a few really interesting things
First: while there is a command called
gnome-terminal, you won't see any
gnome-terminal in your process listing. Instead you'll see a
single process called
gnome-terminal-server that is bound to the same TTY as
all your other X11 processes.
gnome-terminal-server instance is multi-threaded, but not
thread-per-window. On my machine it always uses four threads even though I may
have more (or fewer) than four terminal instances.
gnome-terminal-server process can get "blocked" and therefore
prevent terminal input or output on other terminals!
I will explain what I mean. I have previously written about how I use autofs to develop with NFS and virtual machines. Lately I have had an issue where the first time I try to access the autofs system (which has fully booted at that point) sometimes the initial I/O commands in the mounted directory get "blocked". What I mean by "blocked" is that any I/O over the autofs directory blocks for a few seconds when I first try to access it---and then I'll see all the processes finishing their I/O at once. I have written another article about the "uninterruptible sleep" state on Unix, and in that article I specifically discussed how historically that program state is associated with NFS.
I haven't debugged the issue yet. It's probably something I can work out by
looking through the appropriate log files on the NFS server (or enabling the
right verbosity of something to get the logging to happen). Since it only
happens once per boot and blocks for a brief time (usually less than a minute) I
haven't devoted my time to tracking down the root issue yet. My theory is that
it's related to my
PS1 prompt variable having overly complex
git status at least once every time a prompt needs to be
rendered, and the
git status command has to do a ton of networked I/O
operation to read the
.git/ directory and its subdirectories, etc. This is
just a hunch though.
Today I rebooted my machine and in the first gnome-terminal emulator instance
that I opened I tried to
cd into a git directoy within the autofs system. This
caused my system to pause, as I have previously observed.
What was new today was that when I launched new terminals they were all blocked
as well! I tried to open multiple terminals to diagnose the issue as it was
happening, but since I didn't have any terminals open yet (this being the first
terminal emulator process) I wasn't able to actually really debug anything with
gdb or strace. After a minute or so the initial window's
completed and immediately all of the other windows were rendered as well.
What does this mean?
Well, first of all it's interesting that
gnome-terminal was designed like
this, i.e. without a thread/process per window. It's easy to see why someone
might have tried to make this "optimization": it reduces memory footprint to
multiplex multiple terminals at once.
It also means that internally within
gnome-terminal there's some kind of mutex
that it's possible to hold while entering the uinterruptible I/O state. This is
a bad practice: because certain I/O operations can block for a long time
(potentially forever), one should not hold any mutexes while doing an
uninterruptible I/O operation.
My blog has quieted down recently---it's been nearly a month since my last post, which is one of the longest periods since I restarted blogging recently.
Do not fear, intrepid reader! I have not given up blogging.
Rather, I want to start producing higher quality posts in general. That means fewer ad-hoc posts like this one, and more posts that are systemic narratives of interesting systems.
The next series of posts that I hope to publish involves telling a very long multi-article story about the free software movement, what it means to me personally, and how I have seen it change over the years. I want to talk about the various actors in the free software movement, especially "corporate" influence on free software. I also want to write about the x86/Python/weird low level ptrace stuff I've been doing recently.
Doing this involves a lot of work:
- I have to finish researching and writing all of the content (which has been started).
- My current workflow is me editing markdown files in a private git repository, and I have my own code for rendering HTML the way I want and so forth. This is great for me because I get to use Emacs with 80-character column widths, but horrible for anyone else who wants to help edit my work.
In other words, to get better content I need a better copy editing system. I plan to make some behind-the-scenes changes to how I write blog posts so that I can (hopefullly) write new content in Google Drive and then have it rendered the same as currently. That is a lot of code that I have to write still.
In other words, I hope to re-launch my blog with higher quality content soon. Stay tuned.