In early June this year I was researching updates in newer versions of OpenSSH and learned about the relatively new Ed25519 key format. Ed25519 is based on the technology behind elliptic curves, and has a number of advantages over RSA keys. Ed25519 keys are currently believed to be more secure than 2048-bit RSA keys (what I was using at the time) and are smaller and faster. I was also pleased to see that the venerable Daniel J. Bernstein (i.e. djb) was behind this new technology. Thus I decided to switch from my old 2048-bit RSA key to a shiny new Ed25519 key. The only real caveat that this should have is to impact the ability to SSH to older hosts; Ed25519 support was added in OpenSSH 6.5 (released January 2014) so upgrading could be a problem if you have older hosts in your roster. That didn't apply to me, so upgrade I did.
At the same time that I made this switch I also had to make a bunch of changes to my workflow controlling how I was loading keys into my SSH agent. At the time I was working with hosts that were considered to be insecure, so we had policies in place for how those hosts could be accessed and what keys could be forwarded to those hosts. It was at this time that I discovered Debian Bug #472477. This bug was reported in March 2008 and is related to the fact that gnome-keyring-daemon doesn't actually have the ability to unload SSH keys from memory. It's more than 7 years later now and this bug is still open, and no progress has been made on fixing it. I dug into this a little bit, and was shocked when I figured out what was going on.
OpenSSH isn't written as a library, it's written purely as an application. There's no way to link against any of the key parsing code in OpenSSH. This is significant because there are multiple key formats (e.g. RSA and Ed25519 keys have a totally different text format). I found this also when I looked into how pam-ssh works. If you look at the pam-ssh source code you'll see that there's a ton of old OpenSSH code that was copy-and-pasted from the OpenSSH repo into pam-ssh, and that's how the ghetto key management code works.
When the inexorable progress of time makes its slow but steady pace applications
that assume they know how OpenSSH works start to break. In the case of the
Ed25519 standard applications like pam-ssh and gnome-keyring-daemon are slow
to learn about the new key formats because the code needs to be reimplemented in
those applications. Currently gnome-keyring-daemon cannot load Ed25519 keys
because no one has imported that behavior.
Likewise, there is no "keyring" or "agent" library provided by OpenSSH; applications like gnome-keyring-daemon that implement the behavior of a keyring or agent have to implement all of the logic themselves. This manifests itself in things like Bug #472477 where no one actually implemented the unloading behavior of keys from gnome-keyring-daemon.
When I looked around, I was pleased to find that there's a newer project out
there that handles a lot of these problems in a more elegant way. In particular
I'm referring to the wonderful Envoy project.
By contrast to gnome-keyring-daemon, Envoy decides that it shouldn't concern
itself at all about things like parsing SSH configs or parsing SSH key files.
Envoy just takes on the task of launching and managing an instance of the
ssh-agent process. When you ask to add or remove a key it shells out to
ssh-add. And it can load Ed25519 keys just fine because ssh-agent can, so it
doesn't need any special code for this task.
The Envoy approach seems much more sane to me. Additionally Envoy has a bunch of other cool features like support for launching instances in contained cgroups.
I start envoyd on boot using systemd. It runs as a regular system daemon
process as root. If you're not comfortable running envoyd as root, note that
this stuff is optional, and you can run envoy exactly like ssh-agent(1) if you
like. In my .bashrc file I have a bit of code that checks if Envoy is present
and if so it unsets SSH_ASKPASS and SSH_AUTH_SOCK and runs source <(envoy -p).
I've been on this setup for about four months now and love it. I've dug around the Envoy code a bit and found it to be understandable and well written. I'm also happier and feel more secure using something that manages ssh-agent processes directly rather than incorrectly reimplementing OpenSSH behavior.