missives from evan klitzke…

Cornering At Low Speeds


Learning to corner on a bicycle effectively at low speeds will help you a lot when riding in dense urban environments. This comes up a lot in stop-and-go traffic as cars or other bikes cut in front of you, as you notice pot holes, etc. This skill is especially helpful if you have clipless pedals (or cages/toe straps) since it's harder to put your foot down quickly.

One way to think about turning on a bicycle is that you can decrease your turning radius (i.e. turn more tightly) by leaning the bike more. How much you can lean the bike before falling depends on how fast you're going and how much traction you have. When you're going really fast you can have a more aggressive lean angle; for instance, in MotoGP races you'll see motorcycles at high speed lean so far that the rider's knee will touch the pavement. When you're going slowly normally you cannot have an aggressive lean angle because the bike will tip over. This means that normally when you're going really slowly you can't pivot the bike around obstacles quickly.

If you lean too aggressively, the bike will start to tip over. If this happens you can usually correct the action by accelerating. As you accelerate it changes your angular momentum and the bike will right itself.

There's a really effective way to combine these two properties on a bicycle. Here's how it works. You stand up and get out of the saddle and lean over at an aggressive angle. By standing you increase the net torque on the system because you move your center of gravity out further from where the tires contact the road; this makes the turn more aggressive. Then as you are near the middle of the turn you accelerate by using as much force as you can to turn the pedals. Because you're out of the saddle, you should be able to get extra force when cranking the pedals because you'll naturally use your weight to help rotate the pedals. By accelerating like this the bike will right itself (and you'll gain speed).

If you do this properly you'll feel like you're "throwing" your weight around; first when you stand up and lean over, and then again as you use your weight to force the cranks around faster. The whole thing will quickly feel really natural as you get used to how far you can lean and how fast you can accelerate. This technique is easiest if you're going slowly enough that you can accelerate through the turn with a single rotation of the cranks because that will let you use your weight to rotate the cranks most effectively.

On a fixed gear bicycle this process is a little bit more difficult because you can't control the pedal position to stand or control the crank angle as you begin accelerating. I would argue though that this skill is more important to master on a fixed gear bicycle since foot retention is more important on fixed gear bicycles, and putting your foot down on a fixed gear bicycle is more annoying since you can't easily rotate the cranks to the right starting position.

One more note: be careful when employing this technique when the road is wet. When the road is wet you have less traction than usual, and if you've developed intuition for how much torque you can apply on a dry surface you can lose traction in the rain.

Impressions From Running a Full Bitcoin Node


A while back I wrote an article about running a full bitcoin node. Since that time I've actually set up the node and run some experiments with it, so I wanted to report back on my experience.

Running Bitcoin Core

Getting Bitcoin installed on Ubuntu is really easy; it's just a matter of enabling the Bitcoin PPA. This is likely a matter of running the following command on your machine:

sudo apt-add-repository ppa:bitcoin/bitcoin

As I noted in my previous article, compiling bitcoin from the source is another possibility if you're not on Ubuntu but is a bit more work. The main thing you'll run into is building Berkeley DB 4.8 from scratch (if you want the portable wallet feature) and possibly memory issues if you have a system with less than 1 GB of memory. I successfully built Bitcoin Core from source on both Debian and Fedora so it's definitely possible, I just ended up going with Ubuntu and the PPA approach because of the ease of keeping Bitcoin Core up to date that way.

Syncing the blockchain still takes a long time. About a year ago a "headers-first" synchronization option was added to Bitcoin 0.10.0 which is supposed to dramatically reduce the time it takes to initially sync the blockchain; there are some release notes describing the feature here. The way it works is your bitcoind process fetches all of the "headers" for the blockchain blocks, and the headers contain cryptographic hashes of the blocks. Afterwards your client can download the blocks in parallel from other nodes in the network, sort of similar to BitTorrent. In practice, I found that while my bitcoind process did indeed fetch all of the blocks from the network very quickly, the CPU time it takes to actually verify all of the blocks ends up being the limiting factor in syncing the blockchain. The node I'm running has a particularly anemic CPU so I'd imagine that if you had a desktop computer this would be less of an issue. Still, it took several days for me to sync the blockchain.

Sending and receiving money securely is easy. To receive money:

bitcoin-cli getnewaddress

This will generate a new (unique) wallet address for you to receive money. If you use this command every time you want to receive money it will help anonymize your transactions.

To send money:

bitcoin-cli sendtoaddress <destination-address> <number-of-btc>

There are a bunch of other RPCs that are useful, but these should be enough to get started. I was pleasantly surprised to see that on my system there is bash completion for bitcoin-cli, which makes exploring the RPCs a bit easier.

Setting up Bitcoin Core to use Tor is also really easy. You just need to add a proxy line to ~/.bitcoin/bitcoin.conf which points to your Tor SOCKS proxy (which by default will be There are more detailed instructions here.


The last thing I tried was setting up JoinMarket which is an open source CoinJoin implementation written in Python. Setting it up is easy, the instructions in the README should be enough to get you started.

Once you have JoinMarket installed you can transfer bitcoin into the "zero mixing depth". Subsequent operations will send bitcoin to deeper mixing depths. Unfortunately, I wasn't able to actually get this working. When I ran the tumbler.py script it was able to send some bitcoin into the deeper mixing depths, but it never got close to the bottom depth. The tumbler would give me an estimate like "10 hours" and I waited up to 2+ days and never got things close to the bottom mixing depth. I also tried sending coins with the sendpayment.py script and got errors.

While I was testing JoinMarket I only put about 0.2 BTC into my JoinMarket wallets (currently worth about $60 USD) while I tested it. I suspect that my transactions are failing for two reasons: there aren't enough users on JoinMarket, and my transaction amounts are too small to get picked up by other users. That said, I'm not really interested in transferring more BTC into my JoinMarket wallets until I am able to definitely diagnose this as the problem. I also want to take a minute to figure out how to get coins out if things go awry. Superficially it appears easy since you can easily list the wallet addresses for all of the coins, but I need to spend a bit more time experimenting with this.

I've seen Chris Belcher (the author of JoinMarket) in the #bitcoin room in Freenode and I also saw that JoinMarket has its own Freenode channel. One of these days when I feel sufficiently motivated I'll see if I can track someone down on Freenode who can help me with my problems. In the meantime, I am using conventional tumblers, which I really hate doing. I'd love to be able to report good things about JoinMarket in the future.

The Caveats of TCP_NODELAY


One common mistake I see in socket programming is incorrect usage of TCP_NODELAY (the socket option that disables Nagle's Algorithm). There's a ton of advice on the internet where people are promoting the usage of this socket option as a panacea for network latency. Often the posts I've seen suggesting that TCP_NODELAY will be helpful are actually technically correct but miss the larger picture of the context in which it's appropriate to use this option, and generally how to write high performance socket programs.

This came up today at work. Someone was trying to load test their service with a program written using the Tornado HTTP client. This library comes in two varieties: a high performance libcurl binding and a pure Python version that just uses the low level Tornado IOStream stuff. In both cases the benchmark was getting poor latency numbers even though the benchmark was set up with a concurrency of ten. The engineer writing the load test refactored it to use the Python multiprocessing module with the regular blocking urllib2 library and got a huge speedup (again, with concurrency ten), in line with the expectation for the service.

A bunch of theories were thrown back and forth about why this was happening. One of the engineers looking at this had read Julia Evans' recent blog post about TCP_NODELAY and suggested that this may be the culprit. The engineer even found a related Tornado Github issue where Tornado wasn't setting TCP_NODELAY in earlier versions. There was some confusion about which Tornado version we were using in the benchmarks, whether urllib2 was using TCP_NODELAY, and so on.

I was kind of skeptical that TCP_NODELAY was related for a few reasons. First, in my experience most applications that use this flag are not written correctly in the first place. Indeed, in the blog post Julia references discussing the issue with Ruby's Net::HTTP module, you can see right off the bat that the Ruby library is not written well. To wit:

Ruby’s Net::HTTP splits POST requests across two TCP packets - one for the headers, and another for the body. curl, by contrast, combines the two if they’ll fit in a single packet.

Unnecessarily sending data out in multiple packets here is bad programming practice in the first place (more on this in a second). The benchmark that came up at work was making HTTP GET requests which don't have bodies and should fit in a single packet, even if the library was written sloppily. Additionally one variation of the Tornado load tester was using the libcurl bindings, and as noted in this post libcurl properly buffers its writes to send full packets.

In this post I'm going to discuss:

A Brief Overview of TCP and the Kernel

The programming interface exposed to you by the kernel for TCP sockets (actually, all SOCK_STREAM sockets) is a file descriptor that you can write(2) and read(2) on. As you might guess, the write(2) system call sends data, and the read(2) system call reads data.

In reality how this works is that the kernel has two buffers allocated for the file descriptor in kernel memory: a write buffer and a read buffer. The write(2) system call doesn't actually cause TCP packets to get sent; what it actually does is copy data into the kernel's write buffer for the socket. Likewise read(2) fetches data out of the read buffer (or blocks if the read buffer is empty). There are some further details about blocking and non-blocking sockets and other advanced socket options, but those details aren't relevant for this discussion.

When there is unsent data in the write buffer the kernel tries to send it out over the network. This takes the form of a data payload wrapped in a TCP packet, which itself is wrapped in an IP packet, which itself is wrapped in a link layer packet (e.g. an Ethernet frame). The time when this TCP packet it sent is potentially much later than when you actually issued the write(2) call that copied data into the outgoing write buffer. You can't use the write(2) call to know if the data has been sent (and even if you could, sent data can be dropped or lost and may need to be transmitted anyway).

The TCP protocol itself has a rule for when the kernel is allowed to send TCP packets. There's a window size for the TCP connection that tracks how many unacked packets can be in transit at any time. When this window size is hit, data written by the application will be copied into the kernel write buffer but not sent immediately. There are two cases when an application is likely to run into this: (1) new TCP connections start with a tiny window size and therefore write delays are common, and (2) applications that are actually saturating the network bandwidth will eventually run up against the window size. The situation relevant here is (1), the behavior of a new TCP connection. In the original RFC for TCP the window size started at one packet. This means that the first write(2) call on a new TCP stream will cause a packet to be immediately sent, but a quick second write(2) will not cause data to be sent if the first packet hasn't been acked. This has been somewhat fixed; RFC 3390 increases the initial window size to up to four packets, and there have been efforts to increase that even further.

Besides the TCP rules for when it is permissible to send packets, there are some recommendations for how TCP stacks should work. In some cases your operating system may choose to delay sending TCP packets even when the current window size would allow a packet to be sent. There are two relevant options here: Nagle's algorithm (which can be disabled by TCP_NODELAY) and TCP delayed acknowledgement (which on Linux sort of can be controlled by TCP_QUICKACK, with some caveats).

Nagle's algorithm notes that a userspace application might be making lots of small calls to write(2) it may be beneficial to have the kernel try to coalesce the writes. If the kernel naively translated all of these writes into TCP packets then the kernel would end up sending a lot of small packets. For instance, consider the statsd protocol. In this protocol a stat is sent by the stat name, a value, and optionally a "format". A typical stat sent in this protocol is very small; say, 50 bytes. An application that is trying to send 10 stats might therefore issue 10 calls to write(2) each sending 50 bytes. It's easy to imagine how this would be the case: you structure your library so there's a send_stat() function which writes the data, and then call that function 10 times. What Nagle's algorithm does is says: if there is unacked data sent, and if the write buffer in the kernel is smaller than the MTU, then wait a little to see if the application writes more data. If the write buffer reaches the MTU size then the data will be transmitted. If the in flight data is acked then the data will also be transmitted, even if it is smaller than the MTU. In the example I just gave, this would mean that the kernel will automatically coalesce the stats sent into a single 500 byte packet. As you can see, this is a really good thing, and generally not something you want to disable.

The TCP delayed ack feature is again an attempt to minimize the number of small packets sent. The way it works is a TCP packet can ack multiple data packets at once. Therefore a TCP stack implementing the delayed ack feature may wait up to some amount of time before acking packets in the hope that it will be able to ack more packets at once. On Linux this can cause up to a 40 ms delay when acking packets. Again, this is usually a good thing since it decreases the number of packets that have to be sent (which is usually the limiting factor in network performance).

The Problem

The problem happens if Nagle's algorithm causes a sent packet to be delayed and when the sending program needs the packet to be sent before it can read data on the connection. This situation is actually problematic even if Nagle's algorithm alone is on, but delayed acks amplify the problem.

If a program writes a single packet and then reads, then the packet will be transmitted immediately (since there are is no unacked data the rule about unfull buffers doesn't apply). The opposite end will process the packet and then send its response (which will include an ack flag). Thus, there is no unnecessary delay.

If a program issues multiple writes and doesn't have to read anything then there is no problem. The worse case here is that delayed ack is turned on which means that at the end of the write sequence there is at most a 40 ms artificial delay before the sender knows the data is acked. However if the sender isn't blocked while it waits for a response (e.g. because there are no responses, or because the communication is fully bidirectional) then this won't be noticed.

But consider what happens when a program issues multiple writes followed by a read that is dependent on the write data. The final write may queue due to Nagle's algorithm. The other end won't process the request (and therefore will not start sending a response) because it's waiting for the final packet of data. The other end also won't ack the data it got until the delayed ack timer fires. When this finally does happen the sending program gets its acks and is finally able to actually transmit its last packet. This more or less guarantees the 40 ms delay (unless the write payload just happened to exactly be a multiple of the MTU, which is unlikely), a delay which will be noticed in many applications.


Off the bat, you should try to send as much data to the kernel as possible in write(2) system calls. If that's difficult to do because of how requests are crafted then you should investigate the writev(2) system call which implements "vectorized I/O". This makes it easier to write applications that generate data to be sent into multiple small buffers and want to efficiently copy the data into the kernel.

In the Ruby example, if the Net::HTTP module issued a single write(2) or writev(2) call then the POST request would have likely fit into a single packet which would have avoided the need to use TCP_NODELAY. It's still possible that even if this happened the POST body could be sufficiently large to cause the request to be split. However this shouldn't be too common for most web applications. A good rule of thumb is you can typically fit at least 1400 bytes of payload before the packet will be split.

When And How to Use TCP_NODELAY

An application that is very latency sensitive, particularly if it doesn't transmit a lot of data, can safely use TCP_NODELAY. A good example here would be an SSH session. In an SSH session it's common that the user will be typing input to the remote server, and the user will certainly be typing at a slow pace compared to the bandwidth capabilites of the network. If Nagle's algorithm is on then that means that keystrokes will end up being delayed frequently. This will make a lot of applications difficult to use—imagine using vi or emacs with Nagle's algorithm on.

If you understand this and you want to enable TCP_NODELAY anyway, then you definitely can as along as you follow one rule: you must make your best effort to send large writes to the kernel. That is, it is incumbent on you to ensure that small amounts of data in the send buffer are actually because there's no more data to be sent, not because your application is doing something hacky and is issuing lots of small write(2) system calls. Again I'd like to encourage the use of vectorized I/O here. For instance if you have a line oriented protocol you may find it easiest to structure your program by creating multiple buffers each storing a single line, and then you can transmit all of the data at once in a single syscall (without unnecessary memory copies!) using writev(2).

Multi-threaded programs that share sockets between threads without locks are particularly susceptible to problems related to Nagle's algorithm. This is because even if the application is structured so that a single thread doesn't do the write-write-read pattern, if there is one writer thread and another reader thread then the pattern can still manifest. This is one good argument for not sharing sockets between threads without mutexes; usually a better option is to use an event loop.

Note that if you are using a proxy like HAProxy then all bets are off. Even if you disable Nagle's algorithm, the proxy could have it on.


Linux implements an interesting option called TCP_CORK. You can think of it as an even stronger variation of Nagle's algorithm. It says: if there is data in the write buffer, and that data is less than the MTU size, and if there is no unacked data, continue to delay sending the write buffer data up to 200ms in the hope that more data will enter the write buffer. In other words: it tells the kernel to try extra hard to delay sending small packets, allowing the kernel to delay sending such packets by up to 200ms.

This is something that I added to statsrelay which significantly increased the throughput of that program. Prior to using this option many programs would issue tons of small writes into statsrelay. This would cause lots of small packets to be sent over the network (even though TCP_NODELAY was not on). In some cases we had network interfaces that were reaching their max packets-per-second due to all of these small packets even though the total bandwidth was well below what the interface could handle. After turning on TCP_CORK the number of packets per second the statsd boxes received dropped dramatically (up to 10x if I recall correctly). This comes at the cost of some latency, but the underlying data store (Carbon) only supported second resolution for data anyway so in practice this was not a problem.

A Final Note

If you made it this far, you may wonder what the original problem was with the Tornado load testing application. It ended up actually being an issue with how the futures were being created and collected. All of the possible requests the application could make were created as futures at once with timestamp of when the future was created. However, only ten futures at a time would actually be active. This meant that the last futures being collected were showing long response times even though the actual requests were much faster.

The Hate Is Real


The techie hate in the Mission has been getting intense lately. Every day I see wheatpastes and graffiti decrying techies. The new condo buildings in my neighborhood (especially the "V20" condos at 20th and Valencia, and the Vida SF building near 22nd and Mission) constantly have the front windows smashed in. For more than a week now there's been graffiti on the sidewalk in front of my building saying "Warning: Hot Rental Market, Controlled Burns Scheduled". Recently Prop I was on the ballot, which, if passed, would have banned nearly all new market-rate housing in the Mission while the city developed a "neighborhood stabilization plan". Today I was at Four Barrel and someone had dripped this dark thick paint over all of the sitting surfaces so that if you sat there you'd get paint on your clothes. There wasn't a message clearly attached, but I think it's pretty obvious to anyone who lives in the area why that shop was targeted.

It's kind of crazy living in a neighborhood where I'm targeted with this kind of hate. It's surreal that there is graffiti outside my building implying that arsonists are going to burn it down. It's surreal that graffiti like this isn't immediately buffed by the city.

I don't have any intention to move. In fact, it seems likely that I will buy property in this neighborhood in the event that my current employer has a liquidity event in the next few years.



Last week it was announced that Rdio would be acquired by Pandora for $75m (after previously raising $125m in earlier venture capital rounds). As part of this deal, Pandora will acquire all of the "technology and intellectual property" from Rdio, and Rdio will be shutting down its subscription service effective November 23. The rumor is that Pandora plans to launch a paid choose-what-you-listen-to streaming service similar to the Spotify/Rdio offering sometime in 2016.

I'm rather saddened by the whole thing. I've been a paid Rdio user more or less since it launched, and have gotten a lot of my friends to switch to Rdio. I loved that Rdio focused on having an HTML/Javascript based player from day one. In my opinion Rdio has the best user interface for album-centric music listening, especially with respect to discovering new music. That said, the writing has been on the wall for Rdio for a number of years now, so their recent acquisition announcement didn't surprise me (if anything I'm surprised it didn't happen sooner).

What upsets me the most about this is the future we're headed into. It appears that in not too long the only real major options for streaming albums will be from Spotify, Google, and Apple (and maybe Pandora?). It's a tough market to compete in because all of the companies are making razor thin margins due to the licensing costs. This means that there's very little room to differentiate in terms of cost or selection. Ultimately when there's so little competition it ends up hurting consumers the most.

What makes this situation especially infuriating is that no one really seems to be profiting from it. Consumers have very few options to choose from, and the options they do have are essentially priced the same with the same music selection. The companies offering these services aren't making that much money, as evidenced by the acquisition of Rdio (at a price lower than what they raised in VC funding). In fact the most recent news I cound find about Spotify's financials show that Spotify is still operating at a net loss, and their operating loss is increasing with time. The artists aren't really happy because they make almost no money from streaming services, e.g. as evidenced by Taylor Swift's recent spat with Spotify. The only actor here who really seems like they might be doing well is the record companies who take in the lion's share of the subscription payments for these services. Even the record companies aren't happy though, since revenue from streaming services seems to come at the expense of traditional album sales.

In my opinion the major problem here is that the distribution rights to almost all of the music that consumers want to listen to is owned by only three companies: Universal Music Group, Sony Music Entertainment, and Warner Music Group. Even smaller "indie" labels that maintain artistic control often sign away distribution rights to these companies for practical reasons (e.g. because these distribution deals may be necessary in order to get radio air time). The fact that so few companies own so much of the music we listen to is bad for everyone. In fact, the problem is getting worse: I remember when Napster was first making waves people were talking about the "big five" record labels who were pursuing legal action against Napster, and since then EMI and BMG have been absorbed leaving us with only three major record labels left.

I don't really see how the current situation can get any better since all signs indicate that ownership of distribution has been getting more centralized not less centralized. Maybe one day anti-monopoly laws will be used to untangle this mess, but that seems far fetched. In the meantime it appears that I'm going to be switching to Google Play Music (until Pandora's new streaming offer materializes at least), and I'll continue to hope for a brighter future.

I Broke My Wrist (Pt 3)


I had my wrist surgery two days ago. They performed an internal fixation with a titanium Herbert screw. Today I met with my physical therapist and got a pretty neat thermoplastic splint, as pictured below.

a great image

On Monday I'll meet with my doctor and find out more about the timeline for physical therapy and splint removal. The physical therapist I talked to today told me that it's likely that I'll be in the splint for another eight weeks (although it's too early now to really know). Because the scaphoid bone has so little blood flow it takes longer to heal than most other fractures. After the splint is onde I'll still have more physical therapy while I regain full mobility.

I'm pretty over sitting around and not being able to do anything, so I think I'm going to start biking to work (very carefully) again soon.

Uninterruptible Sleep


One of the curious features of Unix systems (including Linux) is the "uninterruptible sleep" state. This is a state that a process can enter when doing certain system calls. In this state, the process is blocked performing a sytem call, and the process cannot be interrupted (or killed) until the system call completes. Most of these uninterruptible system calls are effectively instantaneous meaning that you never observe the uninterruptible nature of the system call. In rare cases (often because of buggy kernel drivers, but possibly for other reasons) the process can get stuck in this uninterruptible state. This is very similar to the zombie process state in the sense that you cannot kill a process in this state, although it's worth that the two cases happen for different reasons. Typically when a process is wedged in the uninterruptible sleep state your only recourse is to reboot the system, because there is literally no way to kill the process.

One infamous example of this has been Linux with NFS. For historical reasons certain local I/O operations are not interruptible. For instance, the mkdir(2) system call is not interruptible, which you can verify from its man page by observing that this system call cannot return EINTR. On a normal system the worst case situation for mkdir would be a few disk seeks, which isn't exactly fast but isn't the end of the world either. On a networked filesystem like NFS this operation can involve network RPC calls that can block, potentially forever. This means that if you get the right kind of horkage under NFS, a program that calls mkdir(2) can get stuck in the dreaded uninterruptible sleep state forever. When this happens there's no way to kill the process and the operator has to either live with this zombie-like process or reboot the system. The Linux kernel programmers could "fix" this by making the mkdir(2) system call interruptible so that mkdir(2) could return EINTR. However, historical Unix system since the dawn of time don't return EINTR for this system call so Linux adopts the same convention.

This was actually a big problem for us at my first job out of college at Yelp. At the time we had just taken the radical step of moving images out of MySQL tables storing the raw image data in a BLOB column, and had moved the images into NFS served from cheap unreliable NFS appliances. Under certain situations the NFS servers would lock up and processes accessing NFS would start entering uninterruptible sleep as they did various I/O operations. When this happened, very quickly (e.g. in a minute or two) every single Apache worker would service a request handler doing one of these I/O operations, and thus 100% of the Apache workers would become stuck in the uninterruptible sleep state. This would quite literally bring down the entire site until we rebooted everything. We eventually "solved" this problem by dropping the NFS dependency and moving things to S3.

Another fun fact about the uninterruptible sleep state is that occassionally it may not be possible to strace a process in this state. The man page for the ptrace system call notes that under rare circumstances attaching to a process using the ptrace system call can cause the traced process to be interrupted. If the process is in uninterruptible sleep then the process can't be interrupted, which will cause the strace process itself to hang forever. Remarkably, it appears that the ptrace(2) system call is itself uninterruptible, which means that if this happens you may not be able to kill the strace process!

Tonight I learned about a "new" feature in Linux: the TASK_KILLABLE state. This is sort of a compromise between processes in interruptible sleep and processes in uninterruptible sleep. A process in the TASK_KILLABLE state still cannot be interrupted in the usual sense (i.e. you can't force the system call to return EINTR); however, processes in this state can be killed. This means that, for instance, processes doing I/O over NFS can be killed if they get into a wedged state. Not all system calls implement this state, so it's still possible to get stuck unkillable processes for some system calls, but it's certainly an improvement over the previous situation. As usual LWN has a great article on the subject including information about the historical semantics of uinterruptible sleep on Linux.



Two years ago I permanently deleted my Facebook account. It's one of the best decisions I've made as an adult.

It's kind of hard to explain the confluence of factors that led me to delete my account, and even if I could some of the details are more personal than I'd like to share anyway. But the short version is that there was a lot of stuff going on in my life that was causing me to be unhappy. When I would go on Facebook I felt like I was thrust into the middle of a situation that was causing me a lot of anxiety and wasn't making my life any better; in fact, it was making my life worse.

There are certainly a lot of upsides to having a Facebook account. Without question, it makes maintaining social relationships with people that you don't see every day much easier. It's a convenient and easy way to store photos and share photos with friends and family. It also makes it a lot easier to coordinate events and parties, and to get invited to other people's events and parties. It's easy to create accounts on a lot of third-party sites using Facebook login.

However, Facebook also makes a lot of real world social relationships a lot more complicated. The dynamics of the platform encourage you to "friend" people liberally which means that you give a lot of people—many of whom you probably don't know that well—a lot of accesss to information about your life. There are privacy settings that let you control which groups of people can see what content, but then you are forced to artificially group your contacts into a strict set of buckets that doesn't actually mirror how relationships work in the real world. I don't have a strict mental compartmentalization of everyone I know, they all exist in various contexts to me on a gradient that isn't expressed with groups or circles. Even if I did, I don't want the cognitive burden of choosing who should be able to see what for everything I do online.

Facebook makes dating people really weird. To start, there is this stupid binary and public division when you're "in a relationship" with someone or no longer in a relationship with them; a distinction that can lead to unnecessary drama. If you manage to avoid that you invariably end up with a lot of photos, content, etc. linked to past romances. This leads to a weird situation where when you enter into new relationships with other people it's easy for them to learn a lot about your past relationships, and vice versa. That dynamic is part of what leads people to editorialize their profiles and photos after breakups (another recipe for drama). The liking and commenting semantics of Facebook make it a lot easier to speculate about possible romantic relationships and interest than it would be otherwise.

After deleting my Facebook account (and my LinkedIn account), I'm a lot happier about what information people can easily look up about me. I have a public account on Instagram, Twitter, and Tumblr; but I don't post anything that I would consider to be very personal to any of those accounts. If you want to figure out where I work or where I went to school, you can put two and two together if you're diligent; but it's not that forthcoming from any of my public accounts. You can't easily tell who I socialize with in real life from any of these accounts.

It's different for every person, but there have been numerous studies showing that Facebook usage is negatively correlated with happiness. While I'm sure this is not the case with every person, it's something that I experienced and it's easy for me to see why this can be the case. If you like Facebook and feel like it brings value to your life, by all means continue to use it. But if you suspect that it might not be affecting your life in a positive way then I encourage you to reconsider whether it's something you want to have.



Over the past eighteen months or so I've developed an interest in Satanism that has gone from ironic to not-serious-but-maybe-not-quite-ironic. This has become a perpetual source of questions as people try to figure out how serious I am about the whole thing, what does Satanism even mean, and so forth. The purpose of this piece of exposition will be to explain my interest and views on the matter.

A Brief Tour of Satanism

People who call themselves "Satanists" divide the term into two major groups. Theistic Satanists are people who actually believe in Satan as a divine being. I am personally a bit skeptical that there are really any true theistic Satanists of advanced age; to be a theistic Satanist would mean that you essentially believe in the Abrahamic concept of God and Satan and heaven and hell and all that, and yet you choose to embrace the path that will lead to eternal damnation and suffering. The other group is non-theistic Satanists, who don't actually believe in Satan as a divine being who is in opposition to the Abrahamic God, but do embrace some system of ethics or beliefs that is non-religious in nature but is somehow "Satanic". Most self-labelled Satanists are of the non-theistic variety, and the most prominent non-theistic Satanist movement is LaVeyan Satanism.

I think to understand the whole thing you need to start with the historical context for Satanism. Historically, the term has been levied at essentially any group of people or movement that is allegedly un-Christian in any way. For didactic purposes an interesting example here is the Leviathan cross, pictured below:

the leviathan cross

This symbol is commonly used as a "Satanic cross". Anton LaVey adopted it as a Satanic symbol associated with his Church of Satan, which he started in 1966.

The actual history of the Leviathan cross is rather interesting. It was one of the symbols used by the Knights Templar which was a Christian military order that existed in Western Europe about a thousand years ago. The Knights Templar were largely responsible for organizing and funding the first two Crusades. After the mid-12th century the fortunes of the Knights Templar started to turn, and the group fell out of favor with the Catholic Church and the various nation states of the day. The reason for their fall from favor is complicated but mostly has to do with political reasons based on military defeats in the Middle East, money owed to the Templars by various nations, and groups (such as the Catholic Church) who felt threatened by the power and influence of the Templars.

The church and various European governments started attacking the Templars and accused them of being a non-Christian organization and more specifially of charges such as heresy. Many of the Templars were tortured, forced to give false confessions, and burned at the stake. Later, after the rise of Freemasonry (another group that was attacked in much the same way) people tried to link the Freemasons to the Knights Templar as a way to discredit the Freemasons.

Thus the Leviathan cross is actually only Satanic in the sense that it's historically linked to a group that the Catholic church opposed. In fact the Knights Templar were more of a Christian fundamentalist group than anything else (which is why they started the Crusades in the first place) and any suggestion that they were practicing Satanism is absurd. The fact that today people recognize the symbol at all is largely due to Anton LaVey who claimed it as a Satanic symbol.

Likewise, many other aspects of "Satanic" imagery have very modern roots. The two other most prominent Satanic symbols are probably the inverted pentagram and the depiction/concept of Baphomet. Both of these symbols are due to a single person, the 19th-century French occultist Eliphas Levi. These are essentially products of Levi's imagination and were inventions for his treatises on magic. Other Satanic symbols that we recognize today were mostly inventions of 19th and 20th century occultists (e.g. Aleister Crowly and Anton LaVey) who were specifically looking to create Satanic symbols. Thus these Satanic symbols are mostly inventions of non-theistic Satanists who were specfically looking to create Satanic symbols.

In the modern era Satanic symbols and imagery are probably most associated with heavy metal music and most specifically with the black metal genre of music. Many bands within this genre incorporate Satanic concepts in their songs, album artwork, and sometimes performance costumes. Generally this is of a mostly fantastical nature. A very small number of bands associated with this movement—particularly in Norway—have actually expressed and acted in extremely misanthropic and anti-Christian ways. In particular, in the early 1990s there were church burnings in Norway infamously linked to black metal groups like Burzum. However, these incidents are extremely fringe, are hardly representative of the thousands of other bands associated with metal and black metal, and were mostly linked to the 1990s Norwegian black metal scene.

Satan in the Bible

Before proceeding, I want to briefly talk about what is actually said regarding Satan in the Old Testament and the New Testament.

In the Old Testament the story of Satan is described briefly in Isaiah 14:12-15 and Ezekiel 28:12-19. Satan is also mentioned in passing in a few other Old Testament books, notably including Job ch. 1-2 where God has a discourse with Satan about his devout subject Job. In Christian doctrine the serpent from Genesis who tempts Eve is commonly held to be an incarnation of Satan. It's worth noting that while this is Christian doctrine, the serpent is not actually identified as Satan in Genesis, so other Abrahamic religions (e.g. Judaism) do not necessarily agree on this point.

In the New Testament Satan also makes a few very brief appearances. Satan appears during the story of the temptation of Christ, which is told in the Gospels of Matthew, Mark, and Luke. The Book of Revelation makes numerous references to Satan in a propheptic and apocalyptic context.

All said, there's really not a lot of material on Satan in the Bible. If we go purely off the Biblical texts it would be rather hard to formulate a philosophy or religion around Satan, as there simply isn't enough material to really understand who Satan is or what he stands for (other than in a superficial manner). In fact, much of the popular Christian concept of Satan actually comes from later interpretations of Satan from non-monastic sources. For instance, the epic poem Inferno from Dante's Divine Comedy provides much of the popular understanding of Satan, although this is purely a work of fiction. Similarly William Blake wrote a number of influential works such as The Marriage of Heaven and Hell that have influenced the popular conception of Satan—but these works are Blake's personal beliefs, and not religious texts.

So Like... What Does It All Mean?

To be clear: I don't consider myself to be a Satanist in any sense of the word. My so-called religious affiliation is atheism or perhaps more specifically secular humanism. That being said, by my own admission I have a lot of curiosity for Satanist aesthetics and I've accumulated a small collection of Satanist artwork, records, and even tattoos. The more I've dug into this movement the more I've found it intriguing and the more I've understood the reclaiming of Satanism.

With the historical context for Satan and Satanism in place, it should be clear that it's impossible to give a unified coherent explanation or study of Satanism, because one does not exist. Multiple independent groups of people have claimed or adopted Satanist imagery and symbolism. The concept of Satanists as devil worshippers is essentially a myth. Even if there are some bona fide theistic Satanists, the form of theistic Satanism they practice is a modern creation that has no real historical lineage.

Without question there are a small number of historical precedents for extremely anti-social groups or people who are linked to Satanism. However, it's also worth noting that there are a far greater number of anti-social groups and people linked with far-right interpretations of mainstream religions including Christianity, Judaism, and Islam.

The concept of Satanism as a movement that is anti-Christian or is somehow defined by an opposition to humanist or Christian ethics is fundamentally flawed. In fact, I think this viewpoint is largely one that has been created by Christians who assume to know what Satanism stands for without actually having researched the matter. If Christianity is right, then anti-Christianity must be wrong by definition, and therefore the view that Satanism is anti-Christian is an easy way to discredit the movement without a real attempt at critical analysis. My thesis then is that Satanism is a rejection of this—that is, a rejection of non-critical thinking. By adopting Satanist imagery or symbolism inviduals are automatically rejecting those who refuse to engage in rational and critical thought.

I Broke My Wrist (Pt 2)


Nine days from now (on November 18) I'm getting surgery for a "open reduction internal fixation of right scaphoid fracture". They're going to make an incision in my wrist and then install a permanent titanium screw that will hold the two fragments of my scaphoid bone together.

In the meantime, my splint has been upgraded to a bigger and more solid splint that immobilizes most of my right arm below the elbow. I'm told that the expected prognosis after surgery is that I'll have to wear a splint full time for a week after the procedure, and then a splint part time for a week after that. I'll also have to do physical therapy afterwards to (hopefully) regain full mobility and strength in my hand.

Fun times ahead.

I Broke My Wrist (Pt 1)


Hot on the heels of my "I Broke My Face" series, a new development has occurred in the ongoing saga of my recent bike accident. When I went to the ER the first time they took CT scans of my head and X-rays of my wrist. At the time the X-rays of my wrist didn't indicate any fractures, so I was given a wrist splint to wear for a few weeks; the doctor told me that it could just be a sprained wrist, and that immobilizing it would help.

It's been nearly a month and my wrist isn't doing any better, so yesterday my doctor ordered new X-rays to be taken. The results came back today and they indicate that I've fractured my scaphoid bone. I have a followup appointment with my doctor on Monday to discuss treatment options. A likely outcome of this is that I'll get a Herbert screw put in my wrist to fuse the bone back together, but I won't find out for a few days.



Strace is one of my favorite commands. I use it probably almost every day at work, and regularly use it on my home computers to debug all manner of things. People using OS X or BSD like to talk about how great DTrace is, but I think that for 99% of the things I want to do strace is vastly simpler and therefore is actually much more useful, despite being more limited in nature.

How To Use It

Previously I wrote an article about setting up eCryptfs on Fedora. When I was first setting this up, I didn't correctly have my user in the ecryptfs group and after running ecryptfs-setup-private I got an error like this:

/usr/bin/ecryptfs-setup-private: line 450: /sbin/mount.ecryptfs_private: Permission denied
ERROR:  Could not mount private ecryptfs directory

What this error was actually telling me was that my ecryptfs directory had been successfully created but couldn't be mounted (which would have been solved by just restarting my login session). If you know this is what's going on the error message makes sense, but it doesn't necessarily make sense if you don't know that up front. So after restarting my session I tried running ecryptfs-setup-private again and instead got this error:

ERROR:  wrapped-passphrase file already exists, use --force to overwrite.

This is equally cryptic if you don't know what's going on. It's kind of clear that there's some sort of file somewhere that exists, and the setup script is trying to not overwrite that file. But it's not at all clear where that file is since the error message doesn't actually include the path.

This is an example of the kind of problem that you can debug in a minute or so once you know how to use strace. Running

strace ecryptfs-setup-private

and scanning the output will show that there aren't any suspicious failed system calls or access checks, but you will see that the command calls clone(2) and receives a SIGCHLD. This indicates that the ecryptfs-setup-private comand is creating child processes or threads that are doing the actual work, so we need to trace those:

strace -f ecrypts-setup-private 2>&1 | less

This will cause the child processes/threads to be traced as well and will send all the output to less. This yields nearly 3000 lines of output on my computer, which is a lot to scan through, but we can guess that since the error message says that the "file already exists" we probably need to look for a system call that returns EEXIST. When I searched for this, it immediately took me to this section of the strace output:

[pid  4798] mkdir("/home", 0777)        = -1 EEXIST (File exists)
[pid  4798] chdir("/home")              = 0
[pid  4798] mkdir("evan", 0777)         = -1 EEXIST (File exists)
[pid  4798] chdir("evan")               = 0
[pid  4798] mkdir(".ecryptfs", 0700)    = -1 EEXIST (File exists)
[pid  4798] stat(".ecryptfs", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
[pid  4798] close(1)                    = 0
[pid  4798] close(2)                    = 0
[pid  4798] exit_group(0)               = ?
[pid  4798] +++ exited with 0 +++

and then about ten lines below that I see

stat("/home/evan/.ecryptfs/wrapped-passphrase", {st_mode=S_IFREG|0600, st_size=58, ...}) = 0

This is pretty obviously the issue that the error message is talking about. I got a little lucky here by searching for EEXIST which wasn't actually the root of the error but was close to where the error occurs. I could have also searched for wrapped-passphrase which was mentioned as the filename in the error message. After doing this I poked around in a bash shell looking at the ~/.ecryptfs directory and figured out what was going on.

Again I got lucky because I had a good guess of what the actual error was; that said, this is the type of thing you'll quickly develop an intuition for if you use strace regularly.

If you don't get as lucky with guessing the error code or filename to look for, usually the right thing to do is to look backwards in the strace output for calls to exit(2) or exit_group(2). Shortly before the exit_group(2) call typically you will see what the program was doing when it decided to bail. Another technique that works is looking for system calls that have negative return codes (again, typically you want to search backwards since you want to look for failures near the end of the program run). Another technique is to look for the write call to stdout or stderr that prints the error message, and try to work backwards from there. In any case when you get stuck the rule of thumb is to try to look at the end of the output first, since most programs exit (or at least print an error) right after encountering a problem.

If the program is really complicated or does a lot of stuff before exiting then pipeing the output to less will be unwieldly. In this case you want to do something like:

strace -ff -o x ecryptfs-mount-private

This will create a bunch of files named x.<PID> with the actual PID substituted. Afterwards you can usally use grep to figure out which files you want to investigate further, and once you've narrowed it down you can analyze the files with less, vi, etc. This technique works well when you have a lot of child processes to trace, since isolating the output of each process makes following the trace of each of the processes simpler as you don't have to deal with interleaved output.

There are a couple of other options you'll want to know. To trace an already running process, use:

strace -p <pid>

To get timing information, you'll want to -tt to get absolute timestamps and/or -T to get elapsed time. I frequently use them together, e.g.

strace -tt -T ls

The leading portion here will show the timestamp and the trailing part will show how long the system call took. This is useful for debugging slow programs, e.g. you can look for system calls that take more than one second easily by grepping the output of this.

By default strings will be truncated after only 32 characters which isn't that many; to get more characters use strace -s <number.

If you know what system calls you're interested in you can filter with -e. For instance, let's say I want to know how the ping program creates a socket for sending ICMP packets; I might look at this with

strace -e socket ping www.google.com

You can specify mulitple system calls this way, e.g.

strace -e socket,setsockopt curl -s www.google.com >/dev/null

There are a myriad of other options that strace will accept, but the ones I've listed here are the ones I most frequently use.

When To Use It

A lot of people know about strace, but in my opinion fail to use it effectively because they don't use it at the right time.

Any time a program is crashing right after you start it, you can probably figure out what's going on with strace. The most common situation here is that the program is making a failed system call and then exiting after the failed system call. Frequently this will be something like a missing file or a permissions error (e.g. ENOENT or EPERM), but it could be almost any failed system call. As I said before, try to work backwards from the end of the strace output.

Any time you have a program that appears to be hung, you should use strace -p to see what system call it's stuck on (or what sequence of system calls it's stuck on). Any time you have a program that appears to be running really slowly, you should use strace or strace -p and see if there are system calls that are taking a long time.

In both of the cases I just listed, there are a few common patterns that you'll see:

Another really common use case for me is programs that aren't loading the right files. Here's one that comes up a lot. I frequently see people who have a Python program that has an import statement that is either loading the wrong module, or that is failing even though the developer thinks that the module should be able to be imported. How can you figure out where Python is attempting to load the module from? You can use strace and then look for calls to open(2) (or perhaps acces(2)), and then grep for the module name in question. Quickly you'll see what files the Python interpreter is actually opening (or attempting to open), and that will dispell any misconceptions about what may or may not be in your sys.path, will identify stale bytecode files, etc.

If there aren't any slow system calls or the system calls don't appear to be looping then the process is likely stuck on some sort of userspace thing. For instance, if the program is computing the value of pi to trillions of decimal places it will probably not be doing any system calls (or doing very few system calls that complete quickly). In this case you need to use a userspace debugger like gdb.

One final piece of advice I have is to try to make an effort to learn as many system calls as you can and figure out why programs are making them. If you've never used strace before the output is going to be really overwhelming and confusing, particularly if you don't have much Unix C programming experience. However, there actually aren't that many system calls that are regularly used or that are useful when debugging these issues. There are also common sequences or patterns to system calls (e.g. for things like program initialization, memory allocation, network I/O, etc.) that you can pick up quickly that will make scanning strace output really fast. All of the system calls you see will have man pages available that explain all of the arguments and return codes, so you don't even need to leave your terminal or have an internet connection to figure out what's going on.

Happy debugging, and may the strace gods bless you with quick results.

eCryptfs on Fedora


For some reason it's hard to find up to date installation instructions for eCryptfs on Fedora, so I'm posting instructions here. These work with at least Fedora 22 and Fedora 23, and maybe other versions. Note that eCryptfs isn't a substitute for full disk encryption; I personally use it in addition to full disk encryption (via LUKS) to get further security on important files.

First, install ecryptfs-utils:

sudo dnf install ecryptfs-utils

Subsequently the ecryptfs kernel module will get automatically loaded on boot, but immediately after installing the RPM it probably won't be loaded. So you have to load the kernel module manually once (or reboot):

sudo modprobe ecryptfs

Next you need to make sure that your user has been added to the ecryptfs group:

sudo gpasswd -a $USER ecryptfs

You'll want to log out of your current session and back in to make sure that this change affects your session. Finally, once you are in the ecryptfs group you need to run:


This will prompt you for your login password and optionally an ecryptfs password. The script will then do some stuff to make ~/Private be your ecryptfs private directory. The way this actually works is it creates ~/.ecryptfs which has the encrypted data and keys, and there are scripts to mount this via FUSE to ~/Private. Thus if you mess something up and want to restart, you can rm -rf ~/.ecryptfs which stores the actual encrypted data and keys and you'll reset things.

Subsequently, when you want to have ~/Private be decrypted and mounted you'll run:


and likewise to unmount ~/Private you'll run:


If you want to do something fancy you might need to deviate from this setup and RTFM, but for the usual use case of "I want an encrypted directory for my personal files" this system is easy to configure and works well.

New Bike Day


My sister had a birthday earlier this month. During the house party she had to celebrate the occasion her bike was stolen. This was a huge bummer, but it made it easy to figure out what to get her for her birthday, and I got to go bike shopping which is one of my favorite things.

I took her to Mission Bicycle Company the day after her party and we put in an order for a new bike. I tried to convince her to do a build similar to my Automatix build (also from Mission Bicycle) but she wanted gears so we went that route. Her build ended up with the Shimano Alfine 8-speed internal gear disc hub. We also did the full Mission Bicycle security package which has security bolts for the saddle and the Pinhead front wheel lock.

Another cool thing about this bike is that normally the shifter that comes with Alfine hubs can only be mounted on upright 22mm diameter handlebars. Mission Bicycles actually works with someone who has designed a custom anodized aluminum shifter that can be mounted on other handlebar sizes. This means you can use standard 25.4mm handlebars with the Alfine hub. My sister took advantage of this and went with 25.4mm bullhorn style handlebars which is kind of a unique build that only Mission Bicycles does as far as I know.

I was also impressed when we got the bike to see that they actually do internal cable routing for these bikes. This is something that you normally only see on higher end road bikes, and as someone who loves steel frames I'm happy to see someone making steel frame bikes this way.

the new bike

Happy birthday Larissa!

Envoy and Ed25519


In early June this year I was researching updates in newer versions of OpenSSH and learned about the relatively new Ed25519 key format. Ed25519 is based on the technology behind elliptic curves, and has a number of advantages over RSA keys. Ed25519 keys are currently believed to be more secure than 2048-bit RSA keys (what I was using at the time) and are smaller and faster. I was also pleased to see that the venerable Daniel J. Bernstein (i.e. djb) was behind this new technology. Thus I decided to switch from my old 2048-bit RSA key to a shiny new Ed25519 key. The only real caveat that this should have is to impact the ability to SSH to older hosts; Ed25519 support was added in OpenSSH 6.5 (released January 2014) so upgrading could be a problem if you have older hosts in your roster. That didn't apply to me, so upgrade I did.

At the same time that I made this switch I also had to make a bunch of changes to my workflow controlling how I was loading keys into my SSH agent. At the time I was working with hosts that were considered to be insecure, so we had policies in place for how those hosts could be accessed and what keys could be forwarded to those hosts. It was at this time that I discovered Debian Bug #472477. This bug was reported in March 2008 and is related to the fact that gnome-keyring-daemon doesn't actually have the ability to unload SSH keys from memory. It's more than 7 years later now and this bug is still open, and no progress has been made on fixing it. I dug into this a little bit, and was shocked when I figured out what was going on.

OpenSSH isn't written as a library, it's written purely as an application. There's no way to link against any of the key parsing code in OpenSSH. This is significant because there are multiple key formats (e.g. RSA and Ed25519 keys have a totally different text format). I found this also when I looked into how pam-ssh works. If you look at the pam-ssh source code you'll see that there's a ton of old OpenSSH code that was copy-and-pasted from the OpenSSH repo into pam-ssh, and that's how the ghetto key management code works.

When the inexorable progress of time makes its slow but steady pace applications that assume they know how OpenSSH works start to break. In the case of the Ed25519 standard applications like pam-ssh and gnome-keyring-daemon are slow to learn about the new key formats because the code needs to be reimplemented in those applications. Currently gnome-keyring-daemon cannot load Ed25519 keys because no one has imported that behavior.

Likewise, there is no "keyring" or "agent" library provided by OpenSSH; applications like gnome-keyring-daemon that implement the behavior of a keyring or agent have to implement all of the logic themselves. This manifests itself in things like Bug #472477 where no one actually implemented the unloading behavior of keys from gnome-keyring-daemon.

When I looked around, I was pleased to find that there's a newer project out there that handles a lot of these problems in a more elegant way. In particular I'm referring to the wonderful Envoy project. By contrast to gnome-keyring-daemon, Envoy decides that it shouldn't concern itself at all about things like parsing SSH configs or parsing SSH key files. Envoy just takes on the task of launching and managing an instance of the ssh-agent process. When you ask to add or remove a key it shells out to ssh-add. And it can load Ed25519 keys just fine because ssh-agent can, so it doesn't need any special code for this task.

The Envoy approach seems much more sane to me. Additionally Envoy has a bunch of other cool features like support for launching instances in contained cgroups.

I start envoyd on boot using systemd. It runs as a regular system daemon process as root. If you're not comfortable running envoyd as root, note that this stuff is optional, and you can run envoy exactly like ssh-agent(1) if you like. In my .bashrc file I have a bit of code that checks if Envoy is present and if so it unsets SSH_ASKPASS and SSH_AUTH_SOCK and runs source <(envoy -p).

I've been on this setup for about four months now and love it. I've dug around the Envoy code a bit and found it to be understandable and well written. I'm also happier and feel more secure using something that manages ssh-agent processes directly rather than incorrectly reimplementing OpenSSH behavior.

Running A Full Bitcoin Node


One of the things I set up and got working recently was operating a full node on the Bitcoin network. There were two motivations for me for doing this.

The first is that I was generally interested in learning more about Bitcoin and scripting the way that I interact with Bitcoin. Running a full node and using the official bitcoin-cli client is a good way to do both of these things; it's also beneficial to the Bitcoin community as a whole.

The second is that after the recent near death of my phone I realized that I'm better with backing up and scripting tasks on servers than I am at doing the same with my phone. Previously I would write down my Mycelium backup codes on a post-it note and then lose the post-it note. By contrast writing an automated cron job that makes a copy of my wallet, encrypts it, and then puts it in a private S3 bucket is easy, cheap, and has good security. There are some good arguments that using a physical copy of the wallet can be more secure, but in practice the way I organize my life I think that th solution of dumping and encrypting a wallet on a Unix server using cron is easier. I also think that it would be harder for an attacker to get a login on one of the Unix servers I administer (which only allow non-root key-based SSH logins with a single key that I am good at securing) than it would if they had my phone (which has encryption at multiple layers, but all based around short numeric PINs that are easy for me to remember).[1]

Additionally, it seems plausible that it would be easier (and funner) for me to write and automate things for managing multiple wallets and the money in them myself rather than relying on the solution the Mycelium developers come up with. As an example, recently I wanted to experiment with JoinMarket. Doing this on a server is easy, doing this with a phone-based wallet would not be.

I looked into a couple of options for running a full node:

I determined that I didn't want to use my laptop mostly because of the overhead of running bitcoind. It uses more disk space, RAM, and network bandwidth than I was happy with for something that will always be on on my laptop. Particularly on the last point, I'd only want to run bitcoind when I was on a good network link and that would mean trying to write complicated scripts to govern if bitcoind should be running or not based on the perceived quality of my internet link.

The mini-ITX server in my apartment thing seemed cool at first, but once I specced it out it didn't really make sense. I wanted to build a fanless system capable of running RAID 1 (arguably this is overkill, but whatever) with sufficient storage. I specced something out with this motherboard using a RAID 1 with mSATA connectors, and after buying the motherboard, CPU, RAM, power supply, disks, and enclosure it worked out to something like $800. Arguably such a system is overkill (e.g. if backups are being made to S3 then RAID is not necessary, and w/o RAID a cheaper motherboard would be used) but that was the system that was compelling to me.

By contrast I looked at using a g1-small machine type on GCE. For the same cost as the mini-ITX server I could run an instance type like this with storage and whatnot for something like two years. Plus with the GCE instance it would be easier to do things like OS or hardware upgrades (boot a new machine, copy the wallet over, power off the old VM). Based on my personal experiences running home servers and running machines on GCE, I felt like the GCE solution would be as reliable or more reliable than something connected to the internet/power in my apartment unit.

If you're really obsessed with security the AWS/GCE solution is probably a nonstarter, since it means you have your wallet on a machine not in your physical posession, which could be subpoenaed by law enforcement, etc. For me, however, the security model of GCE is sufficient with balancing my convenience and my actual real world security profile.

After I decided to go with the GCE route, I spun up a few g1-small images and played around with them until I got to a happy state. I tried using Debian Jessie, Fedora, and Ubuntu 14.04 LTS. The Ubuntu LTS is by far the easiest to get working because you can install bitcoind via a PPA. You can compile bitcoind on Jessie or Fedora easily enough, but it's a fair amount of work and among other things involves building and linking against a specific version non-packaged version of BerkeleyDB 4.8. I also found that compiling bitcoind uses a lot of memory due to the style of C++ coding they're using, and I was worried that perhaps one day you'll need even more memory to compile and link bitcoind which would necessitate an instance type upgrade for something that is a relatively infrequent task. Plus, if I was running a bitcoind I built from scratch it would be my job to keep it up to date, whereas installing packages out of a PPA from someone I trust who maintains it is a lot easier. So as much as I love dislike Ubuntu, I ended going with Ubuntu LTS.

After getting the image running and installing bitcoind, I found that it takes a few days to sync the block chain using this instance type. The limiting factor in syncing the block chain on this instance type is the CPU. The g1-small instance type only has 1 CPU which is anemic and can't even close to keep up with the bandwidth on the instance. When I have to spin up another instance for upgrades I will probably figure out how to copy over the block chain to the new instance, because this would surely be much more efficient than resyncing the entire block chain from scratch.

Once the bitcoin block chain is synced, the bitcoind process uses about 300 MB of RAM when running and uses a small amount of CPU (usually < 1%, but it will burst much higher from time to time.) My 15 minute load average seems to be about 0.10 or less which includes all of the other tasks on the system.

I haven't written any interesting scripts yet, but doing so seems to be really easy. The bitcoin-cli command prints all of its output to stdout as JSON, so consuming its output from Python should be trivial.

One final thing to note is that all of these solutions have non-trivial cost and complexity. Using a lightweight wallet on your laptop or phone is much cheaper and easier than running a full node. What I've described here is more akin to something out of a cypherpunk's wet dream than it is a practical solution for people who just want to partake in the Bitcoin ecosystem.

[1] As I've alluded in other blog posts, I have a pretty whacky non-standard SSH system in place that I've built using envoy. I will blog about this sometime since I think it solves some interesting problems.

I Broke My Face (Pt 4)


It's been a week since I last posted about my new face, and a bunch of stuff has happened.

Last week on Tuesday (the day after I wrote my previous post) I went to California Pacific Medical Center - Pacific (at Clay & Webster) and got surgery. The way it was supposed to work was that I was supposed to fast from midnight Monday until the next morning, and I was supposed to check in at 6:45 AM for a surgery at 8:45 AM. The original goal was that after surgery I'd spend a few hours in the hospital chilling and recovering, and then go home later in the day.

I got almost no sleep that night, and got up at 5:30 AM to get ready to leave for the hospital. I did all that and checked in at the hospital shortly after 6:30 AM. I changed out of all my clothes and got ready. They put an IV in me and took some blood samples, and then told me to wait and eventually someone would come get me and take me up to the OR area to talk to an anaesthesiologist. I waited and 8:45 came and passed and no one came for me. At around 10:00 AM I asked one of the nurses what the deal was, and she eventually found the head nurse who called my doctor. The nurse said that my doctor was "on his way" to the hospital but didn't have an ETA for me.

I took a nap and woke up around noon. Still no doctor, no one told me what was going on, etc. Again I ask what's going on, and the head nurse comes back and tells me they reached someone from his office who is going to come in and explain the situation. One of the assistant doctors (?) from his office comes in and explains that Dr. Miranda got called into an emergency surgery situation, and that I have to wait until that surgery is done. At around 1:00 PM they come around and tell me that that surgery is ending, and the doctor himself comes and talks to me at around 1:30 PM to explain the situation and apologize. They took me up to the OR area at around 2:00 PM and then after talking to the anaesthesiologist and whatnot my surgery began some time after 3:00 PM I believe, almost five hours late.

I was under complete general anaesthesia for about three hours while they opened up my eye socket and maxillary zygoma (cheek bone) and did reconstructive surgery on both of them. I'm not sure exactly what they did still, but they said after that the damage to the cheek area was a lot more extensive than they had thought from the CT scans. I believe that I have a bunch of metal and screws in my face now where they reconstructed my eye orbit and my new cheekbone. When I woke up I had stitches in three different places around my eye (including stitches sealing part of my eyelid shut!) and stitches in my upper gum area inside my mouth where they accessed my cheekbone.

I came up at around 6:00 PM or 6:30 PM and under the doctor's recommendation, spent the night in ICU.

ICU in a major SF hospital like this is crazy. I have so many ICU stories for another day. I was in a room with the weirdest eccentric mega-millionaire Marin county retireee ever who was actually also in on a biking injury (old guy who was doing an overhaul of one of his roadbikes, tried to take it up a huge hill and then allegedly the chain snapped as he mounted at the top and he fell and broke his hip; although I have some doubts about certain aspects of the story).

The tl;dr though is that ICU really sucks. On the one hand it's cool becasue I was conected via IV to a hydrocodone drip, and I could get more painkillers at any time by pressing a button. On the other hand it really sucks because you're connected by the IV to a saline drip that you can't control. The saline drip is meant to keep your electrolytes and whatnot at the right levels, but it also makes you need to pee every hour or two. Since you're connected to an IV, you can't go pee on your own. You need to call a nurse in to disconnect you from the IV. Same with when you're done in the bathroom—a nurse needs to come reconnect you to the IV. Also only nurses are allowed to come connect or disconnect you, not the nurses assistants, so I spent a ton of time buzzing the thing to have someone go find someone else who could come over to let me use the bathroom. In general it's crazy because ICU is way understaffed, so it's this constant no-sleep cycle of going to the bathroom, getting disconnected or reconnected to the IV, having a flobotomist come by regularly to record your vitals, having medicine/food administered, etc.

The most bizarre part of ICU was my release. They had faxed in an order for all of my medication except the painkillers to the Walgreens near my house the night before my surgery. Only no one told me they had done that. Once they found out I didn't have any of my medication, they didn't want to release me since I guess a lot of people are like "fuck that shit" when they get released, go home without any medicine, and then end up in an even worse situation and end up in the ER/OR/ICU again a few days later. So after the most beauracratic process I've ever witnessed they get the drugs switched from the Walgreens near my house to the Walgreens attached to the hospital, and are arranging for someone to come up from that Walgreens and deliver the medicine to me bedside in ICU. Sounds great. I get a phone call from Walgreens while I'm still in my bed, connected to the IV, and they go over the order with me and talk about which medicine my insurance didn't cover (eye drops and some other eye ointment) and do I still want to pay for those things out of pocket (yes). Then they ask me how I'm going to pay, cash or credit? I say credit. They're like OK, we need your credit card information before we can send up the order. I try explaining them I have an IV in my arm, and they're asking me to get up, find my backpack out of the closet, find my wallet within my backpack, and then read off all of the information over the phone to them. The lady is like yes, that's what we need you to do before you can get the medication you need to have before you can be released.

While they're going over the discharge information, I learn a few things:

A few hours after I got home and was trying to sleep (again—since I hadn't slept at all in more than 24 hours at that point) and I started getting a fever and felt swelling in my sinuses. Then blood started coming out of my nose. I freaked out because they told me that if I get a fever or bloody anything it could be a sign of an infection, and that I should go back to ER immediately. I have my friends help me get to the ER of the same hopsital I was operated at (in Pacific Heights), and by the time we get there my temperature is normal, I've stopped the bleeding from my nose, and things seem OK. The doctor there told me that bloody discharge is normal on the first day after surgery since there was entrapped blood in my sinuses from the surgery. Specifically the way it came out with me (watery bloody discharge at first that then gets thicker) is indicative of a normal situation. Thanks for not telling me that when I was released from the hospital.

I spent most of the last week in an opiate binge constantly sleeping and iceing my face. The first 72 hours after the surgery I was on a "clear liquids only" diet which basically means clear juices (e.g. grape juice) and Jello only. This sucks a lot because it's a calorie negative diet and I had been fasting for nearly 24 hours before it, so it makes you feel really woozy and tired all of the time, a feeling which is aided by the constant administration of narcotics.

I started to feel a lot better on Saturday except my eye had a bunch of gross discharge that kind of mixed with my sutures to seal my eye shut. I spent a bunch of the day on Saturday and Sunday just trying to keep my eye clean and lubricated with eye drops, since they told me that if it didn't stay properly lubricated I was at risk to get corneal abrasions if my eye got too dry.

Today (Monday) I saw my doctor again for the first time since the operation and he removed most of my sutures. The result is that now I can sort of open my left eye about halfway, and I can finally wear glasses again and have binocular vision. Even though it's still really swollen and bruised, and my eye still won't fully open, my vision is a lot better than with just one eye, and it's a lot easier to use a computer now. I'm also off my painkillers now which means that I can stay up for more than a couple of hours at a time.

Until I see my doctor again on November 5 I have to stay on a completely liquid diet. After that, if my healing is satisfactory to him I can start eating really soft foods like mashed potatoes and overcooked pasta.

I should be able to return to work on my weird liquid-only diet tomorrow.

I Broke My Face (Pt 3)


Update on the previous posts.

Today I saw Dr. Miranda as well as my usual optometrist, Dr. Tran. I am scheduled for surgery tomorrow at the California Pacific Medical Center Pacific Campus at 2351 Clay St. My surgery is at 8:45 am but I'm supposed to be there at 6:45 am because that's just how it is.

My optometrist did a regular vision check and also checked for muscular, nerve, and retinal issues. She didn't find any problems.

The surgery that Dr. Miranda will be performing tomorrow is listed on my consent papers as an "open reduction internal fixation of left zygoma maxilla and orbit". There are basically three things he's going to be fixing.

The first is that the CT scans show that there was an orbital floor blowout greater than two square centimeters. The orbital floor is a paper thin piece of bone that normally holds the eye up at the correct vertical position. When there is a blowout of this size then typically what would happen is, if uncorrected, the eye itself would sink down into the region where the orbital floor was damaged. This would make my eye look sunken down and asymmetric. In most cases this will not cause long-term vision problems, but the asymmetry would make me look hella goofy, so they're going to fix it. In rare cases they can put the bone fragments back in place, but in most cases they need to construct a new orbital floor (I believe out of plastic?) which is held in place with small metal screws. This general process is called internal fixation. He said that after I heal there shouldn't be any outwardly visible issues, but I may be able to feel the screws when I press that area with my fingers. This will all be done through an incision in my lower eyelid.

The next issue he'll be fixing is a damaged area of bone in my cheek, that is the zygomatic process of maxilla. The CT scans show that the bone in this area is crushed in, so again he'll create an internal fixation to repair the area. Again, this isn't a serious medical problem with my face, but if the procedure wasn't done my cheeks would be asymmetrical and my left cheek would look sunken in. This will be done through an incision inside my mouth, in my inner cheek.

The third issue that he'll be fixing is the pinched nerve I have that is preventing me from feeling much on the left side of my face/jaw/lips. Currently there is damage where my maxillary nerve exits my skull at the foramen rotundum. That is, there are some bone fragments in this area putting pressure on my maxillary nerve. This will be fixed by an incision at or near my eyebrow. This is actually the thing that will take the longest to recover from, the doctor told me it could be several months he said before I regain full feeling in my upper lip.

These procedures are all pretty common with this type of trauma, because the injuries I have are to the weakest parts of the skull in that area. It is common for all three of these procedures to be performed together.

I will be on a liquid only diet for about a week after surgery, and clear liquids only for about a day after surgery. This is because the incision area in my mouth can't be fully sterilized, and the area they're working with are near my sinuses which also cannot be fully sterilized. Therefore there's a risk of infection in my mouth and sinuses after surgery if food or bacteria enter the incision. I'm also going to be given a regimen of antibiotics for preventative purposes.

I have to be up bright and early tomorrow for this procedure, so that's it for now. I'll update again after the surgery.

I Broke My Face (Pt 2)


Quick update for y'all from my previous post. My left eye, which had been completely swollen shut since Thursday morning, started to get unswollen enough to begin opening last night (Saturday night).

I basically spent all day (today, Sunday) sitting at home, napping, and iceing my eye and it has opened up a lot more. I was getting kind of worried earlier in the day because if I tried to open both eyes and use glasses I was getting really bad double vision. Basically everything in my left eye was tilted at about a 30° angle counter-clockwise, so if I tried to look at things with both eyes open it would be really blurry and I'd quickly get a headache. If I looked at things with just my left eye the angle of everything was still tilted, but my vision otherwise seemed fine.

I just took a pretty long nap, and after waking up things are much better. It still strains my eyes to look at things less than about a foot away, but my long distance vision seems pretty good and I can look at objects further than about 18 inches away without any double vision which was not the case this morning. It seems so far that the double-vision has been commensurate with the swelling around the eye. This jives with the research I've done online. Basically what happens is the swelling around the region will push the eye out of place and that's what causes blurred or double vision. So with any luck, once the swelling is fully subsided things will be fine.

I've read that untreated orbital fractures can cause permanently sunken or misplaced eyeballs. This is something that would be corrected by surgery. As far as I can tell right now my eyes don't appear to be sunken or misplaced, but I have a lot of blood in my cornea and there's still swelling so it's hard to tell exactly.

Tomorrow (Monday) I have what I had originally thought was actually going to be actual surgery, but is in fact just a consultation. That doctor should be able to answer most of the remaining questions I have about my healing, vision, recovery time, etc. I'm going to see Dr. Edward Miranda who is the doctor that St. Francis ER referred me to, and he's going to go over my CT scans, look at me, and then advise on what to do next. I checked out his Yelp reviews and he seems pretty highly rated except for people complaining about what health insurance he accepts and wait times in the office. As the referral doctor for ER, from what I could tell on Yelp he has a lot of experience with trauma problems like I have, so he's not "just" a cosmetic plastic surgeon. I already talked to an admin at his office on Friday and they verified that my health insurance plan covers things with my insurance policy at a rate that is acceptable to me.

Again, I won't know until tomorrow, but I read online that typically facial surgery for an orbital fracture happens 10-14 days after the fracture. They do this because they want swelling to be minimized in the region before attempting surgery. That would mean that if surgery is warranted it would probably happen next week.

I'm also going to call my optometrist tomorrow to see if I can get an eye exam. Last time I called I had a situation where I had lost my glasses and had no contacts (a really funny story for another time) and they were able to fit me into an appointment same day and get things worked out for me (i.e. they got me contacts that same day). I just visited them about a month ago and I believe I'm maxed out on my vision insurance for the year, but thankfully I'm in the financial position where if I have to pay out of pocket it will be no big deal. I did already get a really quick check at the ER on my left eye and they said it looked fine to them, but they didn't have all the fancy machines (or optometry expertise) that my optopmetrist has.

So far I've written this entire blog post with glasses on and both eyes open, which is the first time I've been able to use a computer with two eyes in about five days. So things are looking much better than before, and with any luck after tomorrow I'll have all of my further questions answered by real medical professionals.

Will update tomorrow after I talk to the professionals.

Platinum Tunnels


I did a fun thought experiment today. I currently have stretched earlobes, to the size of 00g. Typically I wear double flared stainless steel tunnels like these. For reference, I paid $10.10 for a pair of stainless steel tunnels from this vendor which includes taxes and shipping.

The thought experiment I did was: what might it cost me if I wanted to get the same type of tunnels, but made from platinum?

Today I weighed this pair of tunnels on a 0.001 g accurate scale. The pair weighs out at 2.666 grams. I'm not sure exactly what type of stainless steel these tunnels are made from so I'm assuming it's stainless steel 304 which has a density of 7999 kg/m^3. There are some stainless steel varieties that are denser and some that are lighter, but they're all pretty close to this value (within a few percentage points), and stainless steel 304 seems to be quite common.

Platinum is 21451 kg/m^3 or about 2.7 times denser than stainless steel. When I did the math, if my tunnels were made exactly the same way but made of platinum rather than steel I estimate that they'd weigh 7.150 grams for the pair.

The current value of platinum is $32.62/g so the value of the platinum in such tunnels would be $233.23 just for the platinum.

I have no idea what the actual cost would be to get something like this made. I did some research into how platinum rings are made. What they normally do is take a bunch of platinum and then cast it into a tube of some thickness for a specific ring size. Then they take the tube and cut it into a bunch of sections where each section is ring-shaped. These ring-shaped sections are then worked to get the desired final shape and moulding and whatnot. This is an efficient process because a lot of rings of the same size can be made easily. Making tunnels like I have be fairly nonstandard since it would require a custom casting or milling process.

Android Developer Options & ADB


My Android phone broke today. This morning I had just gotten out of the shower and was using my phone when the touchscreen stopped responding to input. This can happen when moisture gets into the phone, so I did the old put-your-phone-in-a-bag-of-rice (actually, lentils in my case) trick to see if I could desiccate the phone and get it working again.

After about 10 hours of being in the lentils I pulled it out, and the touchscreen was working again! All was well with the world for about 5 minutes, until it mysteriously stopped responding to input again. I'm currently re-lentiling the phone to see if I can get it working again.

Besides the fact that having a broken phone is a bummer, I have some data on the phone that is going to be annoying to restore if I can't get the touchscreen working again. So what I really, really want right now is to be able to access data on the phone so I can grab the files I'm afraid of losing. Most of my data is synced to the cloud, but restoring my 2FA stuff is going to be annoying, and I had a small amount of bitcoins stored locally on the phone in Mycelium that I'm going to have to go through the fun exercise of restoring.

It occurred to me just now that if I had the developer options enabled I'd easily be able to access all of my stuff using adb. I'm not currently doing Android development, so I have the developer options disabled as is the default. But if I did have them enabled, getting all of my data off would be a cinch.

It also occurred to me that this type of failure has happened to me numerous times. This is at least the third Android device I've had where the phone has been on and running, but is effectively bricked since the touchscreen doesn't respond. Anecdotally, the two primary failure modes I've seen of smartphones is either that the screen breaks in some way (won't respond to input, is all black, etc.), or the charging port gets damaged and the phone runs out of battery and won't boot anymore. The adb/developer options trick wouldn't help in the second case, but it sure would in the first.

So the lesson of the day is: if you're a hacker with an Android, enable the developer options even if you're not writing Android software. It could save the day if your touchscreen breaks.

I Broke My Face (Pt 1)


A few of you have gotten updates about this via my twitter feed, instant message, text, etc. I'm writing this article to udpate everyone on my current situation.

On 2015-10-15 at about 2:30 am I was involved in a biking accident. I hit a parked car. I don't remember a whole lot so I can't tell you exactly how it happened. I finally bought a helmet about a month ago after riding for about ten years without one, but I was not wearing a helmet at the time since I was just doing a quick six-block ride to El Farolito and didn't think it would be a big deal. The accident happened a block from my apartment and all I could really tell is that I was really fucked up, couldn't see well, but was close enough to home that I could walk home and figure things out. I called my good friend Erik when I got home, and he Ubered over and helped me clean up my face and took care of me.

The next morning Erik came over and helped me get things sorted out—helped me make an appointment with a doctor. I went to One Medical and the doctor I saw was basically like—"yeah, you need to go to the ER to get CT scans". The doctor at One Medical also did a basic check of my wrist and shoulder which were also in a lot of pain and from his preliminary analysis determined that I hadn't broken my wrist/collar bone/shoulder.

Immediately after this I went to St. Francis hospital which is near Hyde & Bush. After doing the usual wait-several-hours-in-the-ER thing I got a CT scan of my left orbital lobe, a CT scan of my head, and an X-ray done of my right wrist.

Here's the prognosis I got.

I probably sustained a mild concussion, but the CT didn't show any brain trauma.

My shoulder hurts a lot but is just bruised. No broken bones, no broken rotator cuff, etc.

My wrist did sustain a hairline fracture, but I don't need a cast. They gave me a removable plastic/velcro brace and told me to wear it as I feel necesssary.

The bad news is that I sustained two fractures to my left orbital lobe. The info sheet they gave me when I left the hospital says "Blow out fracture of orbit; Maxillary fracture." Right now I can't open my left eye at all due to all of the swelling in the region. I also don't have a lot of feeling of the entire left side of my face. All of the trauma is to the front of my face, I didn't hit the top of my head. They did pull open my eyelids and did a basic eye check and they think my eye is fine.

They did tell me they would have stitched up a big cut I got on my eyebrow had I come in the night of the incident, but by then it was too late to suture. I'm going to get a sweet scar there, so if you also count the forehead scar I got after my last major bike accident I've now one-upped Harry Potter on face scars.

I didn't ask for any narcotics because my pain level right now is bearable (as I said, I don't have that much feeling in my face).

This Monday (2015-10-19) I am getting facial surgery. From what I can tell from my online research, basically what they're going to do is try to realign everything as best they can, get bone fragments back where they should be, try to realign my jaw, fix any pinched nerves, etc. That should hopefully get me most of the feeling in my face back, although I think the whole thing with me not being able to open my left eye is mostly contingent on the severe bruising/swelling I have in that area so I think I just have to wait that out.

I'll update y'all on my situation Monday evening after I've had surgery. Obligatory photo (from Snapchat clearly) of what I look like right now. This is after I finally got all of the blood cleaned up, but my bruises got way darker so I arguably look scarier than I did yesterday morning.

my face

Also, my bike is in great shape. I just tore up the handlebar tape and twisted the seat post. The seat got a bit scratched and is bloody but that's easily cleaned. As far as I can tell there are no scratches on the frame.

Side note: if you're ever in a non-life-threatening situation and have the option to choose which ER to go to, I would not recommend St. Francis. If you've been to any of the ERs in SF (I've been to SF General ER twice) you know that they're frequently full of homeless people who aren't necessarily in a real emergency situation but rather are trying to get drugs or other random things (one guy at St. Francis literally just came in to ask them to get him a taxi voucher to send him to a homeless shelter so he could shower). St. Francis is the closest ER to the Tenderloin, and it blew my mind how many homeless people came in to the ER for nonsensical things, how many people lied about what they were being admitted for (as in, they came to the back to get an assessment, and then just asked for drugs even though they told the front desk that they had some other problem).

Update: turns out the appointment I have on Monday is just a consultation to get surgery, not the day I actually will be getting surgery.

Testing Code


I'm not particularly a big evangelist of code testing, and I don't care if you have "unit tests" or "integration tests". I do think tests are great, and I'm actually a strong advocate of having 100% (or very close to that) test coverage, but I don't have particularly strong opinions about how your test suites work. As long as the test suite reasonably matches what happens in production, I'm cool with it, regardless of what steps it took to get working. Pretty much all of my projects that are of my provenance at work have 100% test coverage (including branches) or at least 99%+ coverage.

But; If you don't have tests at all—that is unacceptable. If you have tests that are like "does this thing even build" that is also unacceptable. I should be able to make any change to your code repo, and be reasonably sure that it will at least work in some sense of that word. If I add a new feature, it's reasonable and expected that I'll add new tests. In fact, nearly all non-trivial changes should have tests associated with them.

The number of times I've made a "trivial" change to a code repository (in my opposition, w/o adding new tests [which sometimes is not possible]) and then have been blamed for the resulting horkage drives me crazy. All code should be tested. If I land a "trivial" change that completely breaks some project, even if I did not add new tests, I blame that on poor test coverage, regardless of what steps I could have taken to add new tests for my esoteric/trivial change.

Also, for the love of god, write your code to be testable in the first place. This is a specific code engineering quality that is generally learned via practice, but once you've been in the field a few years you should be writing code that is easily testable. I totally get why new engineers fresh out of school write code that is hard to test, but in my opinion it's unacceptable for 10+ year developers to be writing code that can't be easily tested.

Slack IRC Gateway


The company that I currently work for is trying to migrate from HipChat to Slack. Nothing is certain yet, but a move seems likely, and at the minimum I can say that we're evaluating Slack and I've gotten to check it out and play with it a bit.

The thing that is awesome about Slack is that it has a native IRC gateway. If you use the IRC gateway you don't get all of the whiz-bang features that the native client has, but at least the IRC gateway basically works and is supported by the company. This is a huge contrast to HipChat where the only way to get it working with IRC is using BitlBee which uses the XMPP gateway that HipChat supports and BitlBee itself acts as a bizarre XMPP/IRC bridge that maps the weird HipChat XMPP extensions into things that are sort of like IRC. The BitlBee HipChat stuff mostly works and has been pretty solid for me for about a year now, but it definitely has some quirks and is really strange to use in a lot of ways.

I live and die by IRC, so for me the native gateway is a huge, huge win. There are already a few quirks I've found (e.g. no scrollback when joining a new room), but all in all the experience is miles ahead of using BitlBee with HipChat.

Here's why the IRC gateway is a huge win for me. Most of the same is true with HipChat + BitlBee; this is more a commentary on using an IRC client to connect to these newfangled chat systems than anything else.

First of all, like most serious IRC users I have an IRC session connected using WeeChat running in an always-running tmux session that's on a remote server that I very rarely power cycle (every few months to make sure I stay on top of kernel updates). This isn't the only way to do it, Irssi is another great terminal-based IRC client, and GNU Screen is getting a bit long in the tooth but there's nothing wrong with it at all. There are numerous other IRC clients worthy of praise, WeeChat is just the one I prefer.

The reason this is cool is that I'm always connected to the chat system, so I get chats from people all the time and can respond to them at my leisure. It can be a bit annoying to other people who can't tell if I'm actually on or am just idling, but if anything I'll chalk that up as a plus. If anyone really needs to contact me, I have my phone number in a million places including in PagerDuty so I'm not too concerned about that.

Second of all, WeeChat (and most other IRC clients) is incredibly customizable, and it's easy to filter our messages I don't care about, get highlights on arbitrary words, and a number of other features that are must-have for me. Buffer management in WeeChat is really amazing and incredibly fast once you take the time to learn how it works. I can easily handle really busy times on chat better than my coworkers using the native clients. There are a million plugins (and actually I'm not using any of them), but if you really want to get down that rabbithole it's an option. WeeChat is also scriptable in a bunch of different languages including Python (my main day job programming language). I have dabbled in WeeChat scripting a bit and while I'm by no means a pro, it's pretty easy to get started and do cool things. I might write a blog post about this once I've written a more substantial WeeChat plugin.

Being able to keep client-side logs is also a godsend. There are a million times where I've wanted to find something from my chat history—potentially months (or even years) old, and being able to simply grep the chat logs is awesome. In the same way that it's common to quote from emails when replying, from time to time I've found it incredibly useful to be able to quote from chat conversations e.g. to substantiate timelines that were discussed or the division of responsibility for projects. I've also had a lot of fun doing statistical analyses of my conversations (e.g. what proportion of the messages come from me v.s. from person X) and training Markov chain bots on chat logs.

There is one serious downside which is that given someone's real name I don't know how to look up their nick/alias. For instance, suppose someone tells me I need to talk to Aubrey Graham about a project, but he cleverly set his nick to be drizzy. I can use /whois drizzy to figure out that drizzy is Aubrey Graham, but just given the name Aubrey Graham I'm not sure how to figure out that his nick is drizzy.

One more thing. If you want to use the Slack IRC gateway, I highly recommend Calle Erlandsson's excellent article on the topic where he goes into a lot more details about various configuration things and the plugins available.

If I Ever Start A Tech Company


This is going to be my pitch to potential employees:

Dear Sir/Madam,

I see that you have a LinkedIn and/or GitHub account. Would you like to work for my non-VC funded stealth company? Huge equity potential and, no joke, we have a fridge that is always stocked with PBR tallboys.

Evan Klitzke (CEO)

BitlBee Accounts


I have previously written about BitlBee & HipChat, and this article is an addendum to that piece.

I found BitlBee "accounts" to be kind of a confusing feature to BitlBee, but by my own admission the whole thing is pretty well documented, so maybe it's just a reading comprehension problem on my part. I'm going to explain it all in my own words.

First of all, if you have any problems at all look at the official commands documentation which really does explain everything clearly, although you will be really confused if you skip around as I did.

There's a /save command in BitlBee. You can also have it set up to autosave (which is I believe the default). Regardless, when you type /save in the BitlBee buffer, all of your stuff will be saved assuming that the config files have sufficient permissions and whatnot. By all of your stuff being saved, I mean your account information, account passwords, what channels you're in, what buffers they map to, and so on will be saved. It appears to me that the account/password data is encrypted somehow (presumably using your identify password`), although I wasn't fully able to reverse engineer this and is something I want to follow up on. Anyway, you'll find that your account passwords are not stored anywhere obvious in plaintext.

Completely separate from your notion of your "HipChat" account or your "Twitter" account or your "AIM" account or whatever you're using BitlBee account for, BitlBee has its own system for managing user accounts. If you just use BitlBee to connect to a single service (as I do) this abstraction layer is a bit confusing, but it totally makes sense if you have multiple accounts managed by BitlBee. The idea is that as a user I can authenticate once to BitlBee, and then BitlBee will reauth to all of the N other services I bridge to.

In my experience, here's how you should do it:

When I have to restart Bitlbee/WeeChat (which isn't often), I need exactly two commands to get my whole chat thing set up:

/connect bitlbee       # connects to my local bitlbee from weechat
identify <MYPASSWORD>  # logs into all my bitlbee accounts

That should be everything you need to do.

Totally incidentally, I run my BitlBee instance bound to on a remote server and then connect to it via a WeeChat client connected over the loopback interface. There are a bunch of other great CLI IRC clients out there (irssi) comes to mind, but the important thing is that if you have BitlBee bound on rather than you've vastly improved the security of the whole thing by requiring SSH authentication to the server in question to log into your chat accounts.

Large Data Files


Dealing with large data files is something of a more advanced programming topic; one that I've found that a lot of my younger coworkers don't know a lot about. I'd like to share a bit of wisdom on the topic in this file.

Let's say you have a file that is some reasonable fraction of the size of the memory on a computer. As a concrete example, let's say you have a 4GB JSON file on a machine with 8GB of RAM.

You might think it would be safe to do something like this with it in python:

with open('very_large_file.json') as large_file:
    data = json.load(large_file)

This is a valid Python program, but it's probably going to end disastrously. That JSON file is 4GB of text, but when it's loaded into Python it's going to turn into a lot of integer objects and string objects and list objects and dictionary objects and who knows what. Some of those objects might be pretty big. In C frequently an int is 4 bytes, but in Python that innocuous int object is stored in a PyObject * container that is quite a bit bigger. In my experience, if you do something like this you'll end up with some small integer multiple of the size of the original file in actual memory, so something like this might requie 8-12GB of memory to load in Python after all is said in done. Some other languages like JavaScript might have slightly better memory efficiency since JSON is part of their reason d'etre, but even then the string '4' is probably much smaller than the JSON number representation of 4. It all depends on exactly what data types you're storing, how much of the JSON file is whitespace and field delimiters, and a few other things, but a safe bet here is that the loaded JSON representation is going to be much larger than 4GB, and likely will exceed the total system memory and cause swapping.

Clearly things will be even worse if you have a data file that's much bigger than your resident RAM. For instance, say you still have your 8GB of memory and have to process a data file that's 100GB. The json.load or json.loads methods you'd get out of the box clearly aren't going to work at all in this case.

Also, I'm using JSON as an example here only because it's in vogue right now, but the same class of problems is true of XML and most other data interchange formats (YAML, ASN.1, Protobuf, Thrift, etc.).

This is also a problem when generating huge files. Suppose, hypothetically, that you have a SQL database and you want to generate all of it or some subset of it into JSON or XML dump or something like that. Relational databases are designed for handling large data sets much larger than resident memroy, so it's common to run be in a situation where a naive program will exceed all of the memory on the system and cause swapping or an OOM situation when generating a large data file.

Fortunately, a long time ago smart people thought of a way to handle this -- generative parsing. People generally use the Java/XML terminology here because parsing and generating big XML feeds used to be a big thing (and still is), but the model here applies to any programming language.

Two two dominant paradigms or parsing or generating a big XML/JSON/whatever are called the DOM or ElementTree model, and the SAX model. In particular SAX was I believe the name of a particular implementation of an event-driven XML parser/generator library.

In the DOM or ElementTree model you get an object that represents the whole document and you just do what you want with it. This is analagous to working with the DOM in browserside JavaScript. It's also what you get when you use json.loads or json.dumps in Python with JSON, or what you'd get with lxml.etree in Python with XML. It's definitely the easiest model to work with, because you just have a big object and access methods on it the usual way. This model is simple, but only works well when you have relatively small data sets.

The SAX model is totally different. The basic idea is that when parsing, you get a callback for each token encountered in the document. Keeping state is up to you. Same with generating a document: you call a bunch of methods that emit tokens, and the parser will handle turning that into valid JSON/XML/whatever. If the library you're using is in an object-oriented language like C++, Python, or Java you'll often create a subclass using inheritance and overload methods on the class, whereas in C you'll generally just register callbacks with function pointers. In either case the idea is pretty much the same.

I ran into this recently where I had to generate a huge data feed that is currently going to be about 1GB per data file, but potentially might be much larger in the future (up to 1TB for now). The hosts this script runs on actually have a fair amount of free memory, so I could have just done things the simple way; but as they say, "every byte is sacred", and I wanted my program to be able to generate and parse huge data files efficiently.

The first version of the program I wrote emitted all of the data into a CSV file. CSV is great because you can just emit one line at a time, so it's not memory hungry if you write your code iteratively. Same with parsing—you can parse one line at a time, so if you write your code correctly it will be efficient. However, I ran into a bunch of problems:

So I decided to encode the dump as a hierarchical JSON file. The format is basically like:

  'timestamp': 1443404160,
  'hostname': 'foo',
  'databases': {
    'foo': {
      'tables': {
        'bar': [...]
    'quux': {
      'tables': {
        'quux1': [...],
        'quux2': [...]

As you can see, it's easy to extend this to add more metadata, handle multiple databases, handle multiple tables, etc.

In C I've previously used YAJL. YAJL is awesome. It's really fast, and it's really easy to use. It also gives you a lot of knobs, so for instance you can control memory usage vs encoding/decoding efficiency.

In Python I didn't really find any good solutions, until I stumbled across yajl-py. Yajl-py is a really simple wrapper around YAJL that uses ctypes. If you've already uses YAJL, or if you're already used to SAX style parsers and generators it's a great solution, and you'll pick it up in minutes.

I have nothing but positive words to say about the yajl-py project, and the maintainer has been friendly and accepting as I've contributed a few small fixes and patches. I've also found the speed to be very good.

Note: I could have also used lxml for this project to instead generate XML, since lxml already has a SAX style interface, is very fast (it uses the libxml2 and libxml2 is itself very performant). I mostly decided to use JSON because I prefer it as a file format (at least for the type of data that I needed), a lot of younger programmers are more familiar with JSON, and I liked the control I got with choosing how the buffering and flushing would work with YAJL (although there may be an option for this in lxml, I didn't check too hard).



I just want to quote Kanye West from his most recent interview in Vanity Fair:

I think if Michelangelo was alive or Da Vinci was alive, there's no way that they wouldn't be working with shoes, as a part of what they work on. Definitely one of the things they'd work on would be shoes. I've gone three years without a phone. I don't go a day without shoes.

The Thrill Of Riding


There is one experience that I will never be able to explain to another human being—the thrill of riding a bicycle. Some people get it, and some people don't. If you get it, you know exactly what I'm talking about, and if you don't, well you don't.

My Background

I grew up in a middle class community in the East Bay called San Ramon. Most people who are from the Bay Area have heard of San Ramon, but don't know a lot about it. What you need to know about it is that it was mostly built up in the '70s as a bunch of conservative yuppies who couldn't afford to live in SF or the South Bay, and who didn't want to slum it up in Oakland/San Leandro/Castro Valley/etc. In 1983 SBC (formerly Southen Bell Corporation, now a part of AT&T) moved its headquarters to San Ramon. In 2011 Chevron moved their headquarters from San Francisco to San Ramon. Ironically, I'm probably about to move into the old Chevron Headquarters at 555 Market as part of my current job, which is going to be weird for me knowing so many people who's parents were big deals at Chevron in San Ramon.

So growing up in San Ramon, bikes and bike lanes are not really a thing. Everyone works and rides cars to Chevron, SBC (AT&T), or else commutes by car to tech and biotech jobs in the South Bay. So everyone has a car. Nearby there is the Iron Horse Trail which is something like 40 miles end to end, but is honestly pretty lame. It is flat the entire distance so no serious cyclist cares about it, and it's full of little kids on tricycles learning how to ride their bikes and taking up the entire space without regard for people like me trying to blast down at high speeds. It sucks. The hardcore cyclists are out ascending Mt Diablo, but Mt Diablo is classified as an HC Climb, and while I might be close to being able to do something like that now, I definitely wasn't even close when I was in high school.

The Scene

When I started college at UC Berkeley I bought my first "real" bike—a Surly Cross Check that I bought throught the awesome folks at Missing Link Bicycle Cooperative. That bike ended up getting stolen during a road trip I was on with a friend while I had it locked up outdoors from the (fucking awesome) India House Hotel which was in a seriously hood neighborhood of New Orleans. But that bike taught me the thrill of riding in real urban environments—down Shattuck and Telegraph in Berkeley, down Telegraph, San Pablo, MLK, and Adeline in Oakland, down one of my favorite streets in the world Peralta in Oakland, and many others. There's something just incredible weaving through city traffic, avoiding railroad tracks, avoiding 3'x'3' pot holes, and the whole thing while blasting throught at max speed. Plus, in Oakland you will see the craziest most legit graffiti pieces in industrial areas in West Oakland where you literally go from one block with a huge multi-day graffiti piece and nearby there are buildings with bullet holes in the windows. Literally the corner store I lived next to had bullet stores in its windows, and my nearby go-to burrito join had bullet-proof glass you had to slide money through to pay for your meal. I remember walking through that neighborhood and seeing an altercation 100 ft away from me where someone pulled out a hand gun at point blank range on another homie; and I remember getting the fuck out of there as fast as I could. This is also when I started carrying a pocket knife on me at all times. Riding through these neighborhoods during the day is a trip in itself, but being restless and blasting through the neighborhood with insomnia at 3am when no one else is out is fucking beyond incredible.

Around this time is when I started doing community rides like SF Critical Mass, East Bay Bike Party, SF Bike Party, and Midnight Mystery Ride. You will meet the most fun people, from super casual people just checking it out to hardcore fucked up 19 y/o kids attending dead end community colleges who are serious beasts in terms of strength and ride only brakeless fixed gear bikes.

I've had fun living in Berkeley, Oakland, and Pasadena/Los Angeles and going crazy in the streeths there, but by far the most fun on solo rides is the rides since I moved to San Francisco. In San Francisco you have to deal with serious fucking traffic, crazy crab drivers who don't give a shit about you, MUNI tracks everywhere that will fuck you up if you're not careful, and intense hills everywhere. I've gotten up pretty much every hill in the city (including Twin Peaks), and at this point I know if I'm going from point A to point B what's the fastest route, what's the least hilly route, which ways I can go that have bike lines or bike designated streets, and so on. Finally, after almost ten years of aggressive city riding, I finally know how to get out of the saddle and really throw my weight around to do crazy turns on a dime. If a car pulls in front of me and throws its breaks on, I know how to throw my weight around and put mad English on the bike to swerve around pretty much everything.

I fucking love my crazy commute every morning from Valencia down Market St, which is full of the most intense car weaving, snaking through car lines, and running lights you can imagine. But beyond that, sometimes at 11pm or 12am I'll get restless and decide to check out the night life scene in the city. I'll go down Mission, check out 16th St, bike down all of the shitty night "clubs" on or near Folsom, and then go up to Natoma to check out Tempest. Then I'll do my favorite thing which is check out Polk St which is all kinds of fucked up. Usually I come riding southbound from North Point all the way up Polk down to Market. When you top Polk, basically the whole way down until O'Farrell is shitty frat bro/sorority girl bars, and everyone is drunk as fuck and wandering out in the street, taxis, Ubers, Lyfts, and SFPD are going crazy weaving around trying to pick up people or arrest them, and it's insane on a bike. Once you hit O'Farrell, if you have no fear and really go HAM on your bike you can go 25+ mph and make every light from O'Farrell down to Market which is fun as hell and scary as fuck.

Blasting through the Sunset is amazing too. You can get onto Kirkham and it's a bike lane the whole way. And there's not a lot of traffic, but there are stop signs everywhere. Ifyou get into the right speed you can blast down Kirkham looking for oncoming cross-traffic and make crazy speed getting up and down hills. And that takes you into The Wiggle which is a whole separate class of weaving and scofflawing.

The point being—biking is a whole thrill in and of itself. When I'm doing more road biking I get really into my vertical elevation gain, and get really into how many vertical feet I'm doing a week. But the scene is so much fore than that. Endangering your life in intense city traffic is one of the most exhilirating things I can think of. It's a fucking blast.

If you're not a serious cyclist, and want to get into it, hit me up and I'll show you a good time.

BitlBee & HipChat


The chat system we use at work is HipChat. HipChat offers a web client (HTML and Javascript) as well as native clients for Windows, OS X, and Linux. It also offers an XMPP gateway, so if you don't want to use the native client you can use any other XMPP chat client. The XMPP gateway is a little hacky because HipChat has extended the XMPP protocol with proprietary attributes to add custom features, e.g. HipChat has both the concept of a display name (i.e. the user's full real name) as will as an @mention name that is used to alert people in chats. HipChat does not have a native IRC gateway.

I was really unhappy with the native Linux HipChat client for a number of reasons. I found it to be really slow and it used a ton of memory. It also hasn't gotten the same amount of attention as the other native clients, and lags behind in a number of areas. Besides that, I've been using IRC for years and I've already built up a nice workflow with weechat (and previously with irssi) that I wanted to keep using.

If you want to use weechat, irssi, or some other IRC client to connect to HipChat, it turns out there's a way—BitlBee provides an XMPP/IRC gateway. The way it works is BitlBee acts to you as an IRC server implementing the IRC protocol. On the backend it will translate between IRC and XMPP (or one of many other chat protocols like AIM, ICQ, Twitter, etc.).

About three years ago when I first tried using BitlBee with HipChat, it was really rough around the edges. It worked, but barely. There were a lot of problems with how it displayed user names and the workflow for adding and joining channel. Thankfully in the last few years this has gotten a lot better. This article will explain how to get set up with BitlBee and HipChat. Once you get everything working, you'll get a neat IRC setup like this (assuming you're using a console based client):

how it looks

Apologies for the impossibly small text, but you probably get the idea.

How To Use Bitlbee With HipChat

First I would recommend taking a look at the official docs here. This will give you an overview of the state of the world, and below I will provide some more specific advice as to the workflow I use.

If you're feeling adventurous, in the upstream git repo there is a branch called feat/hip-cat that gives you almost native support for HipChat. If you use this branch, when you connect to the server you'll see users show up with their IRC nick set to their @mention name. BitlBee will apply a mangling heuristic to room names to make a best guess as to what to name them (by lowercasing the room and removing spaces and special characters). What this means is if there's a HipChat room with a name like "Engineering" you'll probably be able to join it with an IRC command like /join #engineering, which is not true if you're in the master branch. I ran on this hip-cat for about six months (until just now), and while it mostly works it is rough around the edges. I found problems in the following areas:

Additionally, you'll find that the master branch gets a lot more commits made to it than the hip-cat branch. Due to these bugs, and the fact that I wanted to follow along with all of the latest stuff in master, I have switched to the master branch and it's what I recommend.

Setting It Up

First install BitlBee. I would recommend getting it from git and building it yourself, but that's obviously optional and if you want you can use a version packaged by your favorite distro. If you compile BitlBee from source, make sure that you have an appropriate /var/lib directory to store your profile. I had to manually create /var/lib/bitlbee and set it up with the correct permissions even though I configured BitlBee to use an alternate prefix (i.e. it seems not to respect a command like ./configure --prefix=$HOME/opt in this regard, and you'll find that the ./configure script is not a real autoconf script).

You can start the server with a command like:

bitlbee -D -p 6667 -i

This will daemonize BitlBee and have it bind to If this works successfully, fire up your IRC client, connect to the IP and port you set up, and you should see a greeting banner from BitlBee.

Once you've done this, the following sequence will set up a HipChat account for you:

account add hipchat you@yourcompany.com YOUR_HIPCHAT_PASSWORD
account hipchat on

Note that the BitlBee password does not have to be the same as your HipChat password, and in fact it's a good idea to make them different. If you get an error in the register command above about not being able to write a file (as I initially did), make sure that you have a /var/lib/bitlbee directory on your system and that it's writeable by the BitlBee process.

In the future when you connect to BitlBee you'll be able to re-authenticate by using:


This will log you in and restore all of the channels that you've set up.

Managing Channels

Now that you're logged in and you've created an account, it's time to add some channels. Go to YOURCOMPANY.hipchat.com/account/xmpp to get a channel list and the XMPP names. Let's say that a channel is named 9999_foo and you want it to be mapped locally to the IRC channel whose name will be #bar. To do that, you'd use the following command in the BitlBee control window:

chat add hipchat 9999_foo@conf.hipchat.com #bar

After this you should be able to /join #bar and join the channel.

In the future, you may want to delete channels you've created, change their configuration, set up auto-joining, etc. This is a little bit cumbersome. What you need to do here is to know the internal BitlBee channel id that was given to the channel. You can see that with:

channel list

This will print the list of channels and their numbers.

To delete a channel:

channel CHANNEL_ID del

To set a channel to auto-join:

channel CHANNEL_ID set auto_join true

Chatting With Users

Users will show up as their @mention name. If you need to know a user's real name, you can use the standard /whois command in IRC to get that information. I do not know how to do the opposite, that is search for a user's @mention name (i.e. IRC nick) based on their real name.

Once you know a person's name you can /msg nick in the usual way to chat with users.

Mission Bicycle w/ SRAM Automatix Hub


Today I ordered a new bicycle through MASH SF. I bought a fixed gear track bike built up from the Cinelli MASH Histogram frame. I'm going to write about this bike once I've had it for a while, but for now I want to write about the bike that the Histogram will be semi-replacing.

About a year and a half ago I bought a bike from Mission Bicycle Company. If you're not familiar with them, they make their own steel frames with horizontal dropouts which means that their bikes are built as single-speed/fixed gear or with internally geared rear hubs (in contrast to how most road bikes are built with a rear derailleur). I already had a Surly Pacer and was looking to build something a bit cooler and more unique. Thus, I built up this bike in a pretty unusual way. I asked them to build it with an SRAM Automatix hub. This is an interally geared rear hub with a "centrifugal clutch", and is the only such hub on the market that I know of. The way this works is when the rear wheel is spinning at about 11.5mph the rear wheel shifts "up" with a 137% step. These are pretty unusual—I ride a lot and check out other people's bikes somewhat obsessively as I'm riding (and walking) around, and since I got this bike I've only seen one other bike built up with this hub. Mine is built with a 46t front chainring and a 19t cog on the rear hub. This means that when the bike shifts up, it's similar to riding with a 46t/14t ratio or thereabouts. Since the shifting is automatic, there's no cabling to the rear wheel. I also built the bike with only a front brake, so the only cabling at all on the bike is a small cable going from the handlebars to the front brake caliper. The hub is really small (much smaller than other internally geared hubs you may have seen), and the small size of the hub and lack of cabling aesthetically makes it look like I'm riding a fixed gear or single-speed bike.

my bike

The rest of the bike was built with mid tier components and with bull handlebars. I got security bolts for the front wheel and security bolts for the seat. The whole thing cost something like $1300 with tax which is kind of pricey for an unbranded bike with two gears, but includes the price of the hub and a fancy seat. One change I did make a few weeks after buying the bike is to get the traditional pedals with steel toe clips swapped out for double-sided SPD mountain biking pedals. I thought I was going to want standard pedals so I could get around town with regular shoes, but I found that after riding for so long with SPD cleats on my road bike I just liked the feeling of being clipped in too much to go without. I just throw an extra pair of shoes in my backpack if I'm heading out (and at work I have an extra pair of regular shoes for the office).

The main function of this bike has been as a commuter bike, although I have done a few intense weekend rides (2000' elevation gain) on this bike. As a commuter bike, this bike is awesome. I'm really happy with it. Because it's not branded and is built up with security bolts everywhere, I feel like I can lock it up anywhere in the city with only the rear wheel/triangle locked. This means I only carry a single mini U-Lock and I haven't had any issues.

I can go surprisingly fast on this thing. I'm a pretty aggressive rider in general, so I'm always sprinting wherever I go, but that said I'm able to pass people in spandex on expensive carbon fiber bikes all the time on straight stretches. Over the distances you can typically go in the city between lights, the lack of gears isn't a problem at all, and actually when the higher gear is engaged the ratio is pretty high. I've timed my work commute on my road bike, and it takes me exactly the same amount of time to commute on both bikes (the commute time is mostly dependent on the lights I hit or don't hit). I have a weekend loop that I do, and the loop that I do from my apartment to the top of the Legion of Honor takes 40-45 minutes on my road bike and is about 700' of climbing. The loop takes about 50 minutes on the Mission Bicycles bike, and I'm able to get up from the bottom of Sea Cliff to the top of Legion of Honor which is a Category 4 climb on Strava. The steepest segment I've gotten up on this bike is the 43rd Ave Climb which is 275' and is an 11% grade on the steep block. My time on this bike on the 43rd Ave segment is actually only a few seconds off from my road bike, although subjectively the climb is a lot harder on the Automatix hub.

Overall I'm pretty happy with the Automatix hub, but there are a few minor issues with it. The first is that because it engages the higher gear at a particular speed, there are certain grades where it's really hard to not engage it. What I mean is that there are certain grades (5% or 6%? I'm not sure) where I naturally want to get out of the saddle to climb. However, getting out of the saddle delivers enough power that I end up hitting the speed threshold, and all of a sudden the climb is beyond my max power output. If this happens I have to stop pedaling for a fraction of a second and then try to re-engage the bike on the lower gear. I've never actually fallen this way, but I've gotten pretty close a few times, and it's pretty scary when it happens. To counteract this I end up having to be really careful about my cadence on certain hills which is annoying and inefficient. This tends to be an issue for me when navigating The Wiggle where I have to bike at a really unnatural speed unless I want to go all out and bike up in the high gear.

Another problem is that sometimes the gear can egnage unexpectedly at lower speeds. This typically happens when going over potholes or bumps in the road, especially when I'm close to the shifting speed. Again, this is particularly annoying on hills where it can be dangerous. Another thing I've noticed is that when going below the shifting speed, if you keep pressure applied to the cranks in just the right way you can keep the gear engaged in the higher gear basically all the way down to when you're stopped. This doesn't happen too frequently, but it's caught me off guard when coming to a rolling stop at stop signs. What happens is I'll get down to nearly 0mph, and then try to start pedaling with the cranks vertical. When the cranks are vertical I have almost no mechanical advantage, so I've almost fallen before when this happens.

The last issue is that the hub has the tendency to rattle when going over bumps/potholes. I haven't had any mechanical problems as a result of this, so I don't think it indicates a real problem, but it's definitely disconcerting. On a previous bike I once had the bottom bracket slip out when going over a pothole, and I occassionally get flashbacks of this happening, or worry that my rear wheel is going to slip out of the rear dropouts since a loose rear wheel can also cause rattling.

Despite these minor issues, I want to reiterate that I like this bike a lot, and I plan to keep it and probably still mostly commute on it. I will definitely continue to use this bike for around-town chores and when I have to go places where I'll have to lock my bike up outside. The only thing I'd really consider changing if I were to rebuild it is the lack of a rear brake. 99% of the time I don't need the additional braking power, but descending from the Legion of Honor or down Clipper from the top of Portola can be really terrifying since the front brake will get really hot and loud and it's literally my only braking mechanism.

Update (2015-10-15): I just checked, and the front chainring on my Mission Bicycles bike is a Sugino Messenger 46t. Changing the front chainring is incredibly easy (the only slightly complicated thing is you'll have to adjust your chain length) and will make the bike easier to ride if you're having difficult on hills, or will make it faster if you find that you're always on straightaways or going downhill and are spinning too fast.

I have nothing but good words to say about the Sugino Messenger series, and will be sticking with it. The Sugino Messenger is their mid-tier series (higher end is the "Sugino Zen" series). You can change the front chainring to have more or fewer teeth if you want to trade off speed for climbing/starting power. One thing to note is that the Sugino Messenger 46t is 130mm BCD (a.k.a. PCD) where as the bigger more professional track chainrings tend to be 144mm BCD. This BCD number is how far out from center the cranks and the chainring are drilled for the screws that attach the two pieces. If the diameter is off, you won't be able to attach the crank to you chainring because the screw slots won't line up.

For instance you can actually get 44t 144mm BCD front chainrings which are a smaller gear than what I have but attach to the cranks at wider diameter. If you had 130mm BCD cranks you wouldn't be able to attach the cranks to a 144mm BCD chainring, even if the cranks themsleves were much longer (say 175mm). If you're confused check out Sheldon Brown's authoritative site, or ask anyone at your LBS for 60 seconds of their time to explain to you how it works.

The Commodification of Databases


There's a conversation that I've had a few times in the last ~10 years that I've been a software engineer, that goes something like this. Databases are known to be a hard problem. Databases are the central scalability bottleneck of many large engineering organizations. At the same time, major tech companies like Google and Facebook have famously built in-house proprietary databases that are extremely robust, powerful, and scalable, and are the lynchpins of their business. Likewise, both open source and commercial database offerings have been rapidly evolving to meet the scalability problems that people face. So the question is: will we reach a point where one or a small number of open source or commercial databases become so good, so powerful, and highly available, and so horizontally scalable that other solutions will fall by the wayside? That is, will we converge on just a few commodified databases?

A number of smart people I've worked with seem to think that this is the case. However, I'm skeptical, or think that if it is the case it will be a very long time coming.

The first part of my reasoning here hinges on why the database world has become so fragmented today. When the first relational databases came to market in the late seventies and early eighties they actually came to market in co-evolution with the SQL standard. The first commercial SQL database on the market was from Oracle in 1979 followed very closely by SQL databases from other companies like IBM (who published their first SQL database in 1981). The reason that these companies came to market with SQL databases at nearly the same time is because the SQL standard had been a number of years in the making at that point. EF Codd published his seminal paper "A Relational Model of Data for Large Shared Data Banks" in 1970, and throughout the 1970s people at IBM and elsewhere had been designing the SQL standard.

For at least 25 years SQL reigned supreme as the de facto way to query and interact with large databases. The standard did evolve, and there were always other offerings, especially in the embedded world or in special purpose applications like graph databases or hierarchical databases. But the reality is that for several decades, when people were talking about large scale databases they were nearly always talking about large scale SQL databases.

However, if you want to build a really big highly partitioned database, SQL is not a great way to go. The relational data model makes partitioning difficult—you generally need application specific knowledge to efficiently partition a database that allows joins. Also, in many cases the SQL standard provides too much functionality. If you relax some of the constraints that SQL imposes there are massive performance and scaling wins to be had.

The thing is, while SQL certainly has a lot of defficiencies, it's at least a standard. You can more or less directly compare the feature matrix of two relational databases. While every benchmark has to be taken with a grain of salt, the fact that SQL is a standardized language means it's possible to create benchmarks for different SQL databases and compare them that way. The standardization of SQL is a big part of the reason why it's kept hold for so long.

Non-relational databases and NewSQL databases like Riak, Cassandra, HBase, CockroachDB, RocksDB, and many others all take the traditional SQL database model and modify it in different ways, in some cases drastically. This means these databases are hard to directly compare to each other because their features and functionality differ so much.

There are also orders-of-magnitude improvements to be had by relaxing certain durability guarantees and by having certain special purpose data structures. In the most extreme example, a database that consists of an append-only log can max out hard drives on write speed even though such a database would be infeasibly inefficient to query. You could still think of such a thing as a database, and this is similar to what something like syslog or Scribe or Kafka is. Likewise, a database that consists of a properly balanced hash table can provide extremely efficient reads for single keys at the cost of sacrificing the ability to do range queries and at the cost of potentially expensive rebalancing operations. For instance, in the most extreme example a read-only database like cdb can do reads with one disk seek for misses and two disk seeks for successful lookups. There are so many different tradeoffs here in terms of durability gurantees, data structures, read performance, write performance, and so on that it's impossible to prove that one database is more efficient than another.

Even in more general purpose databases that do log-structured writes and can efficiently perform range queries, the specifics of how you do writes, how you structure your keys, the order you insert keys, how pages are sized, etc. can make huge differences in efficiency.

One final thing to consider on this efficiency aspect. Missing indexes can turn operations that should be blazing fast into ridiculously efficient table scans. It's easy to completely saturate the I/O on a hard drive (even an SSD) with a few concurrent table scans. Some of the smarter databases will prevent you from doing this altogether (i.e. they will return an error instead of scanning), but even then it's easy to accidentally write something tantamount to a table scan with an innocent looking for loop. In many cases it's straightforward to look at a query and with some very basic cardinality estimates come up with a reasonable suggestion index for that query, so in theory it's possible to automatically suggest or build appropriate indexes. However, one can just as easily run the risk of having too many indexes, which can be just as deleterious to database performance. Automatically detecting the database-query-in-a-for-loop case is certainly possible but not trivial. In other words, there is no silver bullet to the indexing problem.

The point is, once you move beyond SQL all bets are off. If we as an industry were to standardize on a new database query protocol it might be possible to return to a somewhat normal world where it's possible to compare different database products in a sane way—but today it's not. Even internally at Google, which is famous for probably having the most state of the art database products, there isn't a single database used—different products there are built on different technologies like BigTable, Megastore, Spanner, F1, etc.

Maybe one day we will reach the point where servers are so cheap and so powerful that people can have massive petabyte scale databases with microsecond lookups on commodity hardware. Then maybe the hardware will truly be so fast that the specifics of the software implementation will be irrelevant (although even then, the datastructures will have different running complexity and therefore benchmarks will still show "big" wins for one technology or another). However, I think that those days are quite a ways off. In the meantime, there will be a lot of companies that need terabyte and petabyte scale databases, there will be a lot of small startups that evolve into tech giants, and there will be a need to build special, custom purpose databases.

I'm not prescient, and the computer industry is one where long term predictions are notoriously difficult. However, at this point databases have been considered a difficult problem for more than fifty years. If nothing else, purely based on the longevity of the challenge thus far I think it's reasonable to assume that it will still be considered a challenging problem in computer science at least a decade from now, if not a few decades from now.



Today I wrote some unit tests for the code that does the static html/xml generation for this blog. I was motivated to do this after my friend James Brown pointed out some bugs he had noticed on the site.

To add the tests, I had to significantly refactor things. Previously the whole thing was a single 251 line Python script. To add the tests, I had to refactor it into an actual Python module with different components, create a setup.py file with console_scripts entry points, create a requirements.txt file, set up pytest, and so on. The tests validate a bunch of things and do crazy things with lxml and XPath queries to do validation of the blog content about what things should and should not be present in the generated files. All in all, the refactored code is a lot easier to test and reason about, but it's also a lot more complicated which is a bit unfortunate.

The reason I expended this this considerable amount of work instead of just dropping in the original one-liner fix is is what one might consider a security or privacy bug. A while back I had written some content that I wanted to share with friends, but that I didn't want linked anywhere or crawled. I figured a clever solution here would be to reuse the blog generation stuff I already have here so I'd get the nice CSS and whatnot, but just add a mode that would cause these pages to not be linked from the index page. I never created a robots.txt file for these pages since by nature of its existence such a file publicizes the secret URLs.

This all worked great, except for one little bug. When I generate the static site content, I also generate a file /index.rss which is picked up by RSS readers for feed syndication. The code generating the RSS file didn't know about the hidden page feature, so these hidden pages ended up in the RSS feed. I didn't notice this since I don't subscribe to the RSS feed for my own site. As a result of this bug, not only was the content visible to people browsing via RSS, it was also actually indexed by Googlebot. I was able to confirm this by doing a Google query with site:eklitzke.org my-specific-search-term. Interestingly, these pages were not indexed by Yahoo or Bing which suggests to me that the crawling backend for Google is unified with their RSS crawler, whereas the same is not true of Yahoo/Bing.

Besides fixing the root bug, all pages I generate in this way (i.e. not linked from the main index page) now specifically use meta noindex feature just in case they are ever linked to again. This is functionally similar to a robots.txt file but doesn't publicize the URLs. I also registered my site with the Google webmaster tools and explicitly requested that they take down the URLs I didn't want indexed.

All is good now. I guess the moral of the story is that for any program that is even remotely interesting, it's worth spending a bit of time to write tests. And hat tip to James for reporting the issue in the first place.

Rendering Videos From OpenGL Is Hard


Lately I've been getting into OpenGL since I love looking at awesome animations and visualizations, and want to start making my own. After a few false starts, I've settled on writing these programs in C++. Besides the great performance that C++ has, C++ seems to have the widest array of bindings to different parts of the graphics and math ecosystem. In C++ I have native bindings to all of the usual C libraries, plus great C++ libraries like GLM that are high performance, actively maintained, and tailored for graphics programming. The stuff I'm doing right now isn't stressing the capabilities of my graphics subsystem by any stretch of the imagination, but I have ambitions to do more high performance stuff in the future, and having familiarity with the C++ OpenGL ecosystem will come in handy then.

The downside is that it's kind of hard to share these programs with others. This is one area where WebGL really shines—you can just send someone a link to a page with WebGL and Javascript on it and it will render right there in their browser without any futzing around. On the other hand, I've seen a lot of videos and animated gifs on Tumblr that are fairly high quality, so I thought that perhaps I could figure out one of these techniques and share my animations this way.

As it turns out, this is surprisingly difficult.

Animated GIFs

I've seen a lot of awesome animations on Tumblr using animated GIFs. For instance, this animation and this animation are pretty good in my mind. After I looked into this more, I realized that these images are very carefully crafted to look as good as possible given the limitations of GIF, and that this is a really bad solution for general purpose videos.

The most important limitation of GIF is that it's limited to a color palette of 256 colors. This means that for images with fine color gradients you need to use dithering which looks bad and is kind of hard to do anyway. It is possible in GIF to use what is called a "local color table" to provide a different 256 color palette for each frame in the animation, so in a global sense you can use more than 256 colors, but within each frame pixels are represented by 8 bits each and therefore you're limited to 256 colors.

Besides the color limitation, generating animated GIFs is pretty difficult. I was pleased at first to find GIFLIB, but if you look at the actual API it gives you it's incredibly low level and difficult to use. For instance, it can't take raw RGB data and generate a GIF automatically: it's your job to generate the color palette, dither the input data, and write out the raw frame data.

There are a few other libraries out there for working with GIF images, but the main alternative, and I suspect what most people are using, seems to be ImageMagick/GraphicsMagick. What you would do in this model is generate a bunch of raw image frames and then stitch them together into an animation using the convert command. There are some great documents on how to do this, for instance this basic animation guide and this optimized animation guide. However, once I really started looking into this it started seeming rather complicated, slow, and weird.

The other thing that I realized is that the good looking GIF animations I was seeing on Tumblr are mostly images that are carefully stitched together in a loop. For instance, if you look at the images I posted previously on a site like GIF Explode you'll see that there's only a small number of frames in the animation (20-30 or so). This is a lot different from posting a 10 second animation at 24fps which will be 240 frames, potentially each with their own local color table.

As a result of these limitations, I decided to abandon the GIF approach. If I do any animations that can be represented as short loops I will probably revisit this approach.

HTML5 Video

The other option I explored was generating an mp4 video using the ffmpeg command line tool. This is an attractive option because it's really easy to do. Basically what you do is call glReadPixels() to read your raw RGB data into a buffer, and then send those pixels over a Unix pipe to the ffmpeg process. When you invoke ffmpeg you give it a few options to tell it about the image format, color space, and dimensions of the input data. You also have to tell it to vertically flip the data (since it has the opposite convention of OpenGL here). The actual invocation in the C++ code ends up looking something like:

FILE *ffmpeg = popen("/usr/bin/ffmpeg -vcodec rawvideo -f rawvideo -pix_fmt rgb24 -s 640x480 -i pipe:0 -vf vflip -vcodec h264 -r 60 out.avi", "w");

Then data can be sent to the file object in the usual way using fwrite(3) to send over the raw image data. After the pipe is closed the ffmpeg process will quit, and there will be a file out.avi with the encoded video.

This actually generates an output file that looks pretty decent (although the colors end up looking washed out due to some problem related to linear RGB vs. sRGB that I haven't figured out). There are definitely noticeable video encoding artifacts compared to the actual OpenGL animation, but it doesn't seem too unreasonable.

The problem here is that when I upload a video like this to Tumblr the video ends up getting re-encoded, and then on the Tumblr pages it's resized yet again in the browser and the animation looks really bad.

In the actual animation, a frame will look like this:

a great image

When I encode the video and send it up to Tumblr, it ends up looking like like this. If you look at the source mp4 file on Tumblr it's definitely a lot worse than the reference image above, but it's not as bad as the way it's rendered on the Tumblr post page.

I may end up sticking with this technique and just hosting the files myself, since my source videos are of good enough quality (better than the re-encoded source mp4 file above), and I don't actually have that many visitors to this site so the bandwidth costs aren't a real issue.

tl;dr Posting high-quality OpenGL animations to the internet is hard to do, and I'm still trying to figure out the best solution.

Linear Algebra; or, The Folly of Youth


I was a really poor student in college. In fact, I was such a poor student that I ended up dropping out of school. I don't regret at all not having degree—it hasn't hurt me one bit, and I've had tons of great career opportunities despite my lack of diploma. However, I do frequently regret a lot of the amazing learning opportunities that I passed up.

This pain has been felt most poignantly by me when I look back on the survey course of Linear Algebra that I took at UC Berkeley, Math 110. This was a class that I took because it was a requirement from my major. It was also supposed to be the "easiest" upper division math class at Cal, and thus it was the first upper division class I took that had real proofs. I spent more time grappling with my inability to understand or create proofs than I did with the actual material. That might have been OK, but it was a subject I didn't particularly care and therefore I kind of fumbled through the class from week to week and finished the class without having really learned very much at all. Later on, when I had gotten the learning-how-to-learn-things aspect of math down, I did a lot better and actually came away from my other classes with a lot more knowledge.

As it turns out, knowing a little bit of linear algebra goes a long way when it comes to a wide range of fields in computer science. For instance, it is at the root of signal processing and computer graphics. It comes up in other places too. Recently for fun I wrote an n-body simulator. Guess what? All of those vector quantities like position and velocity that you can mostly ignore when doing analytic solutions in your college physics classes are really important when you're writing a computer simulation. Now I've been trying to get into graphics programming with OpenGL, and despite the fact that the type of work I'm doing is mostly orthographic projections, there are still vectors and matrices popping up all over the place. Linear algebra is also really helpful if you want to do number crunching on the GPU, since GPU shaders have all of this dedicated hardware for vector and matrix processing. All stuff I'm coming to learn the hard way.

Surprising Things Found While Exploring Bash


A few days ago I was making some changes to my .bashrc file and noticed a few interesting things regarding bash aliases and functions.

In my actual .bashrc file I had only the following lines that were related to setting up aliases:

alias grep='grep --color=auto'
alias ls='ls --color=auto'

if which vim &>/dev/null; then
    alias vi=vim

But here's what I got when I typed alias:

$ alias
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias vi='vim'
alias which='(alias; declare -f) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot'
alias xzegrep='xzegrep --color=auto'
alias xzfgrep='xzfgrep --color=auto'
alias xzgrep='xzgrep --color=auto'
alias zegrep='zegrep --color=auto'
alias zfgrep='zfgrep --color=auto'
alias zgrep='zgrep --color=auto'

Weird, right? The only ones I had defined in my .bashrc were the aliases for grep, ls, and vi. Well, it turns out that my distribution has already decided to add the --color=auto stuff for me for ls, which is pretty reasonable, and I found bug 1034631 which is the origin of all of the weird grep variants automatically aliased for me. That seems a little weird, but I do understand it. However, I do find it amusing that the obscure ls variant vdir isn't colorized even though it is part of coreutils and supports colorization (perhaps I should file an RFE).

But WTF is going on with that which alias? Actually, what's going on here is pretty neat. This alias pipes the list of defined aliases and bash functions defined in the shell to which so you can can see where these come from. And if I run declare -f on my system, I see that there's actually a lot of stuff in there:

$ declare -f | wc -l

A couple of those are functions I defined in my own .bashrc file, but that accounts for perhaps 50 of those nearly 2500 lines of shell script.

Nearly all of the functions that I see listed by declare -f appear to be functions that are installed for bash command completion. There are some real gems in here. Check out this one:

_known_hosts_real ()
    local configfile flag prefix;
    local cur curd awkcur user suffix aliases i host;
    local -a kh khd config;
    local OPTIND=1;
    while getopts "acF:p:" flag "$@"; do
        case $flag in 
    [[ $# -lt $OPTIND ]] && echo "error: $FUNCNAME: missing mandatory argument CWORD";
    let "OPTIND += 1";
    [[ $# -ge $OPTIND ]] && echo "error: $FUNCNAME("$@"): unprocessed arguments:" $(while [[ $# -ge $OPTIND ]]; do printf '%s\n' ${!OPTIND}; shift; done);
    [[ $cur == *@* ]] && user=${cur%@*}@ && cur=${cur#*@};
    if [[ -n $configfile ]]; then
        [[ -r $configfile ]] && config+=("$configfile");
        for i in /etc/ssh/ssh_config ~/.ssh/config ~/.ssh2/config;
            [[ -r $i ]] && config+=("$i");
    if [[ ${#config[@]} -gt 0 ]]; then
        local OIFS=$IFS IFS='
' j;
        local -a tmpkh;
        tmpkh=($( awk 'sub("^[ \t]*([Gg][Ll][Oo][Bb][Aa][Ll]|[Uu][Ss][Ee][Rr])[Kk][Nn][Oo][Ww][Nn][Hh][Oo][Ss][Tt][Ss][Ff][Ii][Ll][Ee][ \t]+", "") { print $0 }' "${config[@]}" | sort -u ));
        for i in "${tmpkh[@]}";
            while [[ $i =~ ^([^\"]*)\"([^\"]*)\"(.*)$ ]]; do
                __expand_tilde_by_ref j;
                [[ -r $j ]] && kh+=("$j");
            for j in $i;
                __expand_tilde_by_ref j;
                [[ -r $j ]] && kh+=("$j");
    if [[ -z $configfile ]]; then
        for i in /etc/ssh/ssh_known_hosts /etc/ssh/ssh_known_hosts2 /etc/known_hosts /etc/known_hosts2 ~/.ssh/known_hosts ~/.ssh/known_hosts2;
            [[ -r $i ]] && kh+=("$i");
        for i in /etc/ssh2/knownhosts ~/.ssh2/hostkeys;
            [[ -d $i ]] && khd+=("$i"/*pub);
    if [[ ${#kh[@]} -gt 0 || ${#khd[@]} -gt 0 ]]; then
        if [[ "$awkcur" == [0-9]*[.:]* ]]; then
            if [[ "$awkcur" == [0-9]* ]]; then
                if [[ -z $awkcur ]]; then
        if [[ ${#kh[@]} -gt 0 ]]; then
            COMPREPLY+=($( awk 'BEGIN {FS=","}
            /^\s*[^|\#]/ {
            sub("^@[^ ]+ +", ""); \
            sub(" .*$", ""); \
            for (i=1; i<=NF; ++i) { \
            sub("^\\[", "", $i); sub("\\](:[0-9]+)?$", "", $i); \
            if ($i !~ /[*?]/ && $i ~ /'"$awkcur"'/) {print $i} \
            }}' "${kh[@]}" 2>/dev/null ));
        if [[ ${#khd[@]} -gt 0 ]]; then
            for i in "${khd[@]}";
                if [[ "$i" == *key_22_$curd*.pub && -r "$i" ]]; then
        for ((i=0; i < ${#COMPREPLY[@]}; i++ ))
    if [[ ${#config[@]} -gt 0 && -n "$aliases" ]]; then
        local hosts=$( sed -ne 's/^['"$'\t '"']*[Hh][Oo][Ss][Tt]\([Nn][Aa][Mm][Ee]\)\{0,1\}['"$'\t '"']\{1,\}\([^#*?%]*\)\(#.*\)\{0,1\}$/\2/p' "${config[@]}" );
        COMPREPLY+=($( compgen -P "$prefix$user"             -S "$suffix" -W "$hosts"—"$cur" ));
    if [[ -n ${COMP_KNOWN_HOSTS_WITH_AVAHI:-} ]] && type avahi-browse &> /dev/null; then
        COMPREPLY+=($( compgen -P "$prefix$user" -S "$suffix" -W             "$( avahi-browse -cpr _workstation._tcp 2>/dev/null |                  awk -F';' '/^=/ { print $7 }' | sort -u )"—"$cur" ));
    COMPREPLY+=($( compgen -W         "$( ruptime 2>/dev/null | awk '!/^ruptime:/ { print $1 }' )"        —"$cur" ));
    if [[ -n ${COMP_KNOWN_HOSTS_WITH_HOSTFILE-1} ]]; then
        COMPREPLY+=($( compgen -A hostname -P "$prefix$user" -S "$suffix"—"$cur" ));
    __ltrim_colon_completions "$prefix$user$cur";
    return 0

Huh? It turns out that when the functions are loaded into bash they're stripped of comments, so the declare -f output makes it look a bit more cryptic than it really is. If you look at the bash-completion source code, you'll see that the original is actually well commented, and that the other weird functions showing up in my shell (including a bunch of non-underscore-prefixed functions like quote and dequote) come from this package. It's still fun to look at. You know you're in for a treat whenever you see IFS being redefined. You can see a lot of fun things like hacks for the lack of case-insensitive regular expressions in awk and sed, and the rather remarkable fact that avahi can be used for host discovery from bash. From this code I also learned about the existence of bash regular expressions.

It's also interesting to see just how many commands commands have completion, and how much code has gone into all of this:

$ complete -p | wc -l

$ find /usr/share/bash-completion -type f | wc -l

$ find /usr/share/bash-completion -type f | xargs wc -l
40545 total

Wow. That's a lot of code. Who wrote it all? Does it all work? Is everything quoted properly? Are there security bugs?

There's even amusing things in here like completion code for cd. That's right, the shell builtin cd has custom completion code. You might think that to complete cd you just have to look at the directories in the current working directory... not so. The builtin cd has two important parameters that can affect its operation, the CDPATH variable (which defines pathes other than the current working directory that should be searched), as well as the shopt option called cdable_vars which allows you to define variables that can be treated as targets to cd. The bash-completion package has code that handles both of these cases.

A lot has already been written about how Unix systems, which used to be quite simple and easy to understand, have turned into these incredibly baroque and complicated systems. This is a complaint that people particularly levy at Linux and GNU utilities, which are perceived to be particularly complex compared to systems with a BSD lineage. After finding things like this, it's a hard to disagree. At the same time, this complexity is there for a reason, and I'm not sure I'd want to give any of it up.

Cars And Turn Restrictions On Market St In San Francisco


I'm a little bit late to this issue, but on June 16, 2015, the SFMTA approved changes in a plan called the "Safer Market St Project" to limit the movement that private cars can make on Market St in San Francisco. SFGate has some good coverage of the issue here and here. I want to share my thoughts on the matter.

The origin of this legislation is something called Vision Zero SF, which is a group that is trying to reduce traffic deaths in SF; their declared goal is to achieve zero traffic deaths in SF by 2024. This is an ambitious goal. San Francisco is a city with nearly a million permanent residents, and during week days when people from around the Bay Area commute into the city the population regularly exceeds that number.

Whether or not you believe it's possible to achieve the Vision Zero goal, what is interesting is to look at statistics on which intersections in the city have the most deaths. This is particularly the case for deaths caused by cars hitting pedestrians or cyclists, since anyone who walks or bikes regularly in the city knows that there are certain areas where road layout, lights, turn restrictions, etc. encourage cars to engage in behavior that is dangerous to pedestrians and cyclists. These are the areas where we can most effectively make changes or traffic restrictions that will save lives. What Vision Zero has found is that a hugely disproportionate percentage of fatalities take place on a few blocks of Market St. According to the document I just linked to, a portion of SOMA that accounts for 12% of San Francisco's streets accounts for 70% of total crashes. This is consistent with similar (and older) data from the San Francisco Bicycle Coalition on the matter regarding accidents regarding cyclists. The SFMTA did analysis on the matter and gave this public hearing presentation which outlines their research on the matter and the proposed turn restrictions (which ended up actually being implemented in June).

Opponents of the legislation pointed out that restrictions of any kind on traffic on Market St will likely lead to higher congestion on nearby streets and generally higher congestion in SOMA in general. It's hard to deny this theory: these restrictions will have a serious impact on drivers and their ability to navigate in the area around Market St. And yet, that's kind of the whole point of the legislation. The legislation is being passed to save the lives of pedestrians and cyclists, not to decrease congestion.

I was disappointed in how a lot of the legislation I read portrayed the matter as one between cars and cyclists. As you can see in the Vision Zero literature, there are many more pedestrian related deaths than cycling deaths. The report I linked to earlier showed that in 2014 there were 17 pedestrians killed in traffic related deaths, nine motorists killed in traffic related deaths, and only three cyclists killed. They do point out that San Francisco is one of the few cities in the country where cycling related deaths are increasing, but I think the fact that there are more than five times as many pedestrian fatalities speaks for itself.

As someone who rides a bicycle up and down Market St every week as part of my work commute, I strongly support the legislation that the SFMTA passed. I hope that they are able to move ahead with similar legislation in the future. Market St is incredibly dangerous to bike down, and I have frequent close encounters with cars on Market St. My personal take on the matter is that of the encouters I have with motorized vehicles, very seldom do I have problems with MUNI buses or taxis. Most of Market St in SOMA already has designated lanes for buses and taxis that puts them in the center lane away from cyclists. Market St is especially dangeorus due to the divided lanes for bus stops. This lane division was not originally anticipated when Market St was built, and consequently the right-most lane is very narrow in many places and does not have sufficient space for cyclists. Additionally, many pedestrians who are getting on or off MUNI will cross the right-most lane to get to the divided bus stop area, and in doing so cross into the way of traffic. The restrictions that the SFMTA passed apply primarily to private vehicles that are in the rightmost lane or turning into the rightmost lane, right where they're the most likely to hit pedestrians and cyclists.

Being Self-Taught


Something I've been thinking about a lot recently is this curious aspect of the profession of programming—what is it that makes some programmers better than others? Obviously this is a complicated question, and a full analysis is outside of the scope of a short blog post (not that I have all the answers anyway).

However, one thing I've been thinking about recently is this weird thing about programming which is that it is almost entirely self-taught. A lot of the people I work with, including many of the brightest ones working on the hardest problems, don't have a formal training in "computer science". Most of them have college degrees in other disciplines. But by the same token, a lot of them don't have college degrees at all.

From talking to peple who actually did pursue a degree in "computer science" from a university, frequently they did not learn at all in a classroom about any of the following things:

To be successful in the field as a programmer you have to know how to do most or all of the above.

The thing that's really interesting to me is that despite this, it's hard for me to think of any profession that's easier to get into. In fact, it's probably more correct to say: because of this it's hard for me to think of any profession that's easier to get into. Everyone in this field is primarily self-taught.

There is more content on the internet about programming than almost any other field I can think of. If you browse the front page of Hacker News regularly you will find a constant stream of new and high quality content about interesting programming related articles. Stack Overflow is full of what I can only guess must be millions of questions and answers, and if you have a new question you can ask it. For those who take it upon themselves to learn how to use an IRC client, there are hundreds or thousands of public programming-related IRC channels where you can ask questions and get feedback from people in real time.

I frequently get questions from people who asked me how I learned about X. How did I learn what the difference is between a character device and a block device? How did I learn about how virtual memory systems work? How did I learn about databases? The answer is the same as how almost every programmer learned what they know: it's mostly self taught, I spent a ton of time reading things on the internet, and I've spent a lot of time in front of a computer writing code and analyzing the runtime behavior of programs.

Another important and related observation to this is that while there is definitely a correlation between intelligence and programming ability and success, from what I have seen the correlation is a lot weaker than you might think. There are a ton of exceptionally brilliant people that I know who work in the field of computer programming who are poor programmers or who aren't successful in their career. Usually these are people who have decided to focus their interests on something else, who have settled down into a particular niche and haven't branched out of it, or who aren't interested in putting in the hard work that it would take to improve as a programmer-at-large.

It definitely helps a lot to work with smart people. It definitely helps a lot to talk and interact with other people, and to get code reviews and feedback from them. But in my estimation, the lion's share of what there is to learn comes from the amount of work that a person spends alone in front of a computer with a web browser, a text editor, and a terminal. I hope that more people who want to become computer programmers or who want to improve take this into consideration.

Resource Management in Garbage Collected Languages


I was talking to a coworker recently about an issue where npm was opening too many files and was then failing with an error related to EMFILE. For quick reference, this is the error that system calls that create file descriptors (e.g. open(2) or socket(2)) return when the calling process has hit its file descriptor limit. I actually don't know the details of this problem (other than that it was "solved" by increasing a ulimit), but it reminded me of an interesting but not-well-understood topic related to the management of resources like file descriptors in garbage collected languages.

I plan to use this post to talk about the issue, and ways that different languages work around the issue.

The Problem

In garbage collected languages there's generally an idea of running a "finalizer" or "destructor" when an object is collected. In the simplest case, when the garbage collector collects an object it reclaims that object's memory somehow. When a finalizer or destructor is associated with an object, in addition to reclaiming the object's memory the GC will execute the code in the destructor. For instance, in Python you can do this (with a few important caveats about what happens when the process is exiting!) by implementing a __del__() method on your class. Not all languages natively support this. For instance, JavaScript does not have a native equivalent.

While the Javascript language doesn't let you write destructors, real world JavaScript VMs do have a concept of destructors. Why? Because real JavaScript programs need to do things like perform I/O and have to interact with real world resources like browsee assets, images, files, and so forth. These constructs that are provided in the context like a DOM or in Node, but are not part of the Javascript language itself, are typically implemented within these VMs by binding against native C or C++ code.

Here's an example. Suppose that you have a Node.js program that opens a file and does some stuff with it. Under the hood, what is happening is that the Node.js program has created some file wrapper object that does the open(2) system call for you and manages the associated file descriptor, does async read/write system calls for you, and so on. When the file object is garbage collected the file descriptor associated with the object needs to be released to the operating system using close(2). There's a mechanism in V8 to run a callback when an object is collected, and the C++ class implementing the file object type uses to to add a destructor callback will handle invoking the close(2) system call when the file object is GC'ed.

A similar thing happens in pretty much every garbage collected language. For instance, in Python if you have a file object you can choose to manually invoke the .close() method to close the object. But if you don't do that, that's OK too—if the garbage collector determines that the object should be collected, it will automatically close the file if necessary in addition to actually reclaiming the memory used by the object. This works in a similar way to Node.js, except that instead of this logic being implemented with a V8 C++ binding it's implemented in the Python C API.

So far so good. Here's where the interesting issue is that I want to discuss. Suppose you're opening and closing lots of files really fast in Node.js or Python or some other garbage collected language. This will generate a lot of objects that need to be GC'ed. Despite there being a lot of these objects, the objects themselves are probably pretty small—just the actual object overhead plus a few bytes for the file descriptor and maybe a file name.

The garbage collector determines when it should run and actually collect objects based on a bunch of magic heuristics, but these heuristics are all related to memory pressure—e.g. how much memory it thinks the program is using, how many objects it thinks are collectable, how long it's been since a collection, or some other metric along these lines. The garbage collector itself knows how to count objects and track memory usage, but it doesn't know about extraneous resources like "file descriptors". So what happens is you can easily have hundreds or thousands of file descriptors ready to be closed, but the GC thinks that the amount of reclaimable memory is very small and thinks it doesn't need to run yet. In other words, despite being close to running out of file descriptors, the GC doesn't realize that it can help the situation by reclaiming these file objects since it's only considering memory pressure.

This can lead to situations where you get errors like EMFILE when instantiating new file objects, because despite your program doing the "right thing", the GC is doing something weird.

This gets a lot more insidious with other resources. Here's a classic example. Suppose you're writing a program in Python or Ruby or whatever else, and that program is using some bindings to a fancy C library that does some heavy processing for some task like linear algebra, computer vision, machine learning, or so forth. To be concrete, let's pretend like it's using bindings to a C library that does really optimized linear algebra on huge matrices. The bindings will make some calls into the C library to allocate a matrix when an object is instantiated, and likewise will have a destructor callback to deallocate the matrix when the object is GC'ed. Well, since these are huge matrices, your matrices could easily be hundreds of megabytes, or even many gigabytes in size, and all of that data will actually be page faulted in and resident in memory. So what happens is the Python GC is humming along, and it sees this PyObject that it thinks is really small, e.g. it might think that the object is only 100 bytes. But the reality is that object has an opaque handle to a 500 MB matrix that was allocated by the C library, and the Python GC has no way of knowing that, or even of knowing that there was 500 MB allocated anywhere at all! This happens because the C library is probably using malloc(3) or its own allocater, and the Python VM uses its own memory allocator). So you can easily have a situation where the machine is low on memory, and the Python GC has gigabytes of these matrices ready to be garbage collected, but it thinks it's just managing a few small objects and doesn't GC them in a timely manner. This example is a bit counterintuitive because it can appear like the language is leaking memory when it's actually just an impedance mismatch between how the kernel tracks memory for your process and how the VM's GC tracks memory.

Again, I don't know if this was the exact problem that my coworker had with npm, but it's an interesting thought experiment—if npm is opening and closing too many files really quickly, it's totally possible that the issue isn't actually a resource leak, but actually just related to this GC impedance mismatch.

Generally there's not a way to magically make the GC for a language know about these issues, because there's no way that it can know about every type of foreign resource, how important that resource is to collect, and so on. Typically what you should do in a language like Python or Ruby or JavaScript is to make sure that objects have an explicit close() method or similar, and that method will finalize the resource. Then if the developer really cares about when the resources are released they can manually call that method. If the developer forgets to call the close method then you can opt to do it automatically in the destructor.

C++ Solution

C++ has a really elegant solution to this problem that I want to talk about, since it inspired a similar solution in Python that I'm also going to talk about. The solution has the cryptic name RAII -- Resource Acquisition Is Initialization. In my opinion this is a really confusing name, since most people really care more about finalization than initialization, but that's what it's called.

Here's how it works. C++ has a special kind of scoping called block scoping. This is something that a lot of people think that syntactically similar languages like JavaScript have, since those languages also have curly braces, but the block scoping concept is actually totally different. Block scoping means that variables are bound to the curly brace block they're declared in. When the context of the closing curly brace is exited, the variable is out of scope. This is different from JavaScript (or Python) for instance, because in JavaScript variables are scoped by the function call they're in. So a variable declared in a for loop inside of a function is out of scope when the loop exits in C++, but it is not out of scope until the function exits in JavaScript.

In addition to the block scoping concept, C++ also has a rule that says that when an object goes out of scope its destructor is called immediately. It even further goes to specify the rules about what order the destructors are invoked. If multiple objects to out of scope at once, the destructors are called in reverse order. So suppose you have a block like.

  Foo a;
  Bar b;

The following will happen, in order:

This is guaranteed by the language. So here's how it works in the context of managing things like file resources. Suppose I want to have a simple wrapper for a file. I might implement code like this:

class File {
  File(const char *filename) :fd_(open(filename, O_RDONLY)) {}
  ~File() { close(fd_); }

  int fd_;

int foo() {
  File my_file("hello.txt");
  return baz();

(Note: I'm glossing over some details like handling errors returned by open(2) and close(2), but that's not important for now.)

In this example, we opened the file hello.txt, do some things with it, and then it will automatically get closed after the call to baz(). So what? This doesn't seem that much better than explicitly opening the file and closing it. In fact, it would have certainly been a lot less code to just call open(2) at the start of foo(), and then had the one extra line at the end to close the file before calling baz().

Well, besides being error prone, there is another problem with that approach. What if bar() throws an exception? If we had an explicit call to close(2) at the end of the function, then an exception would mean that line of code would never be run. And that would leak the resource. The C++ RAII pattern ensures that the file is closed when the block scope exits, so it properly handles the case of the function ending normally, and also the case where some exception is thrown somewhere else to cause the function to exit without a return.

The C++ solution is elegant because once we've done the work to write the class wrapping the resource, we generally never need to explicitly close things, and we also get the guarantee that the resources is always finalized and that that is done in a timely manner. And it's automatically exception safe in all cases. Of course, this only works with resources that can be scoped to the stack, but this is true a lot more often than you might suspect.

This pattern is particularly useful with mutexes and other process/thread exclusion constructs where failure to release the mutex won't just cause a resource leak but can cause your program to deadlock.

Python Context Managers

Python has a related concept called "context managers" and an associated syntax feature called a with statement.

I don't want to get too deep into the details of the context manager protocol, but the basic idea is that an object can be used in a with statement if it implements two magic methods called __enter__() and __exit__() which have a particular interface. Then when the with statement is entered the __enter__() method is invoked, and when the with statement is exited for any reason (an exception is thrown, a return statement is encountered, or the last line of code in the block is executed) the __exit__() method is invoked. Again, there are some details I'm eliding here related to exception handling, but for the purpose of resouce management what's interesting is that this provides a similar solution to the C++ RAII pattern. When the with statement is used we can ensure that objects are automatically and safely finalized by making sure that finalization happens in the __exit__() method.

The most important difference here compared to the C++ RAII approach is that you must remember to use the with statement with the object to get the context manager semantics. With C++, RAII typically is meant to imply that the object is allocated on the stack which means that it is automatically reclaimed and there's no chance for you to forget to release the context.

Go Defer

Go has a syntax feature called defer that lets you ensure that some code is run when the current block is exited. This is rather similar to the Python context manager approach, although syntactically it works a lot differently.

The thing that is really nice about this feature is that it lets you run any code at the time the block is exited, i.e. you can pass any arbitrary code to a defer statement. This makes the feature incredibly flexible—in fact, it is a lot more flexible than the approach that Python and C++ have.

There are a few downsides to this approach in my opinion.

The first downside is that like with Python, you have to actually remember to do it. Unlike C++, it will never happen automatically.

The second downside is that because it's so flexible, it has more potential to be abused or used in a non-idiomatic way. In Python, if you see an object being used in a with statement you know that the semantics are that the object is going to be finalized when the with statement is exited. In Go the defer statement probably occurs close to object initialization, but doesn't necessarily have to.

The third downside is that the defer statement isn't run until the function it's defined in exits. This is less powerful than C++ (because C++ blocks don't have to be function-scoped) and also less powerful than Pyton (because with statements can exit before the calling function).

I don't necessarily think this construct is worse than C++ or Python, but it is important to understand how the semantics differ.


Javascript doesn't really have a true analog of the C++/Python/Go approaches, as far as I know. What you can do in Javascript is to use a try statement with a finally clause. Then in the finally clause you can put your call to fileObj.close() or whatever the actual interface is. Actually, you can also use this approach in Python if you wish, since Python also has the try/finally construct.

Like with Go defer statements, it is the caller's responsibility to remember to do this in every case, and if you forget to do it in one place you can have resource leaks. In a lot of ways this is less elegant than Go because the finalization semantics are separated from the initialization code, and this makes the code harder to follow in my opinion.

My Philosophy on "Dot Files"


This is my philosophy on dot files, based on my 10+ years of being a Linux user and my professional career as a sysadmin and software engineer. This is partially also based on what I've seen in developer's dotfiles at the company I currently work for, which has a system for managing and installing the dotfiles of nearly 1000 engineers.

When I started using Linux, like every new Unix user I started cribbing dot files from various parts of the internet. Predictably, I ended up with a mess. By doing this, you get all kinds of cool stuff in your environment, but you also end up with a system that you don't understand, is totally nonstandard, and is almost always of questionable portability.

In my experience this is less of a problem for people who are software engineers who don't have to do a lot of ops/sysadmin work. A lot of software engineers only do development on their OS X based computer, and possibly a few Linux hosts that are all running the exact same distro. So what happens is if they have an unportable mess, they don't really know and it doesn't affect them. That's great for those people.

When you start doing ops work, you end up having to do all kinds of stuff in a really heterogenous environment. It doesn't matter if you work at small shop or a huge company, if you do any amount of ops work you're going to admin multiple Linux distros, probably various BSD flavors, and so on. Besides that (or even if you have a more homogeneous environment), you end up having to admin hosts that are in various states of disrepair (e.g. failed partially way through provisioning) and therefore might as well be different distros.

Early on, the (incorrect) lesson I got out of this was that I needed to focus on portability. This is really hard to do if you actually have to admin a really heterogeneous environment. For a few reasons. For a starter, even the basic question of "What kind of system am I on?" is surprisingly hard to answer. The "standard" way to do it is to use the lsb_release command... but as you would guess, this only works on Linux, and it only works on Linux systems that are recent enough to have a lsb_release command. If you work around this problem, you still have the problem that it's easy to end up with a huge unreadable soup of if statements that at best is hard to understand, and frequently is too specific to really correct anyway. You might think that you could work around this by doing "feature testing", which is actually the right way to solve the problem, but this is notoriously hard to do in a shell environment and can again easily make the configuration unreadable or unmaintainable.

It gets even worse for things like terminal based emulators. The feature set of different terminal emulators like xterm, aterm, rxvt, and so on varies widely. And it gets even more complicated if you're using a "terminal multiplexer" like screen or tmux. God forbid you try to run something in a vim shell or Emacs eshell/ansi-term. Trying to detect what terminal emulator you're under and what features it actually supports is basically impossible. Even if you could do this reliably (which you can't because a lot of terminal emulators lie), the feature set of these terminal emulators has varied widely over the years, so simply knowing which terminal emulator you're using isn't necessarily enough to know what features it supports.

As I became a more seasoned Linux/Unix user, what I learned was that I should try to customize as little as possible. Forget those fancy prompts, forget the fancy aliases and functions, and forget the fancy 256-color terminal emulator support. The less you customize the less you rely on, and the easier it becomes to work on whatever $RANDOMSYSTEM you end up on. For a number of years the only customization I would do at all was setting PS1 to a basic colorized prompt that included the username, hostname, and current working directory—and nothing else.

Recently I've softened on this position a bit, and I know have a reasonable amount of configuration. In the oldest version of my .bashrc that I still track with version control (from 2011, sadly I don't have the older versions anymore), the file had just 46 lines. It has a complicated __git_ps1 function I cribbed from the internet to get my current git branch/state if applicable, sets up a colorized PS1 using that function, and does nothing else. By 2012-01-01 this file had expanded to 64 lines, mostly to munge my PATH variable and set up a few basic aliases. On 2013-01-01 it was only one line longer at 65 lines (I added another alias). On 2014-01-01 it was still 65 lines. At the beginning of this year, on 2015-01-01 it was 85 lines due to the addition of a crazy function I wrote that had to wrap the arc command in a really strange way. Now as I write this in mid-2015, it's nearly twice the size, at a whopping 141 lines.

What changed here is that I learned to program a little more defensively, and I also got comfortable enough with my bash-fu and general Unix knowledge. I now know what things I need to test for, what things I don't need to test for, and how to write good, portable, defensive shell script. The most complicated part of my .bashrc file today is setting up my fairly weird SSH environment (I use envoy and have really specific requirements for how I use keys/agents with hosts in China, and also how I mark my shell as tainted when accessing China). Most of my other "dot files" are really simple, ideally with as little configuration as possible. Part of this trimming down of things has been aided by setting up an editor with sensible defaults: for real software engineering stuff I use Spacemacs with a short .spacemacs file and no other configuration, and for ops/sysadmin stuff I use a default uncustomized vi or vim environment.

Which brings me to the next part of this topic. As I mentioned before, the company I work at has nearly 1000 engineers. We also have a neat little system where people can have customized dot files installed on all of our production hosts. The way it works is there's a specific git repo that people can clone and then create or edit content in a directory that is the same as their Unix login. The files they create in that directory will be installed on all production hosts via a cron that runs once an hour. A server-side git hook prevents users from editing content in other user's directories. This system means that generally users have their dot files installed on all hosts (with a few exceptions not worth going into here), and also everyone can see everyone else's checked in dot files since they're all in the same repo.

People abuse this system like you would not believe. The main offenders are people who copy oh-my-zsh and a ton of plugins into their dot files directory. There are a few other workalike systems like Bashish (which I think predates oh-my-zsh), but they're all basically the same: you copy thousands of lines of shell code of questionable provenance into your terminal, cross your fingers and hope it works, and then have no idea how to fix it if you later encounter problems. Besides that, I see a ton of people with many-hundreds-of-lines of configuration in their bash/zsh/vim/emacs configuration that are clearly copied from questionable sources all over the internet.

This has given me a pretty great way to judge my coworkers' technical competency. On the lowest rung are the gormless people who have no dot files set up and therefore either don't give a shit at all or can't be bothered to read any documentation. Just above that are the people who have 10,000 lines of random shell script and/or vimscript checked into their dot files directory. At the higher levels are people who have a fairly minimal setup, which you can generally tell just by looking at the file sizes in their directory.

If you want to see what differentiates the really competent people, here are a few things I sometimes look for:

LaRouche PAC


So the LaRouche PAC has been posted up the last few weeks at 3rd & Market, with a bunch of aggravating signs about Greece, Glass-Steagall, and so forth. In classic LaRouche fashion, they have these stupid posters with Obama sporting a Hitler moustache and other nonconstructive inflammatory things designed to provoke people and pull them into discussions/arguments so they can espouse their conspiracy theories.

I had a friend in college (UC Berkeley) that joined and left the LaRouchies multiple times, and dropped out of school as a result. They are a really scary cult. I want to write about that story, so people realize how fucked up they are.

Basically what would happen is my friend would start going to the LaRouche discussion meetings in the evenings that are open to the public, and would get really cauhgt up in that. Then he'd start also going on their weekend retreats from time to time. The way these work, you go out to some cabin or something in the middle of nowhere, you don't have your cell phone or a way to contact the outside world, and they have this highly structured schedule where you talk about LaRouche politics and ideology all weekend for 16 hours a day. Then since he was spending all of this time in the evening meetings and weekend retreats, he'd stop going to classes and would spend time canvassing with the group. At one point, he had effectively moved out of the Berkeley co-op he was in and was living in some house that they had somewhere in Berkeley or Oakland that they let people stay in who are sufficiently dedicated to the cause (and who are spending some minimum amount of time canvassing, donating money, or doing who knows what else).

I remember the first time he joined the group. At first he was telling me about this cool group that had these interesting math/geometry discussions, and didn't mention it was the LaRouche movement. Maybe he didn't even know at first. Apparently there's some sort of shadow organization where they'll do these tracks focused on less political stuff like math/philosophy, and then they try to use that to get you to start going to the more "philosophy" focused discussions, and then that leads into the actual political arm of the organization which is their real interest. He'd be telling me about how they were doing these interesting Euclidean geometry discussions, and talking about logic, and somehow this math/geometry thing was related to the general concept of rationality, reasoning, and higher level thought and philosophy. Anyway, I was like "yeah that sounds cool maybe I'll check it out some time" and never went for whatever reason, I guess just because it sounded too weird to me. Then over the course of the semester, he started telling me about more of the stuff they were discussing, and started getting into the politics of it. At the time I was fairly up to date with what was going on with national and international politics, but not nearly as knowledgeable as someone who spends all day reading/talking about this stuff, so we'd get into these discussions where he'd be espousing these weird views about whatever was the topic of the moment and I would just be like "OK, whatever, clearly I don't know as much about this issue as you but I still disagree—I'm not going to debate you on this, let's talk about something else."

Then basically the last time I hung out with my friend that semester, we were walking around talking about stuff, and he started telling me this really crazy shit about how Lyndon LaRouche actually was controlling the Democratic Party, and somehow also had Congress and the GOP under his thumb, and all of this really out there stuff. Lyndon LaRouche would issue these internal memos where he'd be predicting various political/economic events, most of which either sounded to me very vague or were not substantiable. From my point of view, the things he predicted correctly would be used to "prove" that he was controlling the Democratic Party or whatever, and then for the stuff that didn't come to fruition there would be some excuse about how something had changed at the last minute and LaRouche had to steer in a different direction. My friend didn't realize how bullshit this was. I was just like WTF I don't even know how to explain how crazy this is, and didn't really see him for a few weeks after that.

The next time I saw him was during the finals week of that semester. I was going down into the UC Berkeley main stacks (read: huge campus library) to do some studying or whatever. So I randomly run into him, and we're talking and he asked me what I was doing down there. So I was like "Uh..... I'm studying for finals... why else would I be down in main stacks during finals week?" and during the ensuing discussion I realized that he didn't even know that it was finals week at school, and had completely stopped going to classes or following anything related to actually being a student at UC Berkeley.

What happened is he failed all of his classes that semester and was put on academic probation. His parents found out, because they were paying his tuition and rent and whatnot. They found out about the LaRouche stuff and freaked out, and they got him to take a semester off of school, live at home with them, and they got him out of the cult. He basically came to his senses, realized that the LaRouche thing was ruining his life, and decided to quit the movement and go back to school again.

The next semester that he was actually back in school we start hanging out again and he filled me in on what happened, how the LaRouche movement is a cult (duh, I had already figured it out by this point), and all of that stuff. But these people from the LaRouche movement kept calling him. We'd be hanging out and he'd get a phone call and be like "hold on, I need to take this", and then he'd spend an hour talking to the person about how he had left the group and wasn't interested in going to their meetings. I don't know why he didn't just stop taking the phone calls, or hang up immediately, but somehow he'd always get dragged into a long discussion.

Predictably what happened is at some point he ended up going to one of their meetings, didn't tell me or his parents about it, and got dragged right back into the cult. Then he stopped going to his classes again and cut off contact with me and the other few friends he had (although looking back, I think I might have been his only friend outside of the LaRouche movement at the time). Then he got kicked out of school since he had failed all of his classes one semester, took a semester off, and then failed all of his classes again. His parents found out and lost their shit again. My friend and his parents were from Bulgaria (I think he had come over to the United States in middle school), and they got him to somehow move to Bulgaria so that he could actually get the fuck out of the LaRouche cult and try to get a job or go to school there. I'm not really sure of the details because he had deleted his Facebook (or maybe that's when I had deleted mine), so I didn't really keep in touch. I did hear a few years later that he was still in Bulgaria, so I think things worked out.

tl;dr Fuck the LaRouche movement. It's a fucked up cult. Do not try to talk with them or engage with them, it's a waste of your time. The best thing you can do is ignore them, and if you see anyone reading their literature tell them they're a fucking cult. There's a bunch of this stuff documented on the internet. Usually the Wikipedia articles are informative on the matter, but the LaRouchies have been in a multi-year edit war with Wikipedia trying to remove any damaging facts about their organization, so what's on Wikipedia is not necessarily trustworthy at any given moment.

An Important Difference Between mysql(1) and MySQLdb


I keep forgetting about this thing, and then every six to twelve months when I have to do it again, I waste a bunch of time rediscovering it. It's important enough that I'm going to blog it.

If you're used to using PostgreSQL, you'll know that with Postgres you can connect over the local AF_UNIX socket using peer authentication. This means that as the evan user I can automagically connect to the evan database without a password. Likewise, to become the Postgres superuser, I simply need to do sudo -u postgres psql. This works using some magic related to either SO_PEERCRED or SCM_CREDENTIALS which let you securely get the credentials of the other end of a connected AF_UNIX socket.

MySQL also has a local AF_UNIX socket, and you can use this socket to make connections to MySQL. This is pretty handy, and for many reasons you may prefer to connect to MySQL over the local socket rather than using a TCP connection to localhost.

However, MySQL does not do the peer authentication thing. It doesn't matter if you're the root user connecting over a local socket. If the root user is configured to require a password (which is what I strongly recommend), then you must supply a password, even if you have sudo privileges on the host.

Fortunately, there's an easy workaround here that prevents you from having to type the root password all the time if you're doing a lot of MySQL administration. When you use the mysql CLI program, it will look for a file called ~/.my.cnf and use it to look up various connection settings. In particular, in this file you can set a default user and password. So let's say you've done this nice thing and made a file called /root/.my.cnf that has the root user's MySQL credentials, and you have the file set to mode 600 and all that and everything is great. You can type sudo mysql and you won't have to supply the root MySQL password (just possibly the root sudo password).

Here is a really important thing to know: the behavior of reading ~/.my.cnf is something that the mysql CLI program implements, it is not something implemented by libmysqlclient.so!

What that means is that when you are writing some script to frob MySQL using Python and MySQLdb, this will not work:

conn = MySQLdb.connect(unix_socket='/run/mysqld/mysql.sock',

You might think that if you ran this script as the root user, it could authenticate. Not so. Instead what you want is this:

conn = MySQLdb.connect(unix_socket='/run/mysqld/mysql.sock',

By the way, using the read_default_file option like this is definitely the best way to authenticate to MySQL from Python in general. You should not be putting database passwords in your Python projects---neither in your source code, nor in your project configs. By using a file in the filesystem like this you can move all of the database credentials into Puppet/Chef/whatever and secure the files so that most users can't read them. It may not seem like a big win today, but a few years later, when you're given the task of auditing everything for passwords, knowing that passwords have only lived in your configuration management software is going to help a lot.

How To Be An Urban Cyclist—Part 1


This blog series is going to explain to my advice on being an urban cyclist. The difficulty I've seen with other people is that while a lot of people know how to ride a bike, they may not feel comfortable riding in heavy traffic, on poorly paved roads, or in poorly lit ares. These posts are based on my experience the last six or seven years of my life cycling mostly around Berkeley, Oakland, San Francisco, and Los Angeles.

The first post in the series will cover what kind of bicycle I recommend, and what kind of gear you need to ride.

First you should have a well maintained bicycle. If you're buying a new bicycle, I strongly recommend getting a road bike with drop bars rather than a cheapo mountain bike. Road bikes are simply a lot faster, and if you don't feel fast you're not going to want to bike. There's nothing more frustrating than seeing people whiz by you on their bikes while you're struggling on yours. Simply put: if you don't feel good on your bike, you're not going to use it.

You can get a decent used steel frame bicycle in the Bay Area for $500-$600 or cheaper, depending on exactly what size frame you need, what type of components you want, etc. If you live elsewhere, you can probably get one cheaper. A decent new road bike will be something like $1000 or more if you want to get really fancy. If you're buying a new bike, I'm a big fan of Surly Bikes, but there's nothing wrong with getting a used bike. If you get a used bike, make sure you ride it and test that it can shift smoothly and brake quickly.

You should get and wear a helmet. You'll easily exceed speeds of 20 mph on your bike, and even in dense urban areas cars frequently exceedp speeds of 30 mph or more. For a comparison, falling off the top of a two story building entails an impact of about 20 mph. At 20 mph, much less at higher speeds, you can very easily die in a head on collision.

Next, make absolutely sure that you have both a front and rear light if you're going to light in any kind of low light conditions. Riding in the dark without a light is incredibly dangerous, because you'll be moving quickly, be hard to see, and be making very little noise. I like the silicone lights that don't require any mounting gear that you can put on and off your bike really easily (mine are "Snugg" brand and cost $15 for a pair on Amazon). These are great for riding around and being seen. However, they're not going to illuminate the road in front of you. If you plan on biking in really dark areas you'll want a bigger/brighter clip on light—I'd recommend the ones that are 1 watt or higher power output (most of the ones in the store will be 0.5 watts, which isn't ideal). Make sure you always remove your lights when locking your bike outdoors.

For locks, at the minimum you need a U-lock and cable lock.[1] The U-lock will lock your rear wheel and frame, the cable will lock your front wheel. Note that all of the cables you buy can be cut fairly easily (in a few minutes perhaps); the point of the cable is to deter someone from stealing the front wheel (which is fairly cheap), the U-lock is what will actually be securing your frame. I highly recommend the 5" Mini Kryptonite U-Lock. The 5" locks are not only the smallest ones, but they're also the most secure. U-locks can be easily broken by someone with a jack, if there's enough space to get the jack in between the bars of the lock to bend it. The 5" locks don't admit enough space for someone with a jack to get a hold on the lock. However, you'll really need an adequate rack to lock your bike with a 5" lock. For instance, it's generally not possible to lock your bike to a parking meter with a 5" lock whereas you can with a larger size. When you lock your bike, you need to place the U-lock so that it secures the rear wheel through the rear triangle of the bike. You generally should not directly lock the frame. By locking the rear wheel through the rear triangle, the U-lock is actually going through both the frame and the rear wheel (although it may not look like it!). The cable loops through the front-wheel and back around the U-lock.

In areas with high rates of bike theft, such as San Francisco, you'll need some way to secure your seat as well. I biked and locked my bike outdoors for years in Berkeley, Oakland, and Los Angeles and never had a problem with seat theft. As soon as I started biking in San Francisco, I got my seat stolen twice in the course of a month (both times having left the bike alone for less than an hour). So whether or not you need this really depends on where you live. Bike stores will sell special locks for seats. You can keep the lock on the seat all of the time because you'll only need to remove it in the rare situations when you need to adjust the seat height. If you don't like the look of a seat lock, or want to spend less money, you can also try securing the seat post bolt by using security bolts or hot gluing a BB into the bolt head.

If you're going to ride in the rain, I strongly recommend a detachable rear fender. Otherwise you're going to get a muddy butt. I've never found a front fender to be necessary; if it's rainy enough to need one, you're going to get drenched anyway.

[1] If you have security bolts for your front wheel, you can probably omit buying and carrying a cable lock.

On Not Having A LinkedIn Account


I don't have a LinkedIn account, which some people find to be a bit strange. I'd like to talk a bit about that.

As a software engineer with an awesome job, I really do not need a constant barrage of recruiter spam. Here are the specifics:

My experience with LinkedIn is I'd get a torrential inflow of recruiter spam (i.e. "Join our HOT VC-backed stealth startup!!!") that wasn't useful to me at all.

Worse, I found that some people would "stalk" me on LinkedIn before coming in for job interviews. As in, I'd go in to a job interview, and someone would mention something about my past that they had looked up on LinkedIn. This has happened once with my Twitter account too, which is even creepier.

Since LinkedIn provides no value to me and is yet-another-way-to-track-me, I don't have an account with them. EZPZ.

Final Conflict—Ashes To Ashes


I was recently turned on to Final Conflict's seminal album Ashes To Ashes from this Pitchfork album review. The album review made the album sound awesome, and I'm pretty into some of the other acts from the 80's LA/OC hardcore scene (e.g. Black Flag, TSOL, Adolescents), so I had to check it out.

Put simply, this album is fucking great. I personally have a strong preference for the hardcore sound (i.e. compared to thrash/black/heavy metal) because that's the shit I grew up on, so even though the whole scene was a bit before my time I get nostalgic for it. That said, there are some pretty prominent metal influences in this album that clearly place the album in the late 80s. For instance, the track Abolish Police features some awesome wailing guitar sections not as common in the earlier hardcore stuff (but seen for example in the later Black Flag material). Some of the tracks like Shattered Mirror strongly evoke the sound of some other LA/OC acts like TSOL or Adolescents; in particular, this track reminds me of some of the tracks from the Adolescents' debut album. There are some awesome samples of Reagan-era political speeches on tracks like Political Glory and The Last Sunrise.

tl;dr if you're into hardcore stuff, check this album out.



I added an RSS feed to this "blog", again using Python's excellent lxml module. This ended up being really convenient because of the way I was already using lxml to generate the articles from markdown. There's a method .text_content() on the lxml html nodes, so I can already take the marked up article content and extract the text content from it. Thus, the generator script (lovingly called generate.py) ends up being a thin wrapper that generates HTML from the markdown files, then does some element tree gymnastics, and magically out of this comes an HTML element tree that's rendered as the blog content itself, and an RSS element tree.

tl;dr magic happens and then blog.

Cloud Nothings—Attack On Memory


Right now this album is my favorite thing. Especially the first two tracks, holy shit.



Those of you who know me well know that while my music interests are varied, lately (as in, the past few years) I've mostly been listening to hip hop music. I wanted to do a review of a new album I've been really into lately that isn't a hip hop album. That album is Sunbather by Deafheaven.

I found this album somehow by stumbling across links on Pitchfork. I think I was checking out some bands I had found on Vimeo, and a Deafheaven link came up at the bottom of one of the pages. Anyway, I saw the Pitchfork album review, saw that it was rated well and read the album description, and I decided to check out the album. It's an incredibly easy album to get into because the opening track, Dream House, is so powerful. It's very atmospheric with fast-paced guitars and percussion, and very emotive-but-subdued "screamo" vocals. The next track, Irresistible, blends in perfectly with the first track and provides a really nice contrast; it is a very melodic entirely instrumental track. The album generally follows this pattern of a long black metal/emo/screamo track usually followed by a shorter more melodic track.

I can't really do the full album review the same justice as the experts can, so I refer you to the already linked Pitchfork review, as well as The Needle Drop's album review.

What I really love about this album is how accessible and melodic it is, and yet how emotive and powerful a lot of the tracks are. I don't listen to a lot of black metal (which is I guess how the band labels themselves), and I think black metal is generally a somewhat inaccessible genre for outsiders. Yet I was able to pick this album up really easily. This may because the album is non-traditional to the genre, but I like it.

I'm especially excited because I'm attending the Pitchfork Music Festival in July, and I found out (having already bought tickets) that Deafheaven will be performing there. I'm looking forward to seeing them live!

Hello World


I made a simple static site generator for my new blog incarnation. The generator works using Markdown and lxml to generate sites. I am not using any normal templating tools like jinja or mustache.

Since I think it's kind of interesting, articles are structured as separate files in a directory, and an article itself looks like this:

metadata, e.g. the article date
more metadata

blog content starts here

In other words, there is a preamble section of metadata, a blank line, and then the actual markdown text. I parse the metadata, generate HTML using the Python markdown module, and then transform that into an lxml element tree. The lxml element tree is munged to insert the metadata (e.g. the article date).

I decided on this format because

Mostly I intend on using this space to talk about music, bicycles, computers, life, work, and all of that good stuff.