One of the things I set up and got working recently was operating a
full node on the Bitcoin network. There were
two motivations for me for doing this.
The first is that I was generally interested in learning more about Bitcoin and
scripting the way that I interact with Bitcoin. Running a full node and using
bitcoin-cli client is a good way to do both of these things; it’s
also beneficial to the Bitcoin community as a whole.
The second is that after the recent near death of my phone I realized
that I’m better with backing up and scripting tasks on servers than I am at
doing the same with my phone. Previously I would write down my Mycelium backup
codes on a post-it note and then lose the post-it note. By contrast writing an
automated cron job that makes a copy of my wallet, encrypts it, and then puts it
in a private S3 bucket is easy, cheap, and has good security. There are some
good arguments that using a physical copy of the wallet can be more secure, but
in practice the way I organize my life I think that th solution of dumping and
encrypting a wallet on a Unix server using cron is easier. I also think that it
would be harder for an attacker to get a login on one of the Unix servers I
administer (which only allow non-root key-based SSH logins with a single key
that I am good at securing) than it would if they had my phone (which has
encryption at multiple layers, but all based around short numeric PINs that are
easy for me to remember).
Additionally, it seems plausible that it would be easier (and funner) for me to
write and automate things for managing multiple wallets and the money in them
myself rather than relying on the solution the Mycelium developers come up with.
As an example, recently I wanted to experiment with
JoinMarket. Doing this on a
server is easy, doing this with a phone-based wallet would not be.
I looked into a couple of options for running a full node:
- using my laptop
- buying a small fanless mini-ITX server to put in my apartment
- using a cloud provider like AWS or GCE
I determined that I didn’t want to use my laptop mostly because of the overhead
of running bitcoind. It uses more disk space, RAM, and network bandwidth than I
was happy with for something that will always be on on my laptop. Particularly
on the last point, I’d only want to run bitcoind when I was on a good network
link and that would mean trying to write complicated scripts to govern if
bitcoind should be running or not based on the perceived quality of my internet
The mini-ITX server in my apartment thing seemed cool at first, but once I
specced it out it didn’t really make sense. I wanted to build a fanless system
capable of running RAID 1 (arguably this is overkill, but whatever) with
sufficient storage. I specced something out with
this motherboard using a RAID 1 with
mSATA connectors, and after buying the motherboard, CPU, RAM, power supply,
disks, and enclosure it worked out to something like $800. Arguably such a
system is overkill (e.g. if backups are being made to S3 then RAID is not
necessary, and w/o RAID a cheaper motherboard would be used) but that was the
system that was compelling to me.
By contrast I looked at using a
g1-small machine type on
GCE. For the same cost as the mini-ITX server I could run an instance type like
this with storage and whatnot for something like two years. Plus with the GCE
instance it would be easier to do things like OS or hardware upgrades (boot a
new machine, copy the wallet over, power off the old VM). Based on my personal
experiences running home servers and running machines on GCE, I felt like the
GCE solution would be as reliable or more reliable than something connected to
the internet/power in my apartment unit.
If you’re really obsessed with security the AWS/GCE solution is probably a
nonstarter, since it means you have your wallet on a machine not in your
physical posession, which could be subpoenaed by law enforcement, etc. For me,
however, the security model of GCE is sufficient with balancing my convenience
and my actual real world security profile.
After I decided to go with the GCE route, I spun up a few g1-small images and
played around with them until I got to a happy state. I tried using Debian
Jessie, Fedora, and Ubuntu 14.04 LTS. The Ubuntu LTS is by far the easiest to
get working because you can install bitcoind via a PPA. You can compile bitcoind
on Jessie or Fedora easily enough, but it’s a fair amount of work and among
other things involves building and linking against a specific version
non-packaged version of BerkeleyDB 4.8. I also found that compiling bitcoind
uses a lot of memory due to the style of C++ coding they’re using, and I was
worried that perhaps one day you’ll need even more memory to compile and link
bitcoind which would necessitate an instance type upgrade for something that is
a relatively infrequent task. Plus, if I was running a bitcoind I built from
scratch it would be my job to keep it up to date, whereas installing packages
out of a PPA from someone I trust who maintains it is a lot easier. So as much
as I love dislike Ubuntu, I ended going with Ubuntu LTS.
After getting the image running and installing bitcoind, I found that it takes a
few days to sync the block chain using this instance type. The limiting factor
in syncing the block chain on this instance type is the CPU. The g1-small
instance type only has 1 CPU which is anemic and can’t even close to keep up
with the bandwidth on the instance. When I have to spin up another instance for
upgrades I will probably figure out how to copy over the block chain to the new
instance, because this would surely be much more efficient than resyncing the
entire block chain from scratch.
Once the bitcoin block chain is synced, the bitcoind process uses about 300 MB
of RAM when running and uses a small amount of CPU (usually < 1%, but it will
burst much higher from time to time.) My 15 minute load average seems to be
about 0.10 or less which includes all of the other tasks on the system.
I haven’t written any interesting scripts yet, but doing so seems to be really
bitcoin-cli command prints all of its output to stdout as JSON, so
consuming its output from Python should be trivial.
One final thing to note is that all of these solutions have non-trivial cost and
complexity. Using a lightweight wallet on your laptop or phone is much cheaper
and easier than running a full node. What I’ve described here is more akin to
something out of a cypherpunk’s wet dream than it is a practical solution for
people who just want to partake in the Bitcoin ecosystem.
 As I’ve alluded in other blog posts, I have a pretty whacky non-standard SSH
system in place that I’ve built using envoy. I
will blog about this sometime since I think it solves some interesting problems.