How to build FreeBSD/EC2 images
I have been building FreeBSD/EC2 images for the past three years, and based on the email I have been receiving, most people have been either using these images directly or modifying them to create images which suit their needs. However, there are some people who want to build their own images ab initio — most often, companies which have products built on "customized" versions of FreeBSD — and while I have helped a few people do this, it's better if my help is not needed. To this end, earlier today I published my code for building FreeBSD AMIs. At its core, this process has two steps: First, building a disk image; and second, turning it into an AMI.The process of building a disk image starts out the same as for any other platform: Slice and partition a disk, create a filesystem, then extract the FreeBSD bits from a release ISO. Prior to FreeBSD 10.0 this was a more complex task, since a custom kernel needed to be built; but FreeBSD now has Xen HVM support in the GENERIC kernel, which ships on release ISOs.
Having installed a "clean" FreeBSD system into a disk image, I then add extra code to support using the image in Amazon EC2. This consists of four ports in the FreeBSD ports tree:
- sysutils/ec2-scripts: This is the only truly essential port, as it contains code for configuring the system to allow logins using the SSH public key provided via EC2; logging the SSH host keys to the EC2 console; and handling configinit data to allow an EC2 instance to be provided with configuration parameters when it is launched.
- sysutils/firstboot-freebsd-update: This port downloads and installs updates to the FreeBSD base system when the EC2 instance first boots. This is important since there may be many important security and errata updates between when the AMI is created and when an instance is launched months or years later.
- sysutils/firstboot-pkgs: This port downloads and installs FreeBSD packages (code compiled from the ports tree) when the EC2 instance first boots. This makes it possible for an instance to be launched with newer code than was available when the AMI was created; and (together with the functionality provided by configinit) makes it possible to launch a new FreeBSD instance and have it available moments later with a set of software installed and ready to use.
- sysutils/panicmail: This port is not particularly specific to EC2, although I have had an earlier version of the code in my EC2 images for a long time. The panicmail system provides an automated mechanism for reporting of FreeBSD kernel panics to a centralized location, and we're hoping it will prove useful in identifying and fixing stability issues — both those related to the FreeBSD/EC2 platform, and those occuring in other FreeBSD environments.
Having created a disk image — a task which my FreeBSD/EC2 build code performs in EC2, writing directly to an Elastic Block Store volume, for convenience — the second step is to turn this into an EC2 AMI. This is made more complex by a bit of history: When Amazon EC2 was first created using Xen, there was only one virtualization method available, namely "paravirtualization" ("PV" for short); so that's what EC2 used. Later, as virtualization gained popularity, Intel and AMD added support into their CPUs for "hardware virtualization", and Xen's HVM mode was born; but EC2 stuck with what was working for them, namely PV mode.
Unfortunately, FreeBSD needs Xen HVM mode (or some later hybrid modes). In 2008, Amazon announced support for running Microsoft Windows inside EC2, which also needed Xen HVM mode but had higher prices due to the Windows licenses involved; but the operating system world — not just FreeBSD, but also Linux vendors — continued to move towards taking advantage of hardware virtualization, and finally in July 2010 Amazon announced their first "Cluster Compute" instances with HVM support; for the past three years, all of the new EC2 instance types have fully supported HVM. For this reason, from a single disk image I build two FreeBSD/EC2 AMIs: One which pretends to be Windows (and has the higher "Windows" hourly pricing) but can run on all instance types; and one which can only run on "modern" EC2 instances types but has the standard "UNIX" hourly pricing.
In an ideal world, creating a "Windows" AMI from a disk image would just require a single API call to add the "Windows" label; alas, EC2 is not ideal. Instead, I use a trick I call defenestration: I launch a Windows EC2 instance; "throw the windows out" by detaching and deleting its boot disk; attach the FreeBSD disk image as the root disk; and then tell EC2 to create an AMI from the "Windows" instance. Since EC2 knows that the instance was launched from a Windows AMI, it affixes the "Windows" label automatically (if it didn't, rebundling Windows instances wouldn't work), providing the HVM environment FreeBSD needs — as well as the "Windows tax" we don't.
Creating the "UNIX HVM" image, on the other hand, is just a single API call — but an API call which until a few months ago most people couldn't make. For some reason (I'm sure there was one, but I can't imagine what it was) when Amazon first made HVM available on Cluster Compute instances, they decided to keep the process for creating HVM images secret. Fortunately, the right people within Amazon knew about my work on FreeBSD/EC2, and — under a Non-Disclosure Agreement — provided me with special EC2 "API tools" code and set an "allowed to create HVM images" flag on my AWS account. (This is far from the only thing Amazon has quietly done to help me, but it's something they've confirmed I can now talk about publicly.) Fortunately, Amazon changed their stance on HVM image creation a few months ago, and the VirtualizationType parameter to the RegisterImage API call is now publicly available and documented — which is why I'm able to share my EC2 image building code publicly rather than limiting it to people who have signed Amazon NDAs.
While I'm currently building both "Windows" and "UNIX" images, I imagine that at some point I will stop building the "Windows" images; as I mentioned above, all the new EC2 instance types since 2011 support running FreeBSD without the "Windows" label, and now that Amazon has released the m3.medium EC2 image type at a price of $0.11/hour there aren't many situations left where it would make sense to run one of the older instance types.
I think most FreeBSD users will still find it easiest to continue using the FreeBSD/EC2 images which I am publishing, but for those people who want to build their own images: the code is out there — have fun!
Email delivery headaches
Email delivery used to be easy. Server A connects to Server B over SMTP, states that it has a message for Bob from Alice, then sends the message text ("We meet at midnight"). This worked fine until spam came along; to cope with a deluge of spam, an extra step was added, namely "Server B decides whether it trusts Server A to provide email from Alice for Bob". Even then, it wasn't too bad; sure, wide swaths of IPv4 address space were blacklisted, but if you had a server at a reputable ISP, you would probably not be on any of those lists. Then cloud computing happened.With the launch of Amazon EC2 in August 2006, life suddenly became much easier for spammers: If they found that their IP address was blacklisted, all they had to do was pay another $0.10 to spin up an EC2 instance with a different IP address to keep sending spam. While Amazon started out by disabling spammers' accounts, this quickly proved infeasible; and before long, the entire EC2 address space was on "dynamic IP space" blacklists. When I launched Tarsnap this wasn't a problem for me since I sent all my outgoing email through a server at RootBSD; but a few years ago I decided to move everything to Amazon Web Services, for ease of management, and so I had to stop sending email out directly.
My first choice was to use Amazon's Simple Email Service — after all, it was designed specifically for the use case of "company with servers in EC2 needs to send email". Alas, SES couldn't satisfy my needs: For spam and scam prevention reasons, SES applies strict filters to outgoing mail. Not only are SMTP envelope sender addresses required to be on a pre-verified list, but message From: addresses also need to be pre-verified. For emails from Tarsnap to Tarsnap users this wouldn't be a problem — but Tarsnap also has several public mailing lists, and when Tarsnap users post to the lists, their email goes out (via Tarsnap's mail server) with their original From: header.
Having decided that I could not use Amazon SES, I turned to another email-delivery-as-a-service company, SendGrid. For two years I was generally satisfied with them; I occasionally had Tarsnap users report that gmail has marked messages as spam, but I chalked that up to Google's spam filters being overly aggressive and decided there wasn't much I could do about it. A couple days ago, however, a Tarsnap user reported to me that the DKIM headers being added by SendGrid were invalid — not good! — and worse, a significant proportion of email he received via SendGrid had been failing DKIM checks for months. While SendGrid replied via twitter to say that they were aware of the issue and working on a fix, this was enough to make me reconsider my email-sending strategy.
My new strategy is to combine Amazon SES and SendGrid. When outgoing email needs to be sent by qmail, it no longer goes straight to the standard qmail-remote binary; instead, I replaced that with a shell script which examines the sending address. If that address is in the list of addresses I have white listed with Amazon SES, a qmail-remote-ses binary is invoked to pass the message to Amazon; I trust Amazon to do a good job of making sure email gets delivered (since, after all, they have a lot of experience with email from their retail business), and this will take care of all the most important Tarsnap email. Messages going out with other addresses — mainly email from the public mailing lists — are passed instead to a qmail-remote-smtp binary (which is in fact the old qmail-remote binary) and make their way out via SMTP to SendGrid.
Will this be good enough? I certainly hope so; considering how essential email has been to the development of the internet, and how much it is still relied upon by other services — an email address is a de facto internet-wide user name — it seems absurd how far one needs to go to get email successfully delivered. And if I find it this much of a hassle... well, I'm sure I can't be the only person suffering this headache.
Dear Google Recruiting...
We dated briefly in 2006. You flew me down to visit you in Mountain View, and I had a good time. A few weeks later you proposed to me, but I decided that you weren't really what I was looking for, and I rejected you. I know it's hard to accept, but I really think it's time you moved on.In October 2007, you wrote to me asking if I'd reconsider; I talked to one of my friends who works for you, and he talked to one of his friends in recruiting, who put "DO NOT CONTACT" on my file. I hoped that would be clear, but apparently not. In December 2009, you wrote to me again, and when I asked if you noticed the "DO NOT CONTACT", you replied that it was in the system, but since it had been two years, you decided to ignore it. In May 2010, you wrote to me again. I replied by asking you to change the "DO NOT CONTACT" to "DO NOT CONTACT ON PAIN OF DEATH".
That seemed to work for a while, but in February 2013 you wrote to me again and (when I asked) claimed that the "DO NOT CONTACT" was not in your system, but said that you would update your records. In September 2013, I met two of your employees at EuroBSDCon, and they pulled up my file and confirmed that I had a clear "DO NOT CONTACT" on my file (the "ON PAIN OF DEATH" seemed to have never gotten recorded), and told me that anyone ignoring this could be fired because the last thing you want to do is annoy good developers.
In November 2013, I was contacted via LinkedIn. When I ignored the message, I received another one the following month from the same recruiter, which I also ignored. And now a second recruiter has contacted me on LinkedIn.
You're a great company. I use a lot of your services. A lot of my friends work for you. But I don't want to. I found(ed) a very nice company which I work for, and nothing you say is going to make me leave it in order to be with you. So please, it's time to move on and find someone else. It's getting a bit creepy, and I don't want to have to get a restraining order.
Sincerely,
Colin Percival
Normally I don't publish anything publicly about private communications I have received, but this seems like a special case. The following recruiters contacted me via email or LinkedIn after the initial "DO NOT CONTACT" was placed on my file: Tara Nowroozi, Marcus Walton, Emily Mao, Chris Tessmann, Hannah Schuessler.
EDIT: A number of people are misinterpreting this post as "bragging" about being in "high demand". It isn't. If I thought this I was remotely unusual in having repeated contacts from Google, I wouldn't have posted this; to the contrary, saying the words "Google recruiters" at any of the conferences I attend inevitably provokes developers to rant about how they keep on getting emails — with the stories evenly split between "I keep on telling them I don't want to work for Google" and "I've interviewed several times, never gotten a job offer, and now it just feels like they're taunting me". This isn't about me; it's about Google recruiting being out of control in their "sourcing" efforts.
Thoughts on the Tarsnap logo contest
Coming as I do from the worlds of academia and open source software, it was only natural that when I decided that my online backup service needed a logo, I turned to the "crowd" with a $500 contest. While I considered using one of the many logo-design-competition sites, I decided to run the contest myself for a simple reason: Tarsnap isn't exactly like most commercial products, but it has an enthusiastic user community who understand the mindset behind it. Tarsnap is not just "online backups for the truly paranoid"; it's also very much unix software, in the sense of "do one thing well", "keep it simple (stupid)", and "tools, not policy". Running the contest myself and announcing it via the tarsnap-users mailing list might have decreased the number of graphic designers participating, but I'm sure it increased the number of people who knew something about Tarsnap. (I did, however, keep one such site in mind as a backup plan in case I didn't like any of the submitted logos.)I announced the contest on September 28th, and set a deadline of October 31st for submissions. During this time, 83 people submitted over 100 designs — with over half of them arriving in the first three days. Based on the numbers alone, it sounds as though a shorter contest period would have worked well; but there was a general tendency that the logos submitted later looked like they had received more thought, so I'm glad I resisted the temptation to set an earlier deadline.
Getting people to submit logos was easy; selecting one was much harder. I started by performing a rough pass, throwing out the logos which seemed (to my eye) to be ugly, too generic, or simply not conveying the right message (that Tarsnap is secure, serious, and reliable, as opposed to colourful and fun); this left me with 16 logos. I took these to an informal "focus group" consisting of friends, family, long-time Tarsnap users, and other startup founders (there was considerable overlap between those categories), and came away with two logos which were clearly superior to the others — but for a variety of reasons didn't quite work as they were.
The following few weeks convinced me that Rorschach should have skipped the ink blots and used corporate logos instead. One of the two remaining designs — a reel of tape with a keyhole — was immediately identified by younger members of my focus group as being a Poké ball; upon revisions, it morphed in the eyes of other people into a radiation warning symbol, a peace sign, and a car steering wheel. The other leading logo, after being changed from a stack of documents with a keyhole to a stack of disks (the original was too similar to Buffer's logo) was immediately identified instead as a cylinder lock, a stack of pancakes, and a stack of cookies. After many round trips with the designers of the two logos, I eventually decided that anything I picked was going to end up giving someone the wrong idea — but if people were going to look at the logo the wrong way, a stack of pancakes was at least better than a radiation warning symbol. The winner, as promised, received $500; I decided to create a second prize of $100 for the designer of the other logo, in part as an apology for the time I took up asking him for revisions.
Once I was satisfied with the image — and had redrawn it in MetaPost, for my ease of manipulation — I created three variants for different sizes of usage: The image next to the name "Tarsnap", underlined by the tagline "Online backups for the truly paranoid" (for use at the top of the Tarsnap website); the image above the name "Tarsnap", creating a square logo (for use on the FreeBSD Foundation's sponsors list); and the image alone (for use as a website favicon and on the @tarsnap twitter account).
Overall, I'm quite happy with how this went, and I'm very happy with the final logo. I did however learn some lessons, which I'll list here for the benefit of anyone else considering such a process:
- Everything takes longer than you expect.
- It's essential to have a way to make people aware of the contest — enthusiastic users, twitter followers, name recognition within an online community, or ideally all three.
- Find a group of people to help you narrow down your list, and when they (mostly) agree, listen to them. The submission the final logo was derived from was 8th on my personal preference list.
- Don't expect to find a logo which everybody likes — it won't happen. But if someone has a seriously bad reaction to a logo (e.g., "it looks like a radiation warning sign") rather than merely a lack of enthusiasm, move on to another design.
- If you're going to have a combination of image and text, you can decide how the two should fit together long before knowing what the image will be.
- If you show someone a logo and ask "what does this look like?", be prepared to be astonished.
- Everything takes longer than you expect.
Should all companies get their logos this way? Absolutely not: A large company will probably be better off hiring artists, while at the opposite end of the spectrum, a company without any public presence would need to use an existing design-contest site in order for anyone to even notice that the contest existed. But was this the right way to get a logo for Tarsnap? Yes, I think it was.
Introducing configinit
I have been working on bringing FreeBSD to the Amazon EC2 platform since 2006, and for the past three years I've been blogging about my progress: First FreeBSD on t1.micro instances, then cluster compute instances, then "m1" and "m2" family large and xlarge instances, and finally in early 2012, FreeBSD could finally run on all EC2 instance types. Once I had a hacked-up version of FreeBSD which ran smoothly, I turned my attention towards polishing it: First moving my EC2 scripts into the ports tree, then using binaries from the release ISOs for the FreeBSD world, and finally in early October (with FreeBSD 10.0-ALPHA4) all the necessary bits had been merged to make it possible for me to build EC2 images completely (including the kernel) with "straight off the ISO" binaries. Next on my agenda was taking my images from "pure FreeBSD" to "FreeBSD set up to be used in the cloud", and for that I'm happy to now announce that starting from 10.0-RC1, my FreeBSD AMIs have a new feature: configinit.Anyone who has been around the world of cloud computing for long is likely to have heard of CloudInit. It is a system originally written for Ubuntu which performs configuration of a system at boot-time based on "user-data" provided via EC2 or from a similar environment (e.g., OpenStack). CloudInit works well for its original purpose, but is less than ideal for FreeBSD systems, for two reasons: First, it relies on Python, which is not part of the FreeBSD base system; and second, it is designed around a concept of configuring a system by running commands rather than editing configuration files.
Now, there are merits to both approaches — most notably, configuring a system by running commands is easier to script, while configuring a system by editing text files has the advantage that given a working configuration there's no doubt about how to reproduce it — but the fact that BSD systems are far more edit-configuration-files oriented (to the point that /etc/rc.conf might be the only configuration file which needs to be edited on some systems), and thus CloudInit is less than optimal for configuring FreeBSD systems.
Enter configinit. Rather than providing instructions such as "tell apt to use this mirror" or "run this python code", configinit handles four types of input:
- If the configuration data starts ">/path/to/file" then the data, minus the first line, will be written to the specified location.
- If the configuration data starts ">>/path/to/file". then the data, minutes the first line, will be appended to the specified location.
- If the configuration data starts "#!", it will be executed (in most cases this would be a shell script).
- For any other inputs, configinit attempts to extract the file as an archive, and (if extraction was successful) runs on each part in turn. The extraction is performed using bsdtar, so archives in tar, pax, cpio, zip, jar, ar, xar, rpm and ISO 9660 formats, optionally compressed using gzip, bzip2, xz, or compress can all be used.
This is much simpler than CloudInit, but — in combination with other tools which are already available on FreeBSD, such as my firstboot-pkgs port, it provides very powerful yet easy-to-use functionality. For example, launching a FreeBSD 10.0-RC1 EC2 instance with the following user-data:
will provide a system with Apache 2.2 installed and running (in my test, within 150 seconds of when I clicked "launch" in the EC2 Management Console) — in addition to performing the other default system initialization behaviours of my EC2 images: checking for updates to the FreeBSD base system, downloading an SSH public key to allow SSH logins as ec2-user, logging SSH host keys to the EC2 console, and autoconfiguring swap space using EC2 ephemeral disks.>>/etc/rc.conf firstboot_pkgs_list="apache22" apache22_enable="YES"
I know for my purposes this will be very useful — for example, while I have the process of configuring a new Portsnap mirror mostly scripted now, using configinit I could have it entirely scripted and avoid the need to ever SSH into the mirrors — and from what other FreeBSD users have told me, I don't think I will be alone. Is there anything else I could do to make FreeBSD even more usable in EC2? Quite likely — but I don't know what. If I'm missing something important, please let me know!