How to build FreeBSD/EC2 images
I have been building FreeBSD/EC2 images for the past three years, and based on the email I have been receiving, most people have been either using these images directly or modifying them to create images which suit their needs. However, there are some people who want to build their own images ab initio — most often, companies which have products built on "customized" versions of FreeBSD — and while I have helped a few people do this, it's better if my help is not needed. To this end, earlier today I published my code for building FreeBSD AMIs. At its core, this process has two steps: First, building a disk image; and second, turning it into an AMI.The process of building a disk image starts out the same as for any other platform: Slice and partition a disk, create a filesystem, then extract the FreeBSD bits from a release ISO. Prior to FreeBSD 10.0 this was a more complex task, since a custom kernel needed to be built; but FreeBSD now has Xen HVM support in the GENERIC kernel, which ships on release ISOs.
Having installed a "clean" FreeBSD system into a disk image, I then add extra code to support using the image in Amazon EC2. This consists of four ports in the FreeBSD ports tree:
- sysutils/ec2-scripts: This is the only truly essential port, as it contains code for configuring the system to allow logins using the SSH public key provided via EC2; logging the SSH host keys to the EC2 console; and handling configinit data to allow an EC2 instance to be provided with configuration parameters when it is launched.
- sysutils/firstboot-freebsd-update: This port downloads and installs updates to the FreeBSD base system when the EC2 instance first boots. This is important since there may be many important security and errata updates between when the AMI is created and when an instance is launched months or years later.
- sysutils/firstboot-pkgs: This port downloads and installs FreeBSD packages (code compiled from the ports tree) when the EC2 instance first boots. This makes it possible for an instance to be launched with newer code than was available when the AMI was created; and (together with the functionality provided by configinit) makes it possible to launch a new FreeBSD instance and have it available moments later with a set of software installed and ready to use.
- sysutils/panicmail: This port is not particularly specific to EC2, although I have had an earlier version of the code in my EC2 images for a long time. The panicmail system provides an automated mechanism for reporting of FreeBSD kernel panics to a centralized location, and we're hoping it will prove useful in identifying and fixing stability issues — both those related to the FreeBSD/EC2 platform, and those occuring in other FreeBSD environments.
Having created a disk image — a task which my FreeBSD/EC2 build code performs in EC2, writing directly to an Elastic Block Store volume, for convenience — the second step is to turn this into an EC2 AMI. This is made more complex by a bit of history: When Amazon EC2 was first created using Xen, there was only one virtualization method available, namely "paravirtualization" ("PV" for short); so that's what EC2 used. Later, as virtualization gained popularity, Intel and AMD added support into their CPUs for "hardware virtualization", and Xen's HVM mode was born; but EC2 stuck with what was working for them, namely PV mode.
Unfortunately, FreeBSD needs Xen HVM mode (or some later hybrid modes). In 2008, Amazon announced support for running Microsoft Windows inside EC2, which also needed Xen HVM mode but had higher prices due to the Windows licenses involved; but the operating system world — not just FreeBSD, but also Linux vendors — continued to move towards taking advantage of hardware virtualization, and finally in July 2010 Amazon announced their first "Cluster Compute" instances with HVM support; for the past three years, all of the new EC2 instance types have fully supported HVM. For this reason, from a single disk image I build two FreeBSD/EC2 AMIs: One which pretends to be Windows (and has the higher "Windows" hourly pricing) but can run on all instance types; and one which can only run on "modern" EC2 instances types but has the standard "UNIX" hourly pricing.
In an ideal world, creating a "Windows" AMI from a disk image would just require a single API call to add the "Windows" label; alas, EC2 is not ideal. Instead, I use a trick I call defenestration: I launch a Windows EC2 instance; "throw the windows out" by detaching and deleting its boot disk; attach the FreeBSD disk image as the root disk; and then tell EC2 to create an AMI from the "Windows" instance. Since EC2 knows that the instance was launched from a Windows AMI, it affixes the "Windows" label automatically (if it didn't, rebundling Windows instances wouldn't work), providing the HVM environment FreeBSD needs — as well as the "Windows tax" we don't.
Creating the "UNIX HVM" image, on the other hand, is just a single API call — but an API call which until a few months ago most people couldn't make. For some reason (I'm sure there was one, but I can't imagine what it was) when Amazon first made HVM available on Cluster Compute instances, they decided to keep the process for creating HVM images secret. Fortunately, the right people within Amazon knew about my work on FreeBSD/EC2, and — under a Non-Disclosure Agreement — provided me with special EC2 "API tools" code and set an "allowed to create HVM images" flag on my AWS account. (This is far from the only thing Amazon has quietly done to help me, but it's something they've confirmed I can now talk about publicly.) Fortunately, Amazon changed their stance on HVM image creation a few months ago, and the VirtualizationType parameter to the RegisterImage API call is now publicly available and documented — which is why I'm able to share my EC2 image building code publicly rather than limiting it to people who have signed Amazon NDAs.
While I'm currently building both "Windows" and "UNIX" images, I imagine that at some point I will stop building the "Windows" images; as I mentioned above, all the new EC2 instance types since 2011 support running FreeBSD without the "Windows" label, and now that Amazon has released the m3.medium EC2 image type at a price of $0.11/hour there aren't many situations left where it would make sense to run one of the older instance types.
I think most FreeBSD users will still find it easiest to continue using the FreeBSD/EC2 images which I am publishing, but for those people who want to build their own images: the code is out there — have fun!
Email delivery headaches
Email delivery used to be easy. Server A connects to Server B over SMTP, states that it has a message for Bob from Alice, then sends the message text ("We meet at midnight"). This worked fine until spam came along; to cope with a deluge of spam, an extra step was added, namely "Server B decides whether it trusts Server A to provide email from Alice for Bob". Even then, it wasn't too bad; sure, wide swaths of IPv4 address space were blacklisted, but if you had a server at a reputable ISP, you would probably not be on any of those lists. Then cloud computing happened.With the launch of Amazon EC2 in August 2006, life suddenly became much easier for spammers: If they found that their IP address was blacklisted, all they had to do was pay another $0.10 to spin up an EC2 instance with a different IP address to keep sending spam. While Amazon started out by disabling spammers' accounts, this quickly proved infeasible; and before long, the entire EC2 address space was on "dynamic IP space" blacklists. When I launched Tarsnap this wasn't a problem for me since I sent all my outgoing email through a server at RootBSD; but a few years ago I decided to move everything to Amazon Web Services, for ease of management, and so I had to stop sending email out directly.
My first choice was to use Amazon's Simple Email Service — after all, it was designed specifically for the use case of "company with servers in EC2 needs to send email". Alas, SES couldn't satisfy my needs: For spam and scam prevention reasons, SES applies strict filters to outgoing mail. Not only are SMTP envelope sender addresses required to be on a pre-verified list, but message From: addresses also need to be pre-verified. For emails from Tarsnap to Tarsnap users this wouldn't be a problem — but Tarsnap also has several public mailing lists, and when Tarsnap users post to the lists, their email goes out (via Tarsnap's mail server) with their original From: header.
Having decided that I could not use Amazon SES, I turned to another email-delivery-as-a-service company, SendGrid. For two years I was generally satisfied with them; I occasionally had Tarsnap users report that gmail has marked messages as spam, but I chalked that up to Google's spam filters being overly aggressive and decided there wasn't much I could do about it. A couple days ago, however, a Tarsnap user reported to me that the DKIM headers being added by SendGrid were invalid — not good! — and worse, a significant proportion of email he received via SendGrid had been failing DKIM checks for months. While SendGrid replied via twitter to say that they were aware of the issue and working on a fix, this was enough to make me reconsider my email-sending strategy.
My new strategy is to combine Amazon SES and SendGrid. When outgoing email needs to be sent by qmail, it no longer goes straight to the standard qmail-remote binary; instead, I replaced that with a shell script which examines the sending address. If that address is in the list of addresses I have white listed with Amazon SES, a qmail-remote-ses binary is invoked to pass the message to Amazon; I trust Amazon to do a good job of making sure email gets delivered (since, after all, they have a lot of experience with email from their retail business), and this will take care of all the most important Tarsnap email. Messages going out with other addresses — mainly email from the public mailing lists — are passed instead to a qmail-remote-smtp binary (which is in fact the old qmail-remote binary) and make their way out via SMTP to SendGrid.
Will this be good enough? I certainly hope so; considering how essential email has been to the development of the internet, and how much it is still relied upon by other services — an email address is a de facto internet-wide user name — it seems absurd how far one needs to go to get email successfully delivered. And if I find it this much of a hassle... well, I'm sure I can't be the only person suffering this headache.