FreeBSD ZFS AMIs Now Available

Earlier today I sent an email to the freebsd-cloud mailing list:
Hi EC2 users,

FreeBSD 12.0-RELEASE is now available as AMIs with ZFS root disks in all 16
publicly available EC2 regions:

[List elided; see the GPG-signed email for AMI IDs]

The ZFS configuration (zpool named "zroot", mount point on /, /tmp,
/usr/{home,ports,src}, /var/{audit,crash,log,mail,tmp}) should match what
you get by installing FreeBSD 12.0-RELEASE from the published install media
and selecting the defaults for a ZFS install.  Other system configuration
matches the FreeBSD 12.0-RELEASE AMIs published by the release engineering

I had to make one substantive change to 12.0-RELEASE, namely merging r343918
(which teaches /etc/rc.d/growfs how to grow ZFS disks, matching the behaviour
of the UFS AMIs in supporting larger-than-default root disks); I've MFCed
this to stable/12 so it will be present in 12.1 and later releases.

If you find these AMIs useful, please let me know, and consider donating to
support my work on FreeBSD/EC2 (  If
there's enough interest I'll work with the release engineering team to add
ZFS AMIs to what they publish.

In the interests of both transparency and following my own advice of "we need to write more how-to-do-foo walkthroughs", here's the process I used to create those images.

  1. Launch a FreeBSD 12.0-RELEASE AMI Builder instance; in the us-east-1 region, this is ami-09baac3ede1d33201. Due to issues related to disk device names, the AMI Builder doesn't work on the latest "nitro" instances; and because of what we're going to be doing, we need an instance with lots of RAM — so I used a t2.2xlarge instance for this.
  2. Wait about 10 minutes (the AMI Builder boots into RAM but then needs to spend some time extracting FreeBSD before you can connect); then SSH in as ec2-user and su to root (no password needed).
  3. The AMI Builder starts us off with FreeBSD installed onto a UFS filesystem, so we need to move those bits safely out of the way:
    mdconfig -a -t swap -s 3G -u 2
    newfs /dev/md2
    mkdir /mdisk
    mount /dev/md2 /mdisk
    tar -czf /mdisk/base.tgz --exclude .snap -C /mnt .
    umount /mnt
  4. Next, wipe the old UFS bits out of the way and repartition the disk:
    gpart destroy -F ada0
    dd if=/dev/zero bs=128k of=/dev/ada0
    gpart create -s gpt ada0
    gpart add -a 4k -s 512K -t freebsd-boot ada0
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
    gpart add -a 1m -t freebsd-zfs -l disk0 ada0
  5. Create all of the standard FreeBSD/ZFS datasets:
    zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot ada0p2
    zfs create -o mountpoint=none zroot/ROOT
    zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/default
    mount -t zfs zroot/ROOT/default /mnt
    zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
    zfs create -o mountpoint=/usr -o canmount=off zroot/usr
    zfs create zroot/usr/home
    zfs create -o setuid=off zroot/usr/ports
    zfs create zroot/usr/src
    zfs create -o mountpoint=/var -o canmount=off zroot/var
    zfs create -o exec=off -o setuid=off zroot/var/audit
    zfs create -o exec=off -o setuid=off zroot/var/crash
    zfs create -o exec=off -o setuid=off zroot/var/log
    zfs create -o atime=on zroot/var/mail
    zfs create -o setuid=off zroot/var/tmp
    zpool set bootfs=zroot/ROOT/default zroot
  6. And now we can extract FreeBSD back onto the newly-ZFS disk:
    tar -xf /mdisk/base.tgz -C /mnt
  7. We no longer have a UFS filesystem, so we can get rid of the current contents of fstab(5); but we do need some configuration settings to support ZFS instead:
    : > /mnt/etc/fstab
    echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
    echo 'kern.geom.label.disk_ident.enable="0"' >> /mnt/boot/loader.conf
    echo 'kern.geom.label.gptid.enable="0"' >> /mnt/boot/loader.conf
    echo 'vfs.zfs.min_auto_ashift=12' >> /mnt/etc/sysctl.conf
    echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
  8. And in order to make ZFS expand when we first boot, we need an updated script for /etc/rc.d/growfs:
    svnlite export
    cp growfs /mnt/etc/rc.d/growfs
  9. Finally we can "poweroff" the instance
    shutdown -p now
    and then use the EC2 Management Console to create an image from the instance (or run aws ec2 create-image).

Thanks to the freebsd-cloud and freebsd-fs mailing lists, and Ben Woods in particular, for testing and feedback on my earlier attempts.

Posted at 2019-02-16 21:50 | Permanent link | Comments

The many ways to launch FreeBSD in EC2

Talking to FreeBSD users recently, I became aware that while I've created a lot of tools, I haven't done a very good job of explaining how, and more importantly when to use them. So for all of the EC2-curious FreeBSD users out there: Here are the many ways to launch and configure FreeBSD in EC2 — ranging from the simplest to the most complicated (but most powerful):

Launch FreeBSD and SSH in

This is the most straightforward way to get started with FreeBSD: Using either the EC2 API (most easily accessed via the awscli package) or the AWS Console, spin up a "stock FreeBSD" AMI. You have a few choices to make here:
  1. Whether to use the Community or Marketplace AMIs.
  2. Which EC2 Region you want to launch your instance into.
  3. Which of many EC2 instance types you want to use.
  4. How large you want the root disk to be (the root filesystem will be resized automatically if you select a size larger than the default 10 GB) and whether to attach additional disks.
For a "hobbyist" the easiest answers will probably be to use the Marketplace AMIs, in the EC2 region which is geographically closest to you, with a t2.micro and a single 10 GB disk (the default).

After you launch the instance, you can ssh in as the ec2-user user and then su to root (no password required). From this point on, setting up your EC2 instance is just like setting up a physical server.

Launch FreeBSD and provide user-data

But what if you want a more automated process — something which will yield a consistent configuration, without the time-consuming process of SSHing in to set things up? Enter configinit.

When you launch an EC2 instance, you have the option of providing it with some "user-data". On FreeBSD, this is processed by configinit, which is a tool I wrote for configuring FreeBSD systems. For example, you can pass user-data of

to have FreeBSD install the apache24 package when it first boots and enable the Apache daemon. For more complex configurations you can provide a tarball containing files with directives to append to or replace multiple configuration files; to launch Portsnap mirrors, I have a configinit file which adds lines to rc.conf and newsyslog.conf, creates lighttpd.conf, runs a script to start synchronizing with the portsnap master server, etc.

Use the AMI Builder to create a customized FreeBSD AMI

Maybe rather than having commands run when FreeBSD first boots, you'd prefer to set things up ahead of time. Enter the FreeBSD AMI Builder.

This is an AMI which boots into a memory disk and installs FreeBSD onto the root disk — and then lets you make further changes before you create a new AMI from it. Long time EC2 users may recall that in the early days of EC2 — when everything was Ubuntu — the normal way to create a new AMI was to boot up an instance, make changes, and then package it up; this follows the same idea, but because the AMI Builder boots into a memory disk it avoids "contaminating" the image you're creating with keys and log files.

To make the process of building new AMIs even easier, the AMI Builder can make use of IAM Roles and runs configinit; so you can repeatably create customized FreeBSD AMIs in a matter of minutes with a single command.

I have built FreeBSD AMI Builder images in the us-east-1 region; the currently supported releases are FreeBSD 11.2 (ami-000414bbf0ee227c5) and FreeBSD 12.0 (ami-09baac3ede1d33201).

Build a FreeBSD AMI from a modified FreeBSD source tree

The options above all give you a FreeBSD RELEASE — possibly customized by installing extra packages and editing configuration files, but (unless you launch the AMI Builder and then buildworld/buildkernel inside there) you'll be running the binaries shipped by the FreeBSD Project. What if you have changes to FreeBSD itself — say, if there's a bug which was fixed after the release which you need to backport the fix for, or you need a custom kernel configuration? Enter make ec2ami.

This is in fact the exact mechanism used by the FreeBSD release engineering team to produce official FreeBSD AMIs — of course, they have the AWS keys needed to publish images into the official FreeBSD Release Engineering AWS account, but anyone with an AWS account can follow the same process using their own keys. Just run a buildworld/buildkernel of your modified FreeBSD, set up a few parameters (AWS keys, AWS region, and an S3 bucket to be used for staging the upload), and cd /usr/src/release && make ec2ami.

Build your own disk image

Finally, some people will want to build AMIs which are even more customized than you can easily do via the FreeBSD release code. Maybe you want to have a system which boots into a memory disk, so that it will keep working even if its EBS volume dies. Maybe you want to boot from NFS, making use of Amazon's Elastic File System (aka. NFS-as-a-service). Maybe you want to build an AMI for a completely different OS. For this, there's bsdec2-image-upload.

As the name suggests, that tool simply uploads a disk image and bakes it into an EC2 AMI. If you're going this route, everything else is up to you... but if you need to go this route, you're doing something sufficiently obscure that you almost certainly need to do everything yourself anyway.

I hope I've provided tools which help you to run FreeBSD in EC2, no matter how common or unusual your needs are. If you find my work useful, please consider supporting my work in this area; while this is both something I enjoy working on and something which is useful for my day job (Tarsnap, my online backup service), having support would make it easier for me to prioritize FreeBSD/EC2 issues over other projects.

Posted at 2018-12-26 07:05 | Permanent link | Comments

How to port your OS to EC2

I've been the maintainer of the FreeBSD/EC2 platform for about 7.5 years now, and as far as "running things in virtual machines" goes, that remains the only operating system and the only cloud which I work on. That said, from time to time I get questions from people who want to port other operating systems into EC2, and being a member of the open source community, I do my best to help them. I realized a few days ago that rather than replying to emails one by one it would be more efficient to post something publicly; so — for the benefit of the dozen or so people who want to port operating systems to run in EC2, and the curiosity of maybe a thousand more people who use EC2 but will never build AMIs themselves — here's a rough guide to building EC2 images.


Before we can talk about building images, there are some things you need:

Building a disk image

The first step to building an EC2 AMI is to build a disk image. This needs to be a "live" disk image, not an installer image; but if you have a "live USB disk" image, that's almost certainly going to be the place to start. EC2 instances boot with a virtual BIOS so a disk image which can boot from a USB stick is almost certainly going to boot — at least as far as the boot loader — in EC2.

You're going to want to make some changes to what goes into that disk image later, but for now just build a disk image.

Building an AMI

I wrote a simple tool for converting disk images into EC2 instances: bsdec2-image-upload. It uploads a disk image to Amazon S3; makes an API call to import that disk image into an EBS volume; creates a snapshot of that volume; then registers an EC2 AMI using that snapshot.

To use bsdec2-image-upload, you'll first need to create an S3 bucket for it to use as a staging area. You can call it anything you like, but I recommend that you

You'll also need to create an AWS key file in the format which bsdec2-image-upload expects:


Having done that, you can invoke bsdec2-image-upload:

# bsdec2-image-upload disk.img "AMI Name" "AMI Description" aws-region S3bucket awskeys

There are three additional options you can specify:

After it uploads the image and registers the AMI, bsdec2-image-upload will print the AMI IDs for the relevant region(s). (Either for every region, or just for the single region where you uploaded it.)

Go ahead and create an AMI now, and try launching it.

Boot configuration

Odds are that your instance started booting and got as far as the boot loader launching the kernel, but at some point after that things went sideways. Now we start the iterative process of building disk images, turning them into AMIs, launching said AMIs, and seeing where they break. Some things you'll probably run into here:

At this point, you should be able to launch an EC2 instance, get console output showing that it booted, and connect to the SSH daemon. (Remember to allow incoming connections on port 22 when you launch the EC2 instance!)

EC2 configuration

Now it's time to make the AMI behave like an EC2 instance. To this end, I prepared a set of rc.d scripts for FreeBSD. Most importantly, they If your OS has an rc system derived from NetBSD's rc.d, you may be able to use these scripts without any changes by simply installing them and enabling them in /etc/rc.conf; otherwise you may need to write your own scripts using mine as a model.

Firstboot scripts

A feature I added to FreeBSD a few years ago is the concept of "firstboot" scripts: These startup scripts are only run the first time a system boots. The aforementioned configinit and SSH key fetching scripts are flagged this way — so if your OS doesn't support the "firstboot" keyword on rc.d scripts you'll need to hack around that — but EC2 instances also ship with other scripts set to run on the first boot:

While none of these are strictly necessary, I find them to be extremely useful and highly recommend implementing similar functionality in your systems.

Support my work!

I hope you find this useful, or at very least interesting. Please consider supporting my work in this area; while I'm happy to contribute my time to supporting open source software, it would be nice if I had money coming in which I could use to cover incidental expenses (e.g., conference travel) so that I didn't end up paying to contribute to FreeBSD.

Posted at 2018-07-14 06:30 | Permanent link | Comments

Tarsnap pricing change

I launched the current Tarsnap website in 2009, and while we've made some minor adjustments to it over the years — e.g., adding a page of testimonials, adding much more documentation, and adding a page with .deb binary packages — the changes have overall been relatively modest. One thing people criticized the design for in 2009 was the fact that prices were quoted in picodollars; this is something I have insisted on retaining for the past eight years.

One of the harshest critics of Tarsnap's flat rate picodollars-per-byte pricing model is Patrick McKenzie — known to much of the Internet as "patio11" — who despite our frequent debates can take credit for ten times more new Tarsnap customers than anyone else, thanks to a single ten thousand word blog post about Tarsnap. The topic of picodollars has become something of an ongoing debate between us, with Patrick insisting that they communicate a fundamental lack of seriousness and sabotage Tarsnap's success as a business, and me insisting that using they communicate exactly what I want to communicate, and attract precisely the customer base I want to have. In spite of our disagreements, however, I really do value Patrick's input; indeed, the changes I mentioned above came about in large part due to the advice I received from him, and for a long time I've been considering following more of Patrick's advice.

A few weeks ago, I gave a talk at the AsiaBSDCon conference about profiling the FreeBSD kernel boot. (I'll be repeating the talk at BSDCan if anyone is interested in seeing it in person.) Since this was my first time in Tokyo (indeed, my first time anywhere in Asia) and despite communicating with him frequently I had never met Patrick in person, I thought it was only appropriate to meet him for dinner; fortunately the scheduling worked out and there was an evening when he was free and I wasn't suffering too much from jetlag. After dinner, Patrick told me about a cron job he runs:

I knew then that the time was coming to make a change Patrick has long awaited: Getting rid of picodollars. It took a few weeks before the right moment arrived, but I'm proud to announce that as of today, April 1st 2018, Tarsnap's storage pricing is 8333333 attodollars per byte-day.

This addresses a long-standing concern I've had about Tarsnap's pricing: Tarsnap bills customers for usage on a daily basis, but since 250 picodollars is not a multiple of 30, usage bills have been rounded. Tarsnap's accounting code works with attodollars internally (Why attodollars? Because it's easy to have 18 decimal places of precision using fixed-point arithmetic with a 64-bit "attodollars" part.) and so during 30-day months I have in fact been rounding down and billing customers at a rate of 8333333 attodollars per byte-day for years — so making this change on the Tarsnap website brings it in line with the reality of the billing system.

Of course, there are other advantages to advertising Tarsnap's pricing in attodollars. Everything which was communicated by pricing storage in picodollars per byte-month is communicated even more effectively by advertising prices in attodollars per byte-day, and I have no doubt that Tarsnap users will appreciate the increased precision.

Posted at 2018-04-01 00:00 | Permanent link | Comments

FreeBSD/EC2 history

A couple years ago Jeff Barr published a blog post with a timeline of EC2 instances. I thought at the time that I should write up a timeline of the FreeBSD/EC2 platform, but I didn't get around to it; but last week, as I prepared to ask for sponsorship for my work I decided that it was time to sit down and collect together the long history of how the platform has evolved and improved over the years.

Normally I don't edit blog posts after publishing them (with the exception of occasional typographical corrections), but I do plan on keeping this post up to date with future developments.

The current status

The upcoming FreeBSD release (11.2) supports: IPv6, Enhanced Networking (both generations), Amazon Elastic File System, Amazon Time Sync Service, both consoles (Serial + VGA), and every EC2 instance type (although I'm not sure if FreeBSD has drivers to make use of the FPGA or GPU hardware on those instances).

When a FreeBSD/EC2 instance first launches, it uses configinit to perform any desired configuration based on user-data scripts, and then (unless configinit is used to change this) resizes its root filesystem to fit the provided root disk, downloads and installs critical updates, sets up the ec2-user user for SSH access, and prints SSH host keys to the consoles.

If there's something else you think FreeBSD should support or a change you'd like to see to the default configuration, please let me know.

Posted at 2018-02-12 19:50 | Permanent link | Comments

Recent posts

Monthly Archives

Yearly Archives