FreeBSD ZFS AMIs Now Available

Earlier today I sent an email to the freebsd-cloud mailing list:
Hi EC2 users,

FreeBSD 12.0-RELEASE is now available as AMIs with ZFS root disks in all 16
publicly available EC2 regions:

[List elided; see the GPG-signed email for AMI IDs]

The ZFS configuration (zpool named "zroot", mount point on /, /tmp,
/usr/{home,ports,src}, /var/{audit,crash,log,mail,tmp}) should match what
you get by installing FreeBSD 12.0-RELEASE from the published install media
and selecting the defaults for a ZFS install.  Other system configuration
matches the FreeBSD 12.0-RELEASE AMIs published by the release engineering

I had to make one substantive change to 12.0-RELEASE, namely merging r343918
(which teaches /etc/rc.d/growfs how to grow ZFS disks, matching the behaviour
of the UFS AMIs in supporting larger-than-default root disks); I've MFCed
this to stable/12 so it will be present in 12.1 and later releases.

If you find these AMIs useful, please let me know, and consider donating to
support my work on FreeBSD/EC2 (  If
there's enough interest I'll work with the release engineering team to add
ZFS AMIs to what they publish.

In the interests of both transparency and following my own advice of "we need to write more how-to-do-foo walkthroughs", here's the process I used to create those images.

  1. Launch a FreeBSD 12.0-RELEASE AMI Builder instance; in the us-east-1 region, this is ami-09baac3ede1d33201. Due to issues related to disk device names, the AMI Builder doesn't work on the latest "nitro" instances; and because of what we're going to be doing, we need an instance with lots of RAM — so I used a t2.2xlarge instance for this.
  2. Wait about 10 minutes (the AMI Builder boots into RAM but then needs to spend some time extracting FreeBSD before you can connect); then SSH in as ec2-user and su to root (no password needed).
  3. The AMI Builder starts us off with FreeBSD installed onto a UFS filesystem, so we need to move those bits safely out of the way:
    mdconfig -a -t swap -s 3G -u 2
    newfs /dev/md2
    mkdir /mdisk
    mount /dev/md2 /mdisk
    tar -czf /mdisk/base.tgz --exclude .snap -C /mnt .
    umount /mnt
  4. Next, wipe the old UFS bits out of the way and repartition the disk:
    gpart destroy -F ada0
    dd if=/dev/zero bs=128k of=/dev/ada0
    gpart create -s gpt ada0
    gpart add -a 4k -s 512K -t freebsd-boot ada0
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
    gpart add -a 1m -t freebsd-zfs -l disk0 ada0
  5. Create all of the standard FreeBSD/ZFS datasets:
    zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot ada0p2
    zfs create -o mountpoint=none zroot/ROOT
    zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/default
    mount -t zfs zroot/ROOT/default /mnt
    zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
    zfs create -o mountpoint=/usr -o canmount=off zroot/usr
    zfs create zroot/usr/home
    zfs create -o setuid=off zroot/usr/ports
    zfs create zroot/usr/src
    zfs create -o mountpoint=/var -o canmount=off zroot/var
    zfs create -o exec=off -o setuid=off zroot/var/audit
    zfs create -o exec=off -o setuid=off zroot/var/crash
    zfs create -o exec=off -o setuid=off zroot/var/log
    zfs create -o atime=on zroot/var/mail
    zfs create -o setuid=off zroot/var/tmp
    zpool set bootfs=zroot/ROOT/default zroot
  6. And now we can extract FreeBSD back onto the newly-ZFS disk:
    tar -xf /mdisk/base.tgz -C /mnt
  7. We no longer have a UFS filesystem, so we can get rid of the current contents of fstab(5); but we do need some configuration settings to support ZFS instead:
    : > /mnt/etc/fstab
    echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
    echo 'kern.geom.label.disk_ident.enable="0"' >> /mnt/boot/loader.conf
    echo 'kern.geom.label.gptid.enable="0"' >> /mnt/boot/loader.conf
    echo 'vfs.zfs.min_auto_ashift=12' >> /mnt/etc/sysctl.conf
    echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
  8. And in order to make ZFS expand when we first boot, we need an updated script for /etc/rc.d/growfs:
    svnlite export
    cp growfs /mnt/etc/rc.d/growfs
  9. Finally we can "poweroff" the instance
    shutdown -p now
    and then use the EC2 Management Console to create an image from the instance (or run aws ec2 create-image).

Thanks to the freebsd-cloud and freebsd-fs mailing lists, and Ben Woods in particular, for testing and feedback on my earlier attempts.

Posted at 2019-02-16 21:50 | Permanent link | Comments
blog comments powered by Disqus

Recent posts

Monthly Archives

Yearly Archives