My EC2 wishlist

I've been using Amazon EC2 since 2006, and I've been maintaining the FreeBSD/EC2 platform for over a decade. Over those years I've asked Amazon for many features; some of them, like HVM support (EC2 originally only support Xen/PV) and bidirectional serial console support (EC2 originally had an "output-only" serial console) eventually arrived, but I'm still waiting for others — some of which should be very easy for AWS to provide and would yield very large benefits.

While I've made engineers inside Amazon aware of all of these at various times, I think it's time to post my wishlist here — both so that a wider audience inside Amazon can hear more about these, and so that the FreeBSD community (especially the people who are financially supporting my work) can see what I'm aiming towards.

AWS Systems Manager Public Parameters

FreeBSD release announcements currently include a long list of AMI IDs — two for each EC2 region — and I would publish more AMIs if it weren't for the impracticality of putting all the AMI IDs into the announcements. One might say "there's got to be a better solution" — and indeed there is: AWS Systems Manager Public Parameters. Amazon publishes AMI IDs for Amazon Linux and Windows via the AWS Systems Manager Parameter Store, and Ubuntu AMI IDs are also published via the same mechanism (I assume by Canonical). I wrote code over a year ago to allow FreeBSD to publish AMI IDs the same way, but we can't use it until Amazon authorizes the FreeBSD release engineering account to publish these parameters — and we're still waiting.

In addition to allowing us to publish multiple AMIs (e.g. ZFS and cloud-init), if we had this then we could publish updated AMIs after every security update — using the Parameter Store to allow users to look up the latest updated version — which would dramatically speed up the process of launching new FreeBSD/EC2 instances.

Wishlist item #1: Please give the FreeBSD release engineering account access to store AWS Systems Manager Public Parameters.


A few months ago, Amazon started supporting UEFI booting on newer x86 instances. (ARM instances already used UEFI.) This is great news for FreeBSD, since we can boot much faster on UEFI than via the "legacy" BIOS boot mode — I/O is much faster since UEFI doesn't need to bounce disk reads through a small buffer in the bottom 1 MB of address space, and console output is much faster since we can use the UEFI console rather than a shockingly slow emulated VGA text mode. In fact, the total loader + kernel time (starting when the boot loader starts running, and stopping when the init process is spawned) drops from 10.9 seconds down to 3.9 seconds!

There's just one problem with this: AMIs are marked as either "legacy-bios" or "uefi", and while legacy-bios AMIs can boot on all of the x86 instance types, the UEFI-flagged AMIs can only boot on the instance types which support UEFI. FreeBSD's AMIs are built from disk images which support both boot methods — but when we make the EC2 RegisterImage API call, we have to specify one or the other. While we would love to make FreeBSD AMIs boot faster, we don't want to drop support for customers who are using older instance types.

Wishlist item #2: Please add a new "BootMode=polyglot" option, which marks AMIs as supporting both legacy-bios and uefi boot modes, with UEFI being used on instances where it is available and legacy-bios being used otherwise.

Attaching multiple IAM Roles to an EC2 instance

IAM Roles for EC2 are a very powerful — but very dangerous — feature, making credentials available to any process on the instance which can open a TCP connection to Last year, I released imds-filterd, which allows access to the EC2 Instance Metadata Service (and thereby IAM Roles) to be locked down; as a result, you can now attach an IAM Role to an EC2 instance without the risk that a user-nobody privilege escalation allows an attacker to access the credentials.

There's only one problem: You can only attach a single IAM Role. This means that — even with imds-filterd restricting what each process can access in the metadata service — there's no way to give different credentials to different processes. This becomes a problem if you want to use the AWS Systems Manager Agent, since it requires credentials exposed as an IAM Role; there's no way to use the SSM Agent and another process which also uses IAM Role credentials without them both having access to each other's privileges. This even became a problem for Amazon a few years ago when they wanted to provide "extra" credentials to EC2 instances which could be used to manage SSH host keys: Because these credentials couldn't be attached as an IAM Role, they were exposed via the Instance Metadata Service as meta-data/identity-credentials/ec2/security-credentials/ec2-instance which Amazon's documentation helpfully marks as "[Internal use only]".

As it turns out, the EC2 API already supports attaching an array of IAM Roles to an instance, and the Instance Metadata Service already supports publishing credentials with different names — but the EC2 API throws an error if the array of IAM Roles has more than one name listed in it. Get rid of that restriction, and it will become much easier to properly effect privilege separation... and also easier for Amazon to provide credentials to code it has running on customer instances.

Wishlist item #3: Allow multiple IAM Roles to be attached to a single EC2 instance.

If you work at Amazon and can make one or all of these wishes come true, please get in touch ( I really don't think any of these should be very difficult to provide on Amazon's side, and they would provide a huge benefit to FreeBSD. Alternatively, if you work at Amazon and you're screaming at your laptop "it's not that simple Colin!", please get in touch anyway (yes, I've signed the necessary NDAs).

And if you don't work at Amazon but you work at a large AWS customer: Please draw this list to the attention of your Amazonian contacts. Eventually we'll find someone who can make these happen!

Posted at 2021-06-16 01:45 | Permanent link | Comments

Recent posts

Monthly Archives

Yearly Archives