A FreeBSD AMI Builder AMI

I've been working on the FreeBSD/EC2 platform for a long time; five years ago I finally had it running, and for the past few years it has provided the behaviour FreeBSD users expect — stability and high performance — across all EC2 instance types. Making the platform work was just the first step though; next comes making it usable.

Some people are happy with simply having a virtual machine which runs the base FreeBSD system; for them, the published FreeBSD/EC2 images (which, as of FreeBSD 10.2-RELEASE, are built by the FreeBSD Release Engineer) will be sufficient. For users who want to use "stock" FreeBSD but would like to have some extra setup performed when the instance launches — say, to install some packages, edit some configuration files, and enable some services — I wrote the configinit tool. And for users who need to make changes to FreeBSD itself, I added code for building AMIs into the FreeBSD source tree, so you can take a modified FreeBSD tree and run make ec2ami to generate a reusable image.

There was one group for whom I didn't have a good solution yet, however: Users who want to create FreeBSD AMIs with minor changes, without wanting to go to the effort of performing a complete FreeBSD release build. Ironically, I am exactly such a user: All of the EC2 instances I use for my online backup service make use of spiped to protect sshd and provide encrypted and authenticated tunnels to my mailserver and package server; and so having spiped preinstalled with the appropriate keys would significantly streamline my deployment process. While it's possible to launch a FreeBSD EC2 instance, make some changes, and then ask EC2 to create a new AMI out of it, this rarely produces a "clean" AMI: A lot of code runs when an EC2 instance first launches — creating the ec2-user user, installing the appropriate SSH public key, creating SSH host keys, growing the root filesystem if launched with a larger root disk, downloading and installing updates to FreeBSD, downloading and installing packages... — and much of this needs to be manually reverted before a reusable AMI can be created; not to mention command histories and log files written during the configuration process, which the more fastidious among us may wish to avoid publishing. To solve this problem, I present the FreeBSD AMI Builder, now available as ami-28682f42 in the EC2 US-East-1 region.

Much like my depenguinator, the FreeBSD AMI Builder works by booting a full FreeBSD system into a memory disk; having done that, it no longer needs to access anything on disk, and so it can safely unmount and overwrite the existing on-disk filesystem. A fresh FreeBSD/EC2 image is written onto the disk and then mounted at /mnt, at which point you can SSH into the instance — which, since it is running in a memory disk, can allow you to SSH in without touching the image you're creating — and then make whatever changes you'd like. (The one minor disadvantage to running in a memory disk is that t2.micro instances don't have enough RAM; all the larger instance types work fine, though, and this limitation only applies to the process of building AMIs, not to instances you launch later.)

This becomes even better with a clever feature Amazon added to EC2 a few years ago: IAM Roles. In short, IAM Roles allow you to provide ephemeral AWS keys to an EC2 instance. To make use of this, create an IAM Policy which grants access to the EC2 "CreateImage" API call, and an IAM Role which has that policy attached; then launch the FreeBSD AMI Builder with that Role attached. Now you can run

# mkami "My AMI Name" "My AMI Description"
and through the magic of IAM Roles the EC2 instance will be able to issue the necessary API call to create an AMI out of itself.

For all that the FreeBSD AMI Builder is a useful tool, it also demonstrates what can be done using configinit: I created the AMI by launching a FreeBSD/EC2 instance with this script provided as EC2 user-data. When the instance booted, it fetched the EC2 user-data; identified it as being a shell script and executed it; and that script assembled the disk image required for running the AMI Builder without my ever having to SSH into the instance. In short, that shell script is a FreeBSD AMI Builder AMI Builder.

But wait, there's more! The FreeBSD AMI Builder is itself a FreeBSD/EC2 AMI, so it too has access to configinit. If you set up the aforementioned IAM Role and launch the AMI Builder with this EC2 user-data

pkg -r /mnt fetch -d apache24
pkg -c /mnt install apache24
echo apache24_enable=YES >> /mnt/etc/rc.conf
mkami "FreeBSD 10.2 w/ Apache 2.4" "FreeBSD with Apache pre-installed"
shutdown -p now
it will install Apache 2.4, enable it, create a new AMI, and then shut down the AMI Builder instance. (The separate pkg commands for downloading and installing packages are necessary to work around a bug in pkg relating to hard links and alternate roots.) A short time later, and you have an AMI with Apache 2.4 preinstalled and enabled, ready to be launched. Of course in this case you could simply use the "stock" FreeBSD AMI and have Apache installed at boot time using the appropriate EC2 user-data; but a clean and repeatable customized AMI build in less than ten minutes is nonetheless impressive!

One of the difficulties I've encountered in getting people to use FreeBSD on EC2 is that there are very few "customized" FreeBSD/EC2 images available. I know new EC2 users often find preconfigured images very helpful, and once they start out on Linux they tend to stay there. While I see lots of exciting things happening in the FreeBSD community, we're not nearly good enough at showing it off to the rest of the world. This may make that final step a little bit easier.

Finally, if you find this useful, please consider donating to the FreeBSD Foundation. As I write this they're at $481,324 out of a target of $1,250,000 of donations for the year, with less than six weeks left; they really do wonderful work supporting the FreeBSD project and deserve all the support we can offer them.

Posted at 2015-11-21 12:15 | Permanent link | Comments

Tarsnap email confirmation bypass

Over the past four years, Tarsnap's bug bounties have received quite a bit of attention. Most of it has been very useful — almost 400 mistakes (most either cosmetic or harmless, but some of them significant) have been reported and fixed — but it does also get some unwanted attention: Despite my clear statement that Tarsnap's bug bounties are for problems in tarsnap code, not for problems in the website, I regularly see people running automated vulnerability scanners... which invariably yield a selection of absurd non-vulnerability "vulnerabilities".

One consequence of these unsolicited security scans is that — since they feed a variety of inputs to forms, including the account creation form — I see a lot of obviously fake signup attempts (alas, none yet from the world's most obviously fake domain name). These are harmless, since the signup code sends out a confirmation email and the account isn't actually created until the alleged registrant follows a link in that email; so I wasn't concerned when I received an email last week telling me that someone was trying to create an account as admin@tarsnap.com.

Five minutes later, I was very concerned upon receiving an email telling me that the registration for admin@tarsnap.com had been confirmed and the account created.

This should not have happened, so I immediately started running down a list of possibilities. Was it a forged email? No, the headers showed it being delivered from the CGI script to the tarsnap web server's qmail to the tarsnap mail server's qmail to my inbox. Was a copy of the confirmation email — which should never have gotten past the mail server — being misdelivered somehow? No, the mail logs showed that the email to admin@tarsnap.com went from CGI script to the web server's qmail to the mail server's qmail and then was dropped. Was one of the CGI scripts on the tarsnap web server compromised? There was nothing in the logs to suggest a malformed request of the sort which might have been used to exploit a bug; nor, for that matter, anything to suggest that someone had been probing for bugs, so if a CGI script had been exploited, it was done completely blindly. Nevertheless, I disabled the CGI scripts just in case.

Had someone managed to compromise the web server or mail server? I saw no unexpected processes running; nothing abnormal in the logs; and neither server had rebooted recently (as might be required to install a kernel root kit). Of course you can't rely on anything from within a system to reliably diagnose whether it has been compromised, and I wasn't running auditdistd, so I couldn't be absolutely certain... but realistically, if an attacker was sophisticated enough to compromise either system without leaving any trace, how likely was it that they would blow their cover by creating an obviously impossible tarsnap account?

The mystery was solved a few minutes later when an email arrived from Elamaran Venkatraman: He hadn't compromised any servers or exploited any bugs in my C code; rather, he had found a dumb mistake in tarsnap's account-creation process.

For most people to create a Tarsnap account, only a few things are required: An email address, a password, and checkbox confirming that you agree to the Tarsnap legal boilerplate. You submit those to the Tarsnap server; it generates a registration cookie; it sends that cookie to you as part of a URL in the confirmation email; and when you click through that link and re-enter your password your account is created. So far so good — but some people need a bit more than that. Tarsnap is a Canadian company, and as such is required to remit sales tax for its Canadian-resident customers. Moreover, Tarsnap is required to issue invoices to its Canadian-resident customers — invoices which show the customers' physical mailing addresses — so if a registrant identifies themself as being a Canadian resident, they are taken to a second page to provide their name and mailing address.

But what of that confirmation email? Well, I didn't want someone who self-identified as a Canadian resident to create an account without providing the legally-mandated information, so I couldn't send out that email until they submitted the second page. On the other hand, they having provided their email address and password once already, I didn't want to ask for those again. And so, when I finally got all the paperwork sorted and started accepting Canadian customers in July 2012, I took the option which was simple, obvious and completely wrong: I passed the registration cookie as a hidden variable in the second-page form, to be echoed back to the server.

This of course is what Elamaran had found. To be clear, the registration cookie didn't reveal any server internals; the only thing it could be used for was to confirm an email address. But because it was being sent in the HTML response, anyone could "confirm" any email address, simply by claiming to be a Canadian resident and viewing the page source. Oops.

The fix for this was easy: Use two cookies, one for email confirmation and one for the Canadian-address-obtaining continuation. More importantly, I've moved the cookie-generation to where it belongs — within the routine which generates and sends the confirmation email — and I've added a comment to remind myself that the cookie must never be exposed via any channel other than an email destined for the address being confirmed.

That last part is ultimately the most important lesson from this: Comments matter! I don't know what I was thinking three years ago when I reused that cookie; but unless my memory was much better then than it is now, I almost certainly wasn't thinking about my original design from four years prior. While this was hardly a fatal bug — while I'll never know for certain, I doubt anyone exploited this email confirmation bypass, and the impact would not be severe even if someone did — it's an reminder of the importance of writing useful comments. I often see solo developers excuse a lack of comments in their code on the basis that they understand their code and nobody else will be touching it; this misses an essential point: I am not the same person as I was three years ago, nor do I understand everything I understood three years ago.

People make mistakes, and people edit code without fully understanding how and why it works. Leave breadcrumbs behind, even if you don't intend for anyone to follow you: When you try to retrace your steps, you might get lost without them.

Posted at 2015-09-04 08:00 | Permanent link | Comments

Safe from what?

I woke up this morning to a headline news story on CBC's website: Is your baby monitor safe? According to a literal reading of Betteridge's law of headlines, the answer to this question should be "no", although if you consider the spirit of the law — as a commentary on sensationalist journalism — then the answer should probably be "yes". To me, neither answer makes sense, because the question itself doesn't make sense.

What does it mean for a baby monitor to be safe? Some things are clearly essential: It should not explode; it should not release significant quantities of dangerous gasses; if it's designed to be placed within reach of a baby, it should not pose an electrocution or choking hazard. Less obviously, if it has data or power cables, those should be arranged to avoid effecting tripping hazards. On the other hand, it's far from clear if privacy is an essential component of "safety". Does it matter if someone within range of your wireless network can listen to your baby crying? Probably not — anyone that close to you has likely heard far more of your baby's crying than they want already. Similarly, unless you never leave home, video of your baby sleeping is readily available to anyone who passes you in the street.

The word "safe", like the word "secure", covers a wide range. Is a Tesla car safe? For most people driving through the streets of Vancouver, it's very safe. But if you're driving through a war zone, having a vehicle which is designed to protect you from bullets and shrapnel is probably more important than having a vehicle designed to protect you in a head-on collision. The answer to whether a vehicle — or a baby monitor — is safe isn't yes or no; it's another question: "Safe from what?"

I spent seven years as Security Officer for the FreeBSD operating system, and over the years I wrote somewhere around a hundred advisories for vulnerabilities in FreeBSD. We used a consistent format: Background, Problem Description, Impact, Workaround, Solution. Of these sections, the Impact was always the section I wrote most carefully, and the section I advised users to pay the most attention to; and it usually had a consistent format: "An attacker who..." (description of attacker, e.g., "has an account on a system", or "can connect to your wifi network") "can ..." (description of what the attacker can do, e.g., "gain superuser privileges", or "listen to your baby crying"). Both of those clauses were essential in order for users to decide whether they wanted to update their systems: If there are no attackers whom you're worried about (say, if all of your code runs as root and the vulnerability is limited to nonprivileged users), or you don't care about what attackers can do (say, if the issue limits them to listening to crying babies), then you don't need to worry.

Of course, if you're building a product or supplying a service, you should be concerned about any attacker and any outcome which your potential customers could plausibly be worried about; after all, you're (hopefully!) in a better position to understand the safety and security properties of your product or service than your customers are, but you don't know exactly what attackers or outcomes they will care about. For the online backup service I run, this means being concerned not just about random attackers around the internet, but also considering myself as a possible attacker (since I might be coerced, legally or otherwise); and considering not just disclosure of backups, but also disclosure of whether particular data has been backed up (I call this the RIAA attack: "Has anyone backed up this MP3 file?") and also tampering with backups (can someone change the passwords in your backups so that they can break into your server after you restore from those backups?) as attacks I need to worry about — not because I think everybody is going to be worried about such attacks, but because someone might be.

But if you're simply assessing a product for your own use rather than building a product for a wide audience, you should know what you're worried about and what you don't care about. So don't ask "is this baby monitor safe"; ask "will this baby monitor allow any attackers I'm worried about to do anything I care about".

And if you're writing headlines, please stop using vague terms like "safe" or "secure". The role of a headline isn't, no matter what tabloids might suggest, to convince people to read an article; the role of a headline is to help readers decide if they want to read the article, and imprecision serves no purpose there.

Posted at 2015-09-02 23:30 | Permanent link | Comments

Tarsnap $1000 exploit bounty

For somewhat over four years, Tarsnap has been offering bounties for bugs found in the Tarsnap code. Two thirds of the bounties Tarsnap has paid out have been $1 each for cosmetic bugs (e.g., typos in source code comments), and a quarter of the bugs have been $10 each for harmless bugs — mostly memory leaks in error paths where the tarsnap client is about to exit anyway — but there have also been some more serious bugs: Several build-breakage bugs ($20 each); a variety of cases where tarsnap behaviour is wrong in a user-visible — but generally very obscure — way ($50 each); a few crashes ($100); and of course the critical crypto bug which first convinced me to offer bounties.

Most bugs are straightforward, but occasionally one comes up which is not so clear in its impact. Such is the case with a bug which is fixed in tarsnap 1.0.36. This bug causes the NUL string termination byte to overflow the heap-allocated buffer used for paths of objects examined as tarsnap traverses a directory tree; such one-byte heap overflows have been shown to be exploitable in the past. In the case of tarsnap, I will be very surprised if it turns out to be possible to cause anything worse than a crash, but I can't absolutely rule out the possibility.

In light of this, Tarsnap is offering a $1000 exploit bounty: The first person before the end of 2015 who can convincingly demonstrate a serious exploitation of this bug will receive $1000. While there are many organizations which pay more than this for exploits, I think this is a reasonable prize: After all, I'm already telling you what the bug is which you need to exploit! Fine print: No bounty if you're in Iran, North Korea, or some other problem countries. Bounties are awarded at my sole discretion; in particular, I get to decide whether the "convincingly demonstrate" and "serious exploitation" conditions are satisfied. Payment by US dollar check or paypal. To avoid races, contact me before publishing anything. If you can't accept cash prizes, the bounty can be donated to a mutually-acceptable charity of your choice.

There are two reasons I'm skeptical about the exploitability of this bug. First, the lengths of buffers which can be overflowed are very limited: Just powers of 2 times 1 kB, and only once per size. Sane mallocs will put these sizes of allocations into pages with other 1024 or 2048 byte allocations, or allocate entire pages (for 4kB and larger allocations), and I can't see any "interesting" data structures in tarsnap which we would overflow into. Second, tarsnap's "controllable surface area" is quite limited: Unlike a web server or a setuid program which can be attacked interactively, the avenues for attacking tarsnap seem to be limited to creating interesting directory trees for it to archive and possibly meddling with its network traffic — with that second option being very limited since all of tarsnap's traffic is both encrypted and signed.

But I'm far from an expert on exploitation techniques. In my time as FreeBSD Security Officer, I only wrote exploits for two bugs; for the rest I was satisfied with "this looks like it could plausibly be exploited, so we need to issue a security advisory for it". (There was one occasion when we didn't issue an advisory for a plausible looking "vulnerability", but that was because I discovered by accident that the code path which theoretically yielded arbitrary code execution actually crashed a dozen lines earlier due to an independent bug.) I know there are people who are far far better at this sort of thing than me, enjoy challenges, and may also care about the bounty; I'm hoping some of them will be interested enough to try their hands at this.

The main reason I'm offering a bounty is broader than this specific bug, however. As a researcher, I like to support research, including research into software exploitation techniques; and as a developer I'd like to know what techniques can be applied to exploiting bugs in my code specifically. The more I know, the better I can assess the impact of future bugs; and the more I know, the more I can look for mitigation techniques which will help to reduce the impact of future bugs.

A bug is a bug is a bug, and I would have fixed this bug even if I could prove that it was impossible to exploit. Similarly, I hope that every tarsnap user upgrades to the latest code — there are plenty of other bugs fixed in tarsnap 1.0.36, and even if nobody claims this bounty it's entirely possible that someone figured out how to exploit it but decided to hold on to that for their own personal use. (Rule #1 of bug bounties: No matter how much money you're offering, assume that someone evil found the bug first and didn't report it.)

But knowledge is power, and Tarsnap is fortunately in a position to be able to pay for this. So please, go look for ways that this bug can be exploited — and if you can't manage that, maybe at least you'll find some other bugs which you can win a bounty for.

UPDATE: Thomas Ptacek (founder of Starfighter and general security expert) has offered to match this bounty. So there's now $2000 in the pot for an exploit.

Posted at 2015-08-21 14:00 | Permanent link | Comments

"(null)" is a bad idea

What happens if you compile and run the following simple C program?
#include <stdio.h>

main(int argc, char * argv[])

	printf("%s\n", NULL);
If you believe the C standard, you may get demons flying out of your nose. Most developers who understand the implications of NULL pointers would assume that the program crashes. Unfortunately, on some misguided operating systems, the program exits successfully — after printing the string "(null)".

This is an example of an anti-pattern known sometimes as "defensive programming": If something goes wrong, pretend you didn't notice and try to keep going anyway. Now, there are places for this approach; for example, if you're writing code for a rocket, a plane, or a car where having your software crash is likely to result in... well, a crash. For most code, however, a crash is unlikely to have such serious consequences; and in fact may be very useful in two ways.

The first way a software crash can be useful is by making it immediately clear that a bug exists — and, if you have core dumps enabled, making it easier to track down where the bug is occurring. Now, in the case of printing "(null)" to the standard output, this is probably clear already; but if NULL were being passed to sprintf instead, the resulting string might be used in a way which conceals its bogosity. (In the case of the bug which inspired this blog post, the constructed string was being used as a path, and the resulting "file not found" error was not an unanticipated result of looking up that path.) During the software development and testing process, anything which results in bugs being found faster is a great help.

The second way a software crash can be useful is by mitigating security vulnerabilities. The case of BIND is illustrative here: BIND 4 and BIND 8 were famous within the security community for their horrible track records. BIND 9, in an attempt to avoid all of the problems of earlier versions, was a complete ground-up rewrite — and it is still responsible for an astonishing number of security advisories. However, there is a critical difference in the types of vulnerabilities: While earlier versions of BIND could be exploited in a variety of scary ways, vulnerabilities in BIND 9 almost always take the form "an attacker can cause an assertion to fail, resulting in a denial of service". If something weird happens — say, a NULL pointer shows up somewhere that a NULL pointer shouldn't show up — after software has gone through a lengthy testing process and been released, it's far less likely to be happening by accident; and so it's even more important to abort rather than blindly trying to recover.

Undefined behaviour is undefined behaviour, and developers shouldn't assume that passing a NULL pointer to a function which dereferences its arguments will result in a crash; that's what assert() is for. But at the same time, authors of libraries have a choice when it comes to what behaviour they provide in such circumstances, and when faced by a choice between making the presence of a bug immediately obvious or trying to hide the bug and potentially exposing serious security vulnerabilities... well, I know which I would prefer.

Posted at 2015-08-13 03:05 | Permanent link | Comments

Recent posts

Monthly Archives

Yearly Archives