Tarsnap: No heartbleed here
By now I assume everyone is aware of the "Heartbleed" bug in OpenSSL. I wasn't planning on commenting on this, but considering how many emails I've received about this I've decided that I need to make a public statement: Tarsnap is not affected by this vulnerability.There's a few reasons for this. First, the Tarsnap client-server protocol does not use TLS; instead, it uses a protocol which is about 100x simpler (in terms of the number of pages of documentation required to define it). This means there's less code and far fewer edge cases where bugs can lurk without being triggered on a regular basis. The most secure code in the world is code which is never written; the least secure code is code which is written in order to comply with a requirement on page 70 of a 100-page standard, but never gets tested because nobody enables that option.
So the Tarsnap service is completely unaffected; what about the Tarsnap website, where SSL/TLS is used? There's nothing very sensitive accessible from the Tarsnap website — unlike other backup services, knowing someone's account password will not allow you to access their data — but if you managed to steal cryptographic keys used by the website you could see how much data someone had stored or create an account without performing the "receive a confirmation email and click on a link" handshake. But this is exactly why, almost five years ago, I wrote about securing an HTTPS server by terminating SSL connections with stunnel: In the event of vulnerabilities in the SSL stack, this keeps the bug away from anything sensitive.
Mind you, I was also lucky: The Tarsnap webserver happens to be running an older version of OpenSSL which never had the vulnerable code, so in this case I could have done without any exploit-mitigation techniques. But luck is a fickle mistress; I don't expect to have her on my side the next time a critical vulnerability is found in OpenSSL.
As I commented a few years ago, we don't assess the structure of bridges by asking "has it collapsed yet?", and we shouldn't assess the security of software by asking "can we find any security flaws in it?". Continuing the metaphor, a good engineer designing a bridge won't assume that it will always be in perfect condition; she will design it with enough redundancy that even if a truck damages support struts the undamaged portions of the bridge structure will suffice to ensure that the bridge does not collapse. In software security, this means relying on multiple layers of security: Start by assuming that some of them will be compromised, and design your system to ensure that your entire system does not collapse as a result.
We know how to do this. All it takes is some care and attention.
Tarsnap price cut
On Tuesday of last week, Google cut prices for their Google Cloud Platform services. Not to be outdone, less than 24 hours later, Amazon responded by cutting prices on several Amazon Web Services offerings, including the Simple Storage Service where Tarsnap stores customer data. Now it's my turn: Effective April 1st (I nearly announced this yesterday, but decided to wait until the April Fools' jokes were out of the way) I'm cutting Tarsnap's bandwidth and monthly storage pricing from $0.30/GB to $0.25/GB.This will no doubt annoy my friends Patrick McKenzie and Thomas Ptacek, who for years have been telling me that I should raise Tarsnap's prices. But while Thomas accuses me of running Tarsnap like a public utility rather than a business, and thinks this is a characteristic to be avoided, I see this as a high compliment — and so like a good public utility would, I looked at Tarsnap's revenue and expenses (not just the per-GB costs Tarsnap pays for S3 storage, but also S3 per-request fees, EC2 instances, bandwidth, payment processing costs, and what I consider to be a reasonable salary to pay for the time I spend on Tarsnap), realized that I didn't need Amazon's price cut, and decided to pass it on to Tarsnap users.
Does this mean I will always pass on any future Amazon price cuts? I'm not going to make any promises; Tarsnap's operating costs may go up (if I decide to hire someone, for example). Conversely, as Tarsnap grows I may be able to cut prices further, thanks to overhead costs being amortized over more Tarsnap usage, even without any further price cuts from Amazon (and considering how large these recent price cuts were, I'm not expecting to see any more from Amazon in the near future). All I'm saying is that right now I don't feel any need to increase Tarsnap's profit margins, and rather than pocketing money I don't need, it only seems fair to pass this unexpected windfall on.
I hope Tarsnap users will appreciate the savings — but if not, I'm just an email away: I can always sign people up for a special Patrick McKenzie surcharge.
Tarsnap now accepts Bitcoin
A year and a half ago, I announced that Tarsnap was gaining support for credit card payments, as one of the first companies in Canada to use Stripe's newly internationalized payment processing services. This satisfied a long-standing request from Tarsnap customers — until that point, Tarsnap was only accepting payments via PayPal, a service which for a variety of reasons many people did not want to use. Today I'm happy to announce that Tarsnap is satisfying another frequent request, again with help from Stripe: Tarsnap is the first user of Stripe's support for Bitcoin payments.There is a significant commonality between Tarsnap and Bitcoin — both rely crucially on strong cryptography, and anecdotally it seems that many Bitcoin users rely on Tarsnap to back up their "wallets" (I have no concrete data since I can't tell what data people are backing up) — so it's not surprising that many Tarsnap users would like to pay with bitcoins. There is an even stronger connection to many of the "alternative cryptocoins", since many make use of the scrypt key derivation function which I developed for securing Tarsnap's passphrase-protected key files. I don't have any plans to support those other cryptocoins yet, however — although if Stripe adds support for them I have no doubt that Tarsnap will make use of it.
Stripe's support is crucial here due to the nature of Bitcoin: It doesn't have all the qualities normally expected of money. In particular, Bitcoin's deflationary nature makes it very prone to speculative bubbles, making it a terrible store of value. and a poor unit of account. However, Bitcoin truly shines as a medium of exchange — with its low transaction costs and irreversibility, Bitcoin is the cash of the internet — but since it's not practical to quote long-term prices in bitcoins, people typically deal in terms of "X dollars worth of bitcoins". Stripe takes care of this for me: I tell them I want to get paid X dollars in bitcoins, and they tell me how many bitcoins I should ask for and what address they should be sent to. Stripe then gives me the dollars I asked for (minus a small processing fee, of course).
Stripe isn't the first company to offer "fiat currency denominated" Bitcoin payment processing services, but they are the first service I've found which has a simple and clean API. As with their credit card processing code, most of the payment process takes place using publishable API keys and client-side javascript; it is only the final step, of taking an opaque token and issuing an API call to say "turn this token into money", which requires the use of a Stripe account's private credentials. Of course, using javascript like this can introduce a security problem: Javascript loaded into a page executes within its domain and can access pages using any credentials (e.g., cookies) held by the web browser. To avoid this problem, I am adopting the same technique I use for credit cards, and loading the payment code — including all the javascript — inside an iframe "sandbox".
And since I went to the effort of implementing this, I figured I might
as well share it: On
paymentiframe.com there is
now (in addition to my earlier Stripe credit card iframe) a publicly
usable iframe-generator for Stripe Bitcoin payments — including
an autogenerated QR code, thanks to an excellent open source
QR code library.
A single HTML tag generates an iframe like this:
and when that redirects the web browser and submits a Stripe "Bitcoin
receiver token" to the provided address, a single HTTPS request to
Stripe is all that is needed to complete the Bitcoin payment process.
While Tarsnap is the first company to accept Bitcoin via Stripe, they've
told me that they're going to make Bitcoin support more widely available
over the coming weeks; you can sign up to participate in their beta test
on their website.
So consider this two announcements in one: Tarsnap users who want to pay in Bitcoin can now do so; and anyone who wants to receive payments in Bitcoin will soon have a very easy option for doing that too. Enjoy!
How to build FreeBSD/EC2 images
I have been building FreeBSD/EC2 images for the past three years, and based on the email I have been receiving, most people have been either using these images directly or modifying them to create images which suit their needs. However, there are some people who want to build their own images ab initio — most often, companies which have products built on "customized" versions of FreeBSD — and while I have helped a few people do this, it's better if my help is not needed. To this end, earlier today I published my code for building FreeBSD AMIs. At its core, this process has two steps: First, building a disk image; and second, turning it into an AMI.The process of building a disk image starts out the same as for any other platform: Slice and partition a disk, create a filesystem, then extract the FreeBSD bits from a release ISO. Prior to FreeBSD 10.0 this was a more complex task, since a custom kernel needed to be built; but FreeBSD now has Xen HVM support in the GENERIC kernel, which ships on release ISOs.
Having installed a "clean" FreeBSD system into a disk image, I then add extra code to support using the image in Amazon EC2. This consists of four ports in the FreeBSD ports tree:
- sysutils/ec2-scripts: This is the only truly essential port, as it contains code for configuring the system to allow logins using the SSH public key provided via EC2; logging the SSH host keys to the EC2 console; and handling configinit data to allow an EC2 instance to be provided with configuration parameters when it is launched.
- sysutils/firstboot-freebsd-update: This port downloads and installs updates to the FreeBSD base system when the EC2 instance first boots. This is important since there may be many important security and errata updates between when the AMI is created and when an instance is launched months or years later.
- sysutils/firstboot-pkgs: This port downloads and installs FreeBSD packages (code compiled from the ports tree) when the EC2 instance first boots. This makes it possible for an instance to be launched with newer code than was available when the AMI was created; and (together with the functionality provided by configinit) makes it possible to launch a new FreeBSD instance and have it available moments later with a set of software installed and ready to use.
- sysutils/panicmail: This port is not particularly specific to EC2, although I have had an earlier version of the code in my EC2 images for a long time. The panicmail system provides an automated mechanism for reporting of FreeBSD kernel panics to a centralized location, and we're hoping it will prove useful in identifying and fixing stability issues — both those related to the FreeBSD/EC2 platform, and those occuring in other FreeBSD environments.
Having created a disk image — a task which my FreeBSD/EC2 build code performs in EC2, writing directly to an Elastic Block Store volume, for convenience — the second step is to turn this into an EC2 AMI. This is made more complex by a bit of history: When Amazon EC2 was first created using Xen, there was only one virtualization method available, namely "paravirtualization" ("PV" for short); so that's what EC2 used. Later, as virtualization gained popularity, Intel and AMD added support into their CPUs for "hardware virtualization", and Xen's HVM mode was born; but EC2 stuck with what was working for them, namely PV mode.
Unfortunately, FreeBSD needs Xen HVM mode (or some later hybrid modes). In 2008, Amazon announced support for running Microsoft Windows inside EC2, which also needed Xen HVM mode but had higher prices due to the Windows licenses involved; but the operating system world — not just FreeBSD, but also Linux vendors — continued to move towards taking advantage of hardware virtualization, and finally in July 2010 Amazon announced their first "Cluster Compute" instances with HVM support; for the past three years, all of the new EC2 instance types have fully supported HVM. For this reason, from a single disk image I build two FreeBSD/EC2 AMIs: One which pretends to be Windows (and has the higher "Windows" hourly pricing) but can run on all instance types; and one which can only run on "modern" EC2 instances types but has the standard "UNIX" hourly pricing.
In an ideal world, creating a "Windows" AMI from a disk image would just require a single API call to add the "Windows" label; alas, EC2 is not ideal. Instead, I use a trick I call defenestration: I launch a Windows EC2 instance; "throw the windows out" by detaching and deleting its boot disk; attach the FreeBSD disk image as the root disk; and then tell EC2 to create an AMI from the "Windows" instance. Since EC2 knows that the instance was launched from a Windows AMI, it affixes the "Windows" label automatically (if it didn't, rebundling Windows instances wouldn't work), providing the HVM environment FreeBSD needs — as well as the "Windows tax" we don't.
Creating the "UNIX HVM" image, on the other hand, is just a single API call — but an API call which until a few months ago most people couldn't make. For some reason (I'm sure there was one, but I can't imagine what it was) when Amazon first made HVM available on Cluster Compute instances, they decided to keep the process for creating HVM images secret. Fortunately, the right people within Amazon knew about my work on FreeBSD/EC2, and — under a Non-Disclosure Agreement — provided me with special EC2 "API tools" code and set an "allowed to create HVM images" flag on my AWS account. (This is far from the only thing Amazon has quietly done to help me, but it's something they've confirmed I can now talk about publicly.) Fortunately, Amazon changed their stance on HVM image creation a few months ago, and the VirtualizationType parameter to the RegisterImage API call is now publicly available and documented — which is why I'm able to share my EC2 image building code publicly rather than limiting it to people who have signed Amazon NDAs.
While I'm currently building both "Windows" and "UNIX" images, I imagine that at some point I will stop building the "Windows" images; as I mentioned above, all the new EC2 instance types since 2011 support running FreeBSD without the "Windows" label, and now that Amazon has released the m3.medium EC2 image type at a price of $0.11/hour there aren't many situations left where it would make sense to run one of the older instance types.
I think most FreeBSD users will still find it easiest to continue using the FreeBSD/EC2 images which I am publishing, but for those people who want to build their own images: the code is out there — have fun!
Email delivery headaches
Email delivery used to be easy. Server A connects to Server B over SMTP, states that it has a message for Bob from Alice, then sends the message text ("We meet at midnight"). This worked fine until spam came along; to cope with a deluge of spam, an extra step was added, namely "Server B decides whether it trusts Server A to provide email from Alice for Bob". Even then, it wasn't too bad; sure, wide swaths of IPv4 address space were blacklisted, but if you had a server at a reputable ISP, you would probably not be on any of those lists. Then cloud computing happened.With the launch of Amazon EC2 in August 2006, life suddenly became much easier for spammers: If they found that their IP address was blacklisted, all they had to do was pay another $0.10 to spin up an EC2 instance with a different IP address to keep sending spam. While Amazon started out by disabling spammers' accounts, this quickly proved infeasible; and before long, the entire EC2 address space was on "dynamic IP space" blacklists. When I launched Tarsnap this wasn't a problem for me since I sent all my outgoing email through a server at RootBSD; but a few years ago I decided to move everything to Amazon Web Services, for ease of management, and so I had to stop sending email out directly.
My first choice was to use Amazon's Simple Email Service — after all, it was designed specifically for the use case of "company with servers in EC2 needs to send email". Alas, SES couldn't satisfy my needs: For spam and scam prevention reasons, SES applies strict filters to outgoing mail. Not only are SMTP envelope sender addresses required to be on a pre-verified list, but message From: addresses also need to be pre-verified. For emails from Tarsnap to Tarsnap users this wouldn't be a problem — but Tarsnap also has several public mailing lists, and when Tarsnap users post to the lists, their email goes out (via Tarsnap's mail server) with their original From: header.
Having decided that I could not use Amazon SES, I turned to another email-delivery-as-a-service company, SendGrid. For two years I was generally satisfied with them; I occasionally had Tarsnap users report that gmail has marked messages as spam, but I chalked that up to Google's spam filters being overly aggressive and decided there wasn't much I could do about it. A couple days ago, however, a Tarsnap user reported to me that the DKIM headers being added by SendGrid were invalid — not good! — and worse, a significant proportion of email he received via SendGrid had been failing DKIM checks for months. While SendGrid replied via twitter to say that they were aware of the issue and working on a fix, this was enough to make me reconsider my email-sending strategy.
My new strategy is to combine Amazon SES and SendGrid. When outgoing email needs to be sent by qmail, it no longer goes straight to the standard qmail-remote binary; instead, I replaced that with a shell script which examines the sending address. If that address is in the list of addresses I have white listed with Amazon SES, a qmail-remote-ses binary is invoked to pass the message to Amazon; I trust Amazon to do a good job of making sure email gets delivered (since, after all, they have a lot of experience with email from their retail business), and this will take care of all the most important Tarsnap email. Messages going out with other addresses — mainly email from the public mailing lists — are passed instead to a qmail-remote-smtp binary (which is in fact the old qmail-remote binary) and make their way out via SMTP to SendGrid.
Will this be good enough? I certainly hope so; considering how essential email has been to the development of the internet, and how much it is still relied upon by other services — an email address is a de facto internet-wide user name — it seems absurd how far one needs to go to get email successfully delivered. And if I find it this much of a hassle... well, I'm sure I can't be the only person suffering this headache.