Daemonic Dispatches

ZumoDrive rolls a hard six

I haven't had much time for blogging recently, but sometimes things come up which just beg for a response; case in point: A recent post to the ZumoDrive blog entitled "Sometimes you have to roll a hard six" about the security of the ZumoDrive cloud storage / backup service. I have to give credit to ZumoDrive for one thing: Unlike most online backup services, they published the reasons why they think their service is secure. Sadly, the credit goes no further.

[EDIT 2010-03-11 18:00 I mention this below, but to place my conflict up front: I'm the author of the Tarsnap secure online backup service, which is in some ways a competitor to ZumoDrive.]

To quote from the blog post,

The file being uploaded is transferred to the ZumoDrive server which is hosted by Amazon's EC2 (Elastic Compute Cloud). It is done via 256-bit SSL encryption.

Here's their first mistake: Trusting their security to SSL. There is one, and only one, good reason to use SSL for anything: It takes care of key management for you. When you're only talking to yourself -- or, as in this case, if you wrote both the client and the server code -- you can pre-distribute keys, and all of the complexities of SSL certificate handling (plus the question of whether or not you should trust the Chinese government) go away.

Next up we have this bit:

The EC2 is the workhorse. It's the liaison between the client on your computer and the ZumoDrive datacenter (which is hosted by Amazon S3; more on this below). It also services the ZumoDrive website.

Even leaving aside the questionable subject "The EC2" -- I assume they meant "The ZumoDrive server hosted on EC2" -- this also raised my eyebrows. One of the keys to security is to keep separate things, well, separate -- that way, you reduce the potential for a vulnerability in one area to affect others. This is why qmail has five daemons (qmail-send, qmail-lspawn, qmail-rspawn, qmail-clean, qmail-smtpd) where other mail transfer agents have only one. Why should the security of your backups depend on whether someone can find a vulnerability in a web server?

A few lines further down, we have this classic goof:

[The ZumoDrive server, running on EC2] encrypts your actual file via 256-bit AES encryption and stores it on S3 (Simple Storage Service). AES is military grade encryption.

As Bruce Schneier wrote in his famous list of signs of cryptographic snake oil, "Some companies claim 'military-grade' security. This is a meaningless term. There's no such standard." Many things have changed in the past decade, but this is not one of them: There is still no such thing as "military grade encryption". There are US standards covering the use of encryption to protect CONFIDENTIAL, SECRET, and TOP SECRET information; but merely using 256-bit AES is nowhere near enough: The entire encryption system needs to be approved (including block cipher modes, key management, vulnerability to side channel attacks, et cetera), not merely the choice of block encryption algorithm.

As it happens, there's an even worse problem: ZumoDrive is trying to keep your data safe on S3 by encrypting it on EC2. Stop and think for a moment: Who does this actually protect you from? The people who have out-of-band access to data on S3 -- Amazon employees, law enforcement, and the courts -- all have access to data on EC2; so if they want your data, all this does is force them to fish the decryption key out of ZumoDrive's EC2 instance.

ZumoDrive's security has another major flaw, too: ZumoDrive themselves. Data is uploaded not-very-securely to their not-very-secure server, which then encrypts it completely-and-utterly-brokenly (but with bogus buzzwords) prior to storing it on S3... during that process, the data is on the ZumoDrive server completely unencrypted. Anyone with access to that server -- which I'm guessing is most or all of the people who work at ZumoDrive, plus anyone who can steal their credentials (or install a trojan which steals their credentials) -- can access ZumoDrive data. Thanks to their automatic indexing of file metadata, they would even know where to find my_world_domination_plans.doc.

Finally, while I have to give ZumoDrive credit for admitting their faults (albeit unknowingly) on their blog, it's really not enough to see what they claim ZumoDrive does. As a FreeBSD developer (and Security Officer), I have to speak in favour of open source software -- not from the perspective of licensing, but simply because without seeing the source code (or doing a huge amount of work to disassemble and reverse-engineer a binary) there's no way to confirm that ZumoDrive correctly translated design into code.

The title of ZumoDrive's blog post -- sometimes you have to roll a hard six -- is a quote from Battlestar Galactica, but refers to the game of Craps where a "hard six" involves rolling a 3 on each of a pair of six-sided dice. In that vein, and taking a bit of poetic license, I'd say that ZumoDrive has rolled three security mistakes

• Relying on SSL for security when keys could be distributed out-of-band.
• Hosting their service on the same systems as their website.
• Encrypting data in EC2 and expecting that to protect it on S3.

and three reasons for mistrusting them

• Calling 256-bit AES "military grade encryption".
• Having unencrypted access to user data.
• Not providing source code.

and declare that ZumoDrive have successfully rolled a "hard six" reasons why security-conscious users should avoid their service.

But who am I to speak? I'm just the author of a competing service -- Tarsnap. An online backup service which does not rely on SSL for its security; has its website hosted entirely separately from the backup service; encrypts data using keys held only by the user before uploading it; doesn't allow me to look at user data (because I don't have the necessary decryption keys); and provides source code for the client utility, so that you can see exactly what it is doing. In short, a backup service which does security right.

Supporting FreeBSD

As a FreeBSD user and developer, I obviously care about the success of FreeBSD. I make a small contribution towards this success via my role as Security Officer; but the time I spend working on my Tarsnap online backup service prevents me from making as much of a direct contribution as I would like. Fortunately the FreeBSD Foundation does an excellent job of supporting FreeBSD development; but like most such organizations, they are funded entirely by donations and are always in need of more. In light of this, I am pleased to announce that I will be donating all of the profits made by Tarsnap for the month of December to the FreeBSD Foundation.

Much of the work done by the FreeBSD Foundation is done behind the scenes, but the importance of the work they do is undeniable. They sponsor a range of important FreeBSD development which would likely not get done otherwise; in the past year this has included

• Improvements to the TCP stack,
• Addition of IEEE 802.11s mesh networking,
• A new console driver,
• Support for removal of active disk devices (i.e., to be able to yank out a USB disk without causing a FreeBSD kernel panic),
• Flattened Device Tree support,
• AVR32 platform support, and
• A new highly-available (clustered) storage layer.

In addition to directly sponsoring FreeBSD development, the FreeBSD Foundation plays an important role in indirectly supporting development work by bringing developers together: They sponsor BSD conferences such as BSDCan, EuroBSDCon, and AsiaBSDCon, and provide funding to allow FreeBSD developers to attend (they paid for my travel to and from BSDCan'06, for instance). Further, although the FreeBSD Project runs mostly on donated computer hardware, there are instances where it is necessary to purchase equipment for development purposes, and here too the Foundation plays a critical role.

Finally, the FreeBSD Foundation plays a vital role in taking care of legal matters, whether it be handling the FreeBSD trademarks, making recommendations on how FreeBSD should handle new licenses (e.g., GPLv3), or representing FreeBSD in dealings with organizations which will only deal with official legally-constituted entities. Much of this work falls into areas where every FreeBSD developer has an opinion yet none of us are qualified to have an opinion; having the Foundation get actual legal advice from one or more actual lawyers not only increases the chance of getting things right, but also cuts out the large amount of developer time which would otherwise be wasted by having everybody express their opinions... and their opinions of everybody else's opinions.

If you use FreeBSD, please join me in donating to the FreeBSD Foundation; and if you're not already using Tarsnap, how about starting to use Tarsnap today, thereby supporting FreeBSD and getting yourself some highly secure backups at the same time?

Looking back at 100 blog posts

I found recently, somewhat to my surprise, that as of my last post I had written exactly 100 of these dispatches. Spread over 49 months, this is not a very high posting rate; but I promised myself when I started that I would limit myself to writing when I felt that I had something worth saying, and would not indulge in the common trend towards excessive introspection (or, in the words I used back in 2005, "adolescent gutspill"), and I believe I've done a good job of holding myself to this standard. Nevertheless, I think this is a good time to look back at four years and a hundred posts and say a few words about this blog.

First, writing each post takes a deceptively long time. I'd say on average it takes me about six hours of research, writing, and editing, usually spread over two or three days, per post. Items which fall into the general category of "news items" are generally much faster, while posts one might refer to as "essays" generally take longer -- which isn't particularly surprising, since they are longer: usually over 1000 words, compared to a median of about 500 words. I've been told that my writing sounds as if I just sat down and wrote, and I take this as a compliment: Just like good music, good writing has a natural-sounding flow which should make it seem easy.

Looking at my HTTP access logs, there's a clear trend in which posts attract the greatest readership. Four of the top five posts concern upgrading to FreeBSD; in order, upgrading from one FreeBSD release to a later major version, upgrading from one FreeBSD release to a future release within the same major version, upgrading from FreeBSD 6.1 to FreeBSD 6.2 (written a year before the more general "same major version" upgrade post), and remotely upgrading a system from Linux to FreeBSD. The other four posts in the top eight are all editorial in nature; on order, an exhortation to think before starting to write code, a discussion of the "security mindset" and how it relates to mathematics, a call for companies using open source software to acknowledge authors by sending schwag, and a set of guidelines for users of cryptography. One post which is not in this top rank yet is my post last month about the security cost of excess complexity, but I expect it will reach that level soon -- a month after posting it is still receiving significant attention and is thus moving up in the rankings.

While I think the above all deserve the attention they have received, there are some other posts which I believe deserve more attention than the little they have received. My post in December 2008 providing details concerning the security flaw in Amazon Web Services version 1 signatures has received very few readers, in spite of its importance; and my post in October 2008 concerning the hackability of the Amazon S3 SLA, while not particularly important, is certainly interesting enough as a mathematical exercise that I think it deserved a larger audience than it received. I think my post with the greatest "readership deficit", however, is a post from March 2007, in which I wrote about the important of knowing your attacker and pointed out that security is really a multidimensional concept: In short, the most secure option given the attackers you want to defend against is not necessarily the most secure option given the attackers I want to defend against.

Whither next? I'm sure I will continue to write about topics ranging from FreeBSD to mathematical puzzles, but -- subject, of course, to having something I think is worth saying -- I'd particularly like to write more about security and cryptography: both about how to do it right, and also about how to recognize when it is done right. (These are not nearly as related as they seem -- knowing how to make a movie and being able to judge a movie's quality are very much disjoint skills.)

I have two reasons for wanting to write about security and cryptography. First, as an academic (albeit one presently embedded in industry) I value learning very highly, and feel a duty to help educate people about fields in which I have expertise. There are some people who argue that the dangers posed by novices meddling in cryptography are so great that we should avoid anything which might lead them into such attempts -- that we should instead wrap the field in mystique and teach people only that they should use pre-existing libraries. While I have great respect for my colleagues who espouse such views, I find them entirely wrong-headed in this regard; I do not feel that any knowledge is so dangerous as to make its research and teaching to a reasonable audience undeserved.

My second reason for wanting to write about security and cryptography is rather less philosophical and far more selfish: I believe that my Tarsnap online backup service is exceptional in its security -- but as long as most people are unable to distinguish good security from bad, this does little to attract new users to the service. I did not make Tarsnap secure because I thought it would win me many customers -- I made it secure because I don't want to be responsible for someone losing their data -- but the fact remains that I'd love to see more people using Tarsnap, and educating the world about security is one way to do that.

The above notwithstanding, my selection of topics on this blog -- or, more precisely, the decision, given an interesting topic, of whether or not to take the time required to write about it -- is not entirely based on my own personal preferences: I have acquired a modest audience (I was astonished to find that my RSS feed has over 1000 subscribers) and to a certain extent my writing choices are influenced by feedback from readers.

So tell me, dear readers: What would you like me to write about?

Securing an HTTPS server

In response to numerous comments about "excessive minimalism", I recently put together a new website for my Tarsnap online backup service; and since I was reworking things anyway, I decided that it was a good time to move to a new web server and generally clean up the system configuration. Among the things I cleaned up was how I handle HTTPS: I need it because people enter passwords when creating tarsnap accounts and when logging in to the tarsnap account management interface, but I wasn't satisfied with the (in)security of running Apache with SSL enabled.

There are two reasons why I don't like the idea of running Apache with mod_ssl and OpenSSL to service HTTPS requests. First, SSL is a complex protocol, and thus implementations are likely to be buggy and insecure; and second, SSL certificates are sensitive information. If someone finds a vulnerability in OpenSSL, I don't want them to be able to leverage that into accessing anything else on the server; and likewise, if someone finds a vulnerability in something else (Apache, for instance, or -- although it would greatly shock me -- one of the tarsnap CGI scripts) I don't want them to be able to leverage that into gaining access to the SSL certificate for www.tarsnap.com.

In order to maximize the separation between systems on this machine, I set it up following the "service jail" model: Using FreeBSD's jail functionality (roughly speaking, chroot redesigned to be secure), I have daemons running each in their own "world" and bound to individual non-routable IP addresses; and then I use the ipfw firewall and natd daemon to translate and pass packets back and forth to the external interface. Among the jails on this system, I have a package-building jail on IP address 192.168.128.1; a dns cache jail on IP address 192.168.128.53; an outgoing mail daemon on IP address 192.168.1.25; and an authoritative nameserver on IP address 192.168.0.53.

TCP connections incoming to port 80 on www.tarsnap.com are redirected by natd to an Apache process running on IP address 192.168.0.80, while incoming connections to port 443 -- that being the standard port for HTTPS traffic -- is redirected to IP address 192.168.0.44. There they encounter the stunnel daemon, which is running with the following configuration file:

chroot = /var/run/stunnel
pid = /stunnel.pid
setgid = stunnel
setuid = stunnel
output = /var/log/stunnel.log
syslog = no
[https]
accept = 192.168.0.44:443
connect = 192.168.0.80:80
local = 192.168.0.44
cert = /usr/local/etc/ssl/server.crt
key = /usr/local/etc/ssl/server.key

As one might expect from the configuration file, stunnel takes the incoming HTTPS connections, performs the necessary SSL negotiation, and then passes the underlying HTTP connection on to the Apache daemon at 192.168.0.80. Now if someone compromises OpenSSL, they will still be stuck within the stunnel jail -- able to steal the SSL certificate, to be sure, and also able to intercept other HTTPS connections -- but unable to do anything to Apache (or the CGI scripts which Apache launches) aside from what could already be done from the outside world.

There is one slight inconvenience to this configuration, however: Because HTTPS connections are proxied via stunnel, Apache's logs show stunnel's IP address as the source of each connection, with obvious consequences for log analysis. That said, there is a workaround available: Apache 2.2 can be instructed to log source port numbers, and stunnel logs all connections it handles -- including the source IP address of the incoming connection and the port number used for opening up the connection to Apache. Consequently, it is possible to correlate Apache and stunnel's logs in order to generate a traditional HTTP log file including "real" source addresses. (I might blog further about this in the future -- if you're interested, please leave a comment below.)

Good system security requires multiple layers. It isn't enough to use software, like Apache, mod_ssl, and OpenSSL, which have no presently known vulnerabilities, and upgrade them whenever vulnerabilities are found. Designing secure systems requries accepting that security vulnerabilities will be found -- and exploited before they can be fixed -- and putting components together in such a way as to minimize the potential impact of such vulnerabilities.

Complexity is insecurity

As I've been writing code for my Tarsnap online backup service over the past three years, I've gone out of my way to make it as secure as possible. I've written previously about the importance of carefully designing security systems before writing any code, thinking about mathematical proofs-of-correctness while writing code, cryptographic research concerning key derivation functions, and recommendations for using cryptography, all of which have informed my work on tarsnap; and I've made the tarsnap client source code available for public review -- after all, I refer to tarsnap as being "Online backups for the truly paranoid", and nobody who is truly paranoid would want to download and run code without inspecting the source code and compiling it themselves. However, there is a very important aspect of tarsnap's security which I haven't discussed previously: Complexity -- or rather, a lack thereof.

Complexity can be thought of as a type of code smell: It doesn't necessarily imply that there is a problem, but the presence of complexity is very strongly correlated with the presence of security vulnerabilities. In the design and construction of secure systems, it is important to not only consider mistakes which are guaranteed to cause problems, but to also consider factors which make it more likely that problems will arise -- or, put another way, factors which make it harder to get things right. The notion of "complexity" is largely subjective, but our intuition serves us well; as I'll discuss, there are many factors which can be intuitively seen as constituting "software complexity" and have consequences for software security.

The most obvious such factor is code size. If you have a fixed rate of bugs per line of code -- and evidence suggests that this is a reasonably accurate approximation -- then a project with twice as many lines of code will have twice as many bugs. Now, some bugs are harmless, and many harmful bugs are not security flaws; but there is still a general correlation between code size and security flaws. Considering adding a new feature to your application? Ask yourself not just whether it's worth the time required to implement the feature, but also whether it's worth the risk of introducing new bugs and security flaws.

A less obvious factor is the number of authors. One of the most common sources of bugs -- and security flaws -- is miscommunication between developers. When function A calls function B, which function is responsible for sanity-checking function B's inputs? If an object encounters an internal error, should its deconstructor be called during cleanup? Are strings NUL-terminated ASCII, or NUL-terminated UTF-8? Formal methods of software development, wherein functions' preconditions and postconditions are explicitly spelled out (in some cases, explicitly enough to be used by machine proof systems), are in many cases advantageous more because they avoid such miscommunications than for any other reason.

One oft-cited measure of software complexity is the cyclomatic complexity -- that is, the number of linearly independent cycles in the control flow graph, or roughly speaking, the number of conditional branches in the program. Where security is concened, there is a slightly more nuanced metric which is more important -- the number of rarely taken conditional branches -- because in any program, the place where bugs are most likely to be hidden is in code which is almost never executed. There are two reasons for this: First, having a large number of users provides, in effect, a great deal of fuzz testing, which is likely to uncover bugs in those code paths which are commonly invoked; and second, as software is continually developed, regions of code which are rarely executed tend to receive far fewer eyeball-hours of attention, with the result that bugs in those regions are also far less likely to be uncovered by code inspection. It is no coincidence that so many of the security flaws in widely used software such as OpenSSL concern obscure and rarely-used functionality -- the bugs in well-known and widely-used functionality were fixed a long time ago. (But note that often an attacker can force rarely-used functionality to be invoked, e.g., by negotiating an obsolete version of a protocol.)

Finally, sometimes people add complexity in the form of deliberate obfuscation. This too is dangerous: While it has a slight benefit in slowing down an attacker (Kerckhoff's principle aside, disassembling a system and figuring out how it works will always take a non-zero amount of time), it has the far larger disadvantage of impeding testing and auditing. Robert H. Morris once stated that the #1 rule of cryptanalysis is "look for plaintext" -- because the easiest mistake to make when encrypting something is to accidentally not encrypt it (or, equivalently, to encrypt data and then use the wrong buffer in the next step). Morris' rule applies just as much to testing as it does to cryptanalysis: When you're testing your code, you're likely to notice if there is plaintext where there ought to be ciphertext... but if you decide to "strengthen" a system by adding some extra obfuscation after encrypting, such an error would likely escape unnoticed. The same applies to other situations where obfuscation might be applied: In general, if you make it harder for an attacker to figure out what your code is doing, you also make it harder for yourself (or anyone who is auditing your code) to notice if your code isn't doing what it is supposed to be doing.

Now, a certain amount of complexity is unavoidable: Few people will use a program which has absolutely no functionality, no matter how securely it does nothing. However, being aware of these pitfalls, some steps can be taken to reduce the risk of security flaws:

• Keep the amount of code which is heavily exposed to attackers to a minimum. For example, if you're going to have a service which works with cryptographically signed requests, sign the request as it appears "on the wire" rather than constructing a signature from a parsed representation -- this way, inauthentic requests can be discarded before they reach your request-parsing code. In tarsnap, I at one point use a cryptologically "useless" MAC in order to prevent attackers from passing evil data to zlib's decompression code.
• Design systems which do not use complex data structures on external interfaces. Two data structures which are notoriously complex -- and consequently difficult to parse securely -- are ASN.1 and XML; OpenSSL's ASN.1 code has had no less than 8 security vulnerabilities in as many years, and libxml has a similarly poor track record -- not due to any incompetence on the part of the OpenSSL and libxml authors, but simply because both ASN.1 and XML require very complex parsing code. In tarsnap, when I found that I had to parse XML in order to use Amazon SimpleDB, I ended up writing a parser for a "simplified XML"; which, while not able to parse all of the complexities of XML, is adequate to parse the XML returned by SimpleDB.
• If several people need to work together on the same project, divide the project into independent parts and carefully define -- and document -- the interfaces between them. Put another way: Conway's law -- that the design of software systems tends to mirror the communication structures of the organizations producing them -- is not merely descriptive; it's also good advice. In tarsnap, I build heavily on the excellent libarchive archive handling library; but aside from some very small (and unavoidable) modifications to the library, the code I've written interfaces with the libarchive code only though very clear and well-documented APIs.
• When designing protocols, provide as few run-time options as possible. SSL supports three MAC algorithms, four encryption algorithms, and six key exchange algorithms... and supports falling back to previous SSL versions. Of these, only one or two of each are commonly used, while the rest sit unused and largely untested -- yet exposed to any attacker who can find a security flaw. In tarsnap's client-server protocol transport layer, there is one MAC algorithm, one encryption algorithm, and one key exchange algorithm -- and every time a tarsnap client connects to the server, every single line of code gets tested.
• Eschew obfuscation. Even if you're not willing to make your source code public, write your code as if there's going to be thousands of people reading it and pestering you with emails if anything is unclear.

As far as I am aware, nobody has found any security vulnerabilities in tarsnap. That's not saying very much: There is lots of wildly insecure software in which nobody has found any vulnerabilities, simply because nobody has bothered looking yet. (I know some people have looked at the tarsnap client source code -- I've received emails from several tarsnap users commenting on the code structure, the quality of comments in the code, the elegance of certain components, etc. -- but I am not aware of anyone looking at the tarsnap source code specifically with an eye towards finding vulnerabilities.) Moreover, I'm not so naive as to believe that I didn't make any mistakes while writing the tarsnap code -- I took my time and worked carefully, but even given that I'd be surprised if I managed to write 15000 lines of bug-free code.

But whether security vulnerabilities have been found, and whether there are bugs, are the wrong questions to ask. The right questions to ask are these: How many bugs are there; and what is the probability that any one bug would result in a security vulnerability? By avoiding complexity when possible, and containing it when it is unavoidable, we can make the answers to those two questions "not many", and "very low" -- and thereby maximize the probability that the number of security vulnerabilities found remains zero in the future.