Daemonic Dispatches

Thoughts on (Amazonian) Leadership

Amazon's Leadership Principles are famous, not just within Amazon but also in the tech world at large. While they're frequently mocked — including by Amazonians — they're also generally sensible rules by which to run a company. I've been an Amazon customer for over 25 years and an AWS customer for almost 20 years, and also an AWS Hero for 6 years, and while I've never worked for Amazon I feel that I've seen behind the curtain enough to offer some commentary on a few of these principles.

• Customer Obsession: Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.
  Customer Obsession is great, but I often see Amazonians taking this too simplistically: "Start with the customer" doesn't have to mean "ask customers what they want and then give them faster horses". In the early days of AWS I saw a lot of what I call "cool engineering driven" products: When EC2 launched, it wasn't really clear what people would do with it, but it was very cool and it was clear that it could be a big deal in some form, sooner or later. Some time around 2012, the culture in AWS seemed to shift from "provide cool building blocks" to "build what customers are asking for" and in my view this was a step in the wrong direction (mind you, not nearly as much as the ca. 2020 shift to "build what analysts are asking for in quarterly earnings calls").

  This tension of what customers are asking for vs what customers really need shows up in areas like resilience. Amazon's "Well-Architected Framework" strongly exhorts customers to avoid building production workloads in a single Availability Zone — but Amazon's cross-AZ bandwidth pricing is painful, and Amazon doesn't provide useful tools for building durable multi-AZ applications. Most customers are not going to implement Paxos, and very few customers — certainly not executives who are removed from actual development processes — are going to ask Amazon for Paxos-as-a-service; but if Amazonians sat down and asked themselves "what do customers need in order to design their applications well" they could probably come up with several services which Amazon already has internally. AWS should return to its roots and release important building blocks — the things customers will need, not necessarily what they're asking for.

• Ownership: Leaders are owners. They think long term and don't sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say "that's not my job."
  This principle is both too narrow, and not being fulfilled, in my view. It's not enough to simply act on behalf of the entire company: It's important to act on behalf of the entire technological ecosystem. Some Amazonians are great at this — I recently commited patches to FreeBSD's bhyve because an Amazonian was putting together a standard for interrupt handling in large VMs, and even though Amazon doesn't make any use of bhyve (at least, I don't think it does!) he understood the importance of getting standards widely accepted across the entire virtualization space rather than narrowly in the code Amazon relied upon. There's a saying in computer security, that anything which makes one of us less secure makes all of us less secure: Attackers will leverage an exploit against one system to allow them to attack another system. While the same does not directly apply in other fields, working with others to produce the best results for everyone will be much better in the long-term than focusing solely on what Amazon needs right now.

  But in general Amazon doesn't even live up to its stated (narrow) promise of having leaders acting on behalf of the entire company — it's simply too siloed. Amazon is famously secretive, and this applies internally as well as externally: When AWS launches two similar services, it's often because two teams didn't know what each other was working on. How can leaders act across the entire company if nobody knows what's happening outside of their team? They can't; and if Amazon wants to allow its best people to be true Owners, Amazon needs to start breaking down walls.

• Bias for Action: Speed matters in business. Many decisions and actions are reversible and do not need extensive study. We value calculated risk taking.
  Amazonians talk about "one-way doors" and "two-way doors", and it is quite true that many decisions are can be reversed... but that doesn't always mean that there is no cost associated with reversing a decision. There is a clear and widely recognized tension between "Bias for Action" and another principle, "Insist on the Highest Standards"; but there is also a tension between this and earning and keeping customer trust. When AWS ships a service which is half-baked, it diminishes customer trust in AWS as a whole; even if the problems in that service ultimately get corrected (either by fixing them or in some cases by simply getting rid of a service which should never have existed in the first place) the memory of a failed launch will live on in customers' minds for years to come.

  During my seven-year tenure as FreeBSD Security Officer, people knew me as the guy sending out security advisories; but the most important thing I did was not to ship Security Advisories — that is, it was to stop the train and say "no, we are not going to send this out yet". I knew that for all the importance of getting patches into people's hands in a timely manner, it was even more important to establish trust: If I gave people a broken patch, even once, they would be much slower to install security updates in the future. My team became familiar with the phrase "convince me that this is correct", and I'd like to see more of that at senior levels of Amazon: Principal and Distinguished Engineers need to step in with a bias for inaction, and use the respect they have earned to stop projects which do not meet the highest standards before they undermine trust. Amazon's hiring process famously includes "bar raisers" who can veto hiring decisions; they should also have service bar raisers who can veto launches.

Werner Vogels famously said in his 2024 re:Invent keynote, "Listen to the AWS Heroes". I think he was talking about technical advice, and perhaps speaking mainly to AWS customers; but I like to think that Amazon might also benefit listening to some of what I've said here. We criticize because we care.