AWS Insiders Podcast: Episode 5 – AWS Cost Optimization 101

Rahul Subramaniam

You’re close. I mean, I did major in physics in my undergrad, so astrophysics would probably have been it, but
before… Very early on in 2007-2008, I was grappling with a whole lot of infrastructure issues, and that’s when I
discovered AWS. And the fact that infrastructure was being made available literally with a simple API call, just
blew my mind. And the more I dived into it, the more I used it, and the more I interacted with AWS. I got into a
position where I was breaking almost every AWS service as they were releasing it, which had them call me pretty
much every week about something that I broke, and just interacting with the amazingly smart folks over there
over the years just got me hooked. I was then part of every new service they created. I was involved in trying
everything out that they came up with. And they just became an integral part of how we did business. So I think
our entire business got very deeply integrated with AWS, as well.

Rahul Subramaniam

I think I’d like to phrase it as me being that advanced tester and we collaboratively made the services good to
the benefit of both parties, so it was a win-win.

Rahul Subramaniam

Cost optimization, for me, with AWS, actually started way before we had 45,000 accounts. It started when we had
one account, and I’m talking about early 2007. At that time, shockingly, AWS didn’t even have IAM that everyone
is familiar with, which allows you to manage all your users, access control, and stuff around this. Back then
you had one account. So, one username, one password, and your entire organization would have to be given that
username and password to operate on it.

Rahul Subramaniam

Okay? And can you imagine that world back in 2007?

Rahul Subramaniam

And we had over a thousand engineers that we wanted to enable on AWS. So we had this big nightmare of a scenario
where we had to give away our master username and password, which had all of our credit… Back then it was all
credit cards only. So, it had our corporate credit card punched in over there, and 1000 people could do whatever
they wanted on one account. There was no tagging, there was nothing. It was basically… The setup was primed for
absolute chaos.

Rahul Subramaniam

And so, in 2007, I wrote our first system, which acted like IAM. And it did two things. One, it allowed users to
log into a portal where they could request whatever resources they wanted. And it acted as a proxy to our one
single AWS account. But the second thing that we found very soon was, that people were just launching instances
willy-nilly and never turning them off. And our credit card ran out of limits so fast that we would have to
refund the card multiple times in a month, which is just absolutely crazy.

Rahul Subramaniam

So, one of the first cost optimization measures that we put in place was, we asked every person to put in their
eight-hour shift into the portal and we would automatically turn off or hibernate those machines that they had
launched when it was not their peak working hours.

Rahul Subramaniam

If they wanted, they could go back and turn it on if they were logging in at some odd hours. But just that cut
our AWS costs by 66% because you’re only using it for 8 hours out of the 24 hours, right?

Rahul Subramaniam

Yeah.

Rahul Subramaniam

That was the first cost optimization system I wrote back in 2007 and it’s been a constant journey ever since. If
you’re spending money on a system, there’s always room to optimize those costs.

Rahul Subramaniam

I think the thing that we learned very quickly was the foundations of why we made the big bet on AWS. And that
is that they are innovating at a pace that is just so remarkable. Everything that we thought of as a gap was
very quickly closed in a matter of a year or two. So if you were ever making a long-term bet, you wanted to bet
on AWS, or you would bet on AWS because they have a track record of constantly working on customer problems and
turning all these standard utility functions into amazing services that are just commodity services you can use
with simple API, and whatever gaps are there, you can be assured that if you bring it to their notice, it gets
solved. So with that, it becomes a no-brainer to make the long-term bet on AWS. And I think people are still
doubtful today, but having lived through this over the last 14, 15 years, I have the experience to feel very
confident about that long-term bet.

Rahul Subramaniam

Yeah. Early on when AWS had just started, I think it, as Badri said, it was possible to make a few changes that
would get big returns because there were a lot of gaps in the services, the infrastructure, the API, and stuff
like that. But over the last 15 years, AWS has built so much maturity around how they build up their services
that getting those big savings by doing one thing is just incredibly hard. Early in the days of our cost
optimization efforts, we ran into a bunch of scenarios, for example, migrating databases or moving over
applications to serverless. So, a large number of applications that we acquired were primarily on-premise
monolithic applications. And we tried to switch them all over to services like Lambda when Lambda first came
around.

Rahul Subramaniam

Lambda is not designed to deal with monolithic applications like the ones that we had. Right? And that meant
that we were embarking on this major surgery on our applications, trying to replicate the same function in a
microservices pattern. And suddenly, we had so much chaos that we just didn’t know how to manage it. And we had
several failed exercises like that where either, over some time, we completed maybe 20 or 30% of the
functionality as we moved over to microservices-based on Lambda. Or it was just a non-starter because certain
things were being done in a certain way that customers were comfortable with, and you would have to change the
entire mechanics of it as it moved to the serverless world. Right? So, we just couldn’t make all those
dimensions meet. So, we realized the hard way that those big bang approaches were very few and far between as
the services matured over some time.

Rahul Subramaniam

Yeah. First and foremost, I already have the big burden of being responsible for almost two and a half billion
lines of code that we own across all of the companies in our portfolio. And I had no interest in building a new
product or building and owning a new codebase. That was not the intention at all. Our default is to go and look
for every tool out there in the market that could potentially help us solve this problem. So, we did just that.
We tried out all the tools, but very soon we realized that there were three fundamental problems. Problem number
one was that most of these tools ended up being visualization tools we’re closing that gap on some of the
visualization problems that AWS had. By the way, all of those are gone. Right now, if you look at AWS tools,
they pretty much give you all the data you want, but most of the cost optimization tools that you find in the
market today are still glorified visualization tools that take all the data that’s in AWS and present it to you
in fancy graphs. Okay?

Rahul Subramaniam

The bottom line is it became our problem to go figure out how to realize those savings because you have to slice
and dice all that data and figure out what the insight is, and then go figure out how to realize the savings.

Rahul Subramaniam

The second problem with these tools was that none of them fixed anything. Even if they provided some insights or
some recommendations, they didn’t fix anything for us. More often than not these recommendations like, “Hey,
resize your EC2 instances,” or “Why don’t you move to a completely different serverless platform, because the
per-unit cost there is completely different.” All of those recommendations, while great sounding and the
recommended savings or potential savings was 50, 60%, just realizing that was incredibly hard because you needed
to perform major application surgery to achieve even remotely close to those kinds of savings. And we just
couldn’t get our teams to sign off on or be successful at those major surgeries that they had to perform.

Rahul Subramaniam

And the third issue with a lot of these tools was that they were just insanely complicated. Just navigating a
lot of these tools required almost like a Ph.D. in AWS services where you wouldn’t even know… They just kept
slapping on stuff over the basic visualization that they started with, and you wouldn’t even know where to go
look for insights even if they provided some insights. And a lot of these tools just got so overly complicated,
requiring admin permissions to do anything, that it just became a no-go for a large proportion. So the admin
permissions, and complex UI, became a big issue, as well. And because all of those hurdles were things that made
a lot of these tools no-gos for us, we had to invest in figuring out a simpler way to realize savings, not just
talk about potential savings.

Rahul Subramaniam

By the way, there was a year that we spent trying out all these tools and putting together a SWAT team trying to
get all the savings. We spent a few million trying to realize these big savings but saved nothing.

Rahul Subramaniam

Very simply, what we did was we looked at all the AWS recommendations that they had made around cost. We
filtered them down to the ones that we believed were completely non-disruptive and that we could execute
centrally. And that’s literally what we did. And then, of course, Change Manager came at just about the right
time, where we were able to use Change Manager as a mechanism to deploy those fixes without needing admin
credentials. So, in effect, we were fixing the problem instead of just talking about potential. And we did not
require admin credentials. We followed all of AWS’s best practices and recommendations to realize those savings.
And that was basically what closed all the gaps for us that we had with the other tools.

Rahul Subramaniam

Yeah. I mean, it’s supposed to be the simplest tool out there. With five clicks, you save 10 to 20%. It’s
supposed to be simple.

Rahul Subramaniam

Yeah. Our savings are a combination of what you see in CloudFix today and stuff that will be coming up in
CloudFix because we run all of these finders and fixes on our setup first before running it or deploying it for
other customers. And we run it for quite a while. We measure ourselves every week, “How much do we save on an
annualized basis? What amount of spend do we claw back every week?” That’s our metric, and we measure it in
dollar terms, in concrete dollar terms.

Rahul Subramaniam

And today, for our spend, which across AWS customers isn’t very large. We still manage to claw back about a
quarter of a million dollars a week, which is pretty significant, and this is, again, on an annualized basis.
But I think, on an average, for customers, you could find that whatever be your AWS spend, when we try to buy a
company, or when we are evaluating acquisitions, it’s a simple assumption that we make, which is, “Long term, we
can save 50% via this incremental mechanism, but short term, 10 to 20%…” is a given. It is something that we can
absolutely go on.

Rahul Subramaniam

I’ll add one more dimension to this. The non-disruptiveness of what you’re going after is the second dimension
in this. So I absolutely agree with Badri that the tool versus people is one dimension, but also look at the
non-disruptiveness of the changes that you’re trying to make. And that also, again, it’s like a quadrant. So
there are a bunch of non-disruptive changes that do require people’s assets. So for example, there’s a bunch of
financial engineering or just process-related stuff that can save you a bunch of costs. For example, if you’re
migrating a bunch of workloads from on-prem to AWS, you should absolutely have someone take lead and sign up for
the Migration Acceleration Program, where AWS covers a bunch of your costs while you’re migrating all your stuff
so that you’re not paying double during the migrations. That’s a great way to save a ton of money. Most people
don’t even know about the Migration Acceleration Program, but it’s the easiest thing to get signed up for. All
you need to do is tag your resources in a certain way, and you start getting credit from AWS for all of those
workloads.

Rahul Subramaniam

Another example is, if you have certain services where your consumption of those services is just insanely
expensive, you absolutely should have somebody go talk to the product team and see if you can work out a
discount or a volume discount with that particular product team because AWS does that very often. If they find a
customer using a particular service far more than anyone else, they will make concessions, and you can always go
negotiate that.

Rahul Subramaniam

The third one is, leveraging things like savings plans, CRIs, and the reserved instances, that criteria. You
need someone that you can dedicate to who can understand all of this. It’s just a bunch of financial engineering
where it’s a trade-off between commits and discounts that you get. And you can do that and get up to a 50%
discount on your spending, depending on how much of a trade-off you’re willing to make in terms of commitment
versus the discount you get.

Rahul Subramaniam

And lastly, there is the EDP. And again, I don’t recommend anyone sign the EDP. I treat it almost like a
handcuff around your hands. I mean, it’s not something I recommend, but that is an instrument of last resort, as
well, to get a certain amount of discount if you’re willing to make commitments or if you’re really, really,
really sure of what your spend over the next three to five years is going to look like.

Rahul Subramaniam

Now, those are all the things that you could do by investing in certain people and getting those cost benefits.
And they measure near zero on the disruption side of the equation.

Rahul Subramaniam

Yeah, I think, again, going back to what Badri was saying earlier, the two dimensions of tool versus people, it
is in my opinion, a 100x harder once you get into that people investment domain. So, you want to maximize and
get all your wins from the tools and the automation before you get into asking for an investment of people,
time, and resources, because expertise is scarce, and the resources are scarce. Whatever limited resources you
have, you want to invest them in building features and building your applications, because they have the domain
knowledge about your business. And that’s where you want to invest that skill and expertise.

Rahul Subramaniam

The second dimension, of course, also plays a role. Most people get scared when you start moving towards any
sort of disruptive change because they don’t understand it. They fear what the impact is going to be. And
they’re more likely to figure out ways to reject ideas of change than be participants in it. So, you have to
earn their trust. You have to earn your stripes as Badri said, but try to get as many wins out of the tools
first and exhaust all of those options before you start asking for people’s resources, building up knowledge
amongst the people, and allaying their fears about what these changes might mean for them. And that would be my
approach.

Rahul Subramaniam

Yeah, absolutely. I mean, AWS has invested a lot in creating a well-architected framework that you can use to
define your security parameters very well in the AWS framework so that you can be assured of exactly what a tool
or people are allowed or not allowed to do. Unfortunately, for most organizations, security comes more as an
afterthought. Like, they will first build the application, they will first do a bunch of stuff, and then they
will think about, “Okay, now what should I do about security?” When I say “security,” I also include permissions
and schemes that may not be secure in terms of somebody attacking your infrastructure or something like that.
Security, for me, is also where you let somebody launch a bunch of instances without control or where you let
people launch different kinds of resources that may or may not be auditable by the organization. So, best
practice is setting up your service catalog, setting up policies, and things like that.

Rahul Subramaniam

There were times in the early days when we were just starting with our cost-optimization journey, we had a bunch
of instances that were doing very little to the point that the resource utilization was near zero. And in one of
the very early cost-optimization exercises that we did, we ended up shutting off hundreds of such machines,
which ended up being fairly critical from an operations standpoint. Thankfully, we snapshotted all of these
instances before we decided to shut them off, as a safety measure. So, though there was disruption, we were able
to bring all of these instances back, but had we not put that measure in place, we would probably have suffered
greatly. I think one of the other things that is historic, or it’s been an interesting insight for me at least,
is, over a period of time, as organizations evolve and change, I find it really shocking how little the
organization knows about all of their infrastructures.

Rahul Subramaniam

You have tens of thousands of machines and resources running all over the place. You would think that there’s a
perfectly auditable manifest of exactly what machine runs what, and you know when you’re going to shut it off,
or whatever. You’d be surprised as to how many instances are running because nobody knows what’s in there and
they’re just petrified to shut it off, or EBS volumes that are running or that have been detached. They’re just
literally sitting there, massive discs of terabytes of storage capacity provisioned on them, but when you ask
someone to go delete it because nobody has touched it in a year, they’re like, “I don’t know what that has, so
I’m just petrified to shut it off. And I don’t know anyone else who knows what that instance has.” Right? So
you’d be surprised as to how prevalent that is across large organizations. Especially over a period of time,
people really don’t have a sense of the inventory of infrastructure and resources that are running, and that’s a
big cause of a lot of waste and cost, as well.

Rahul Subramaniam

And I’ll just add one more thing. I think having automation and tooling in place is really key. When you do
manual one-off stuff, you are more likely to make mistakes and regret them. So, whatever project you take on,
make sure that it is automation-driven because that’s how you ensure that what you’re doing is repeatable, not
just for one resource but across your entire setup, and you’re not going to make mistakes, because it’s all
enshrined in code. So, automation is, I think, key. No matter what kind of project you’re delivering, try not to
have manual steps in the process.

Rahul Subramaniam

I mean, in reality, if you did first principles thinking and said, “What is the root cause of all bugs?”
Somebody wrote some line of code that caused a bug.

Rahul Subramaniam

I think one of the advantages that we have is that we are literally working at the bleeding edge of cloud
computing, where we are trying to stay ahead of the curve of this firehose of AWS: Services, product updates,
and announcements. And that is literally the bleeding edge of computing.

Rahul Subramaniam

Care about the dollars saved and realized. Don’t go after the potential. And the second one would be, to rely on
tools and automation as much as you can, because the minute you start needing people to do a whole bunch of
stuff, you just get slowed like crazy.

Rahul Subramaniam

Likewise, an absolute pleasure.