AWS Insiders Podcast: Episode 6 – AWS Cost Optimization 201

Dionn Schaffner

Hey listeners. Welcome to the podcast. We are here today on AWS Insiders with Rahul Subramaniam, and then we’re
going to have a special guest, David Hessing. David, how are you doing today?

David Hessing

I’m doing really well. Good to be here.

Dionn Schaffner

Rahul, how’s it going today?

Dionn Schaffner

Awesome. Actually, there was something that was on my mind. In the U.S, we finished celebrating Mother’s Day,
and I’m a mother of two. But also, I’m in the interesting spot where my mother has passed away. And so, I’m in
this really interesting moment. With all the marketing that’s coming out, the personalization, and the ability
for companies to really churn, collect, analyze, and create insights from data, I got this really interesting
email from a business, based on your past history, the purchases you made. “But if you don’t want to receive
information about Mother’s Day because of where you are in your stage of life, let us know.” And I thought that
was a really interesting use of data, insights, and personalization… The ability for our customers and
businesses to collect all this data, to really target and personalize the customer experience. And I know there
are a lot of systems and services at AWS that support those kinds of things. So I don’t know… What new things
are you seeing that are coming out, the AWS innovations that can help us move in those interesting directions?

Dionn Schaffner

Yeah. And with all those services, right… So now, we’re talking about adding lots of elements to the IT
infrastructure, and we know that that can cause bloat, right? People are launching instances right and left.
They run for a minute, then they’re gone. And so, then that gives us this opportunity for cost optimization.
Some of the things we talked about with Badri were some of the smaller fixes. But I know we’re going to bring in
David, who’s going to tell us about some of the more challenging and the more complex opportunities for cost
optimization. David, take a quick moment and tell our listeners a little bit about yourself and what you’re up
to these days.

David Hessing

Sure. So I am an EVP of cost optimization here at ESW Capital. And for about the last year, I have been focusing
on how we can reduce our costs, primarily AWS costs, across our software portfolio in order to run that software
more cheaply and save money.

Dionn Schaffner

And tell us a little bit about the spectrum of opportunities for cost optimization. We know there are some
elements that a product like CloudFix can go in and fix seamlessly without having to break open the vault, in
terms of security and making changes. And then there are some more difficult opportunities. Can you just take us
through that spectrum and what that looks like?

David Hessing

Sure. So our model, when we go and find savings, starts from CloudFix. CloudFix is the product that we sell but
also use internally, to go and get between five and 10% savings by doing safe fixes that, as you said, are
secure, don’t cause downtime, and are really a no-brainer. Right? They’re simple things. You covered them last
week. AWS recommended things… Moving to the latest generation of something. Things that you want to just do to
save money. But beyond that, right, that only gets you, like you said, to five or 10%.

David Hessing

So our next level is something that we internally call… These don’t have very exciting names… Just
resource-level optimization specs. So we’ll go and write this spec and look at a set of AWS accounts, up to,
usually, about $10 million of spend, and just find where there are opportunities for savings by either
deleting… That’s obviously the easiest and simplest type of savings. Or by tuning resources at the resource
level. And we can get into some details and some examples about how that looks. Those specs actually typically
get us up into the 40 to 50% range of savings from the initial five to 10% from CloudFix.

David Hessing

So the final tier is called code level cost reduction specs, where we’ll actually dive into the guts of a
product and find the opportunities for savings within a product. And that is sort of the hardest type of savings
to get. It obviously, by the name, will require coding and changes to the product. But when we do go and do that
and make the investment in doing that, we’ll get the savings up all the way into the 80 to 90% savings range off
of the initial cost of running that product.

Dionn Schaffner

And we’re hearing from customers, especially when they’re innovating at such speeds, there’s a reluctance to
take a pause and take a moment to do cost optimization, right? Because there are fears about security, there are
fears about change. How do you approach selling the opportunity for cost optimization within an organization?
Because we’re talking about mitigating risks. As you said, there’s downtime with some of these bigger problems.
How do you sell that within the organization? Is it a straight ROI dollars? Like, “Look, here’s what it’s going
to save you.” Or how does that work?

David Hessing

So there are two levels, and you said the first one. The first one really is the ROI dollars, right? So when we
do these specs, that’s right in the spec, it says, “This is how much money we’re going to save, and this is the
effort it’s going to take to save it.” And that’s obviously the easiest sell, right? If the effort is less than
the savings… And we typically use a one-year ROI horizon. If the savings exceed the effort, then you’re going
to do it, because you’re going to recoup that investment within one year. And then, from that point on, that
reduction in your bill is perpetual and you basically just have that savings for as long as you’re running that
software.

David Hessing

The second level is a little harder, right? The second level is more of a total cost of ownership, the total
cost of running the software. So we run a lot of legacy software here, at ESW Capital, because of our business
model of requiring older companies. Not only, but that is a key piece of it. And when you’re running older
software that’s running on servers or running on-prem, or in a data center, or whatever, you have a lot of costs
that are associated just with running that sofware]. Obviously, the main one is headcount. People administering
those servers, making sure they’re up to date with the latest security fixes and patches. And then you have,
also, costs, because that might tend to be brittle and you might have outages. So your software goes down, and
your customers are not very happy, and there are obviously costs associated with that.

David Hessing

So the second model, which we’re working on actually getting to be a very firm model, but even before you have a
firm model that spits out numbers, the second approach to getting these sold internally is to point to the total
ownership cost and how that is going to go down.

Dionn Schaffner

Rahul, you had mentioned, in another conversation, a recent acquisition, and you put your hands to your temple
and you were scratching your face, talking about, “Oh, this code is written in lisp.” Which, by the way, is one
of the languages of my youth. Thank you very much. And you were discussing, sort of, the challenges of going
back in and reworking some of that information, some of that code, in terms of optimizing it into the new
infrastructure. Tell us some good stories about that.

Dionn Schaffner

So we’re talking about a lot of change, right? And as we think about the cost optimization… And one of the
biggest fears our customers have is change, right? We’re going to change a technology stack, we’re moving, we’re
consolidating. Which has ramifications, like you said, from everything, from personnel, and resources, all the
way through the knowledge base. Tell us a little bit… Maybe, David, you could take us through a little bit
about what change management is, and how that is going to be helpful for some of these larger cost optimization
initiatives.

David Hessing

Okay. Sure. The change manager is a service embedded within the AWS systems manager. So it’s a little hard to
find. And what it does is it is a way for you to safely make control changes to your AWS environment. It has a
notion of a change template, which sets up the parameters of what is allowed to be changed within an AWS
account. And you set up, and basically install, so to speak, those templates into your database account. And
then, once that template is there, you then can create change requests against that template. What it allows you
to do is see every single change, and make sure that it’s done, like I said, in a controlled fashion, and a
recorded fashion. And it also is a very, very important way of managing security risk. So CloudFix uses this
mechanism to make the changes that it recommends and finds to get those cost savings. It does that through
Change Manager, in order that there’s no way, there’s no risk, there’s no worry, concern. There’s no technical
way for CloudFix to actually do anything directly within a customer’s environment.

Dionn Schaffner

And you talk about needing this level of trust in order to embrace this technology and this process. When we
think about that, coming from the third-party vendor who we’re going to hire, or the tool, we want to know that
they’ve fought through, or they’ve been through the fires of change hell and trying to get to it. Tell us why we
should trust that they’ve encountered all of those elements, and some of the pitfalls they’ve discovered in
their process of creating this service.

David Hessing

Yeah. So when I started this process, it was really sort of a challenge that was thrown at me, was, “Hey, we, at
ESW Capital, have tens of thousands of accounts, tens of millions of dollars of AWS spend. And we know there’s a
ton of waste in there. Well, go find it.” I got a team to do that, to help me, but it was very open-ended to
start out with, which meant that in the beginning, for a few weeks, I was kind of floundering around, trying to
figure out where to find the savings. So after a couple of weeks of trying to figure out everything going on
across all of our accounts, and all that, we started to see certain patterns.

David Hessing

One, a big pattern we saw is when you are buying the wrong type of thing in AWS… And I’ll explain what I mean
by that. Because AWS has so many services, everything becomes quite granular, right? If you need storage, you
pay for S3 or you pay for EBS. If you need compute, you pay for… Well, there’s a whole range of options, but
the most common option is, that you pay for instances in EC2. What happens sometimes with some of our older
software is you end up paying for computing when, really, what you needed was storage. And that’s a huge savings
opportunity because computing is typically a few orders of magnitude more expensive than storage. So that’s one
big area where you found that. Because of the way our software was running on certain servers, we basically were
incorrectly paying for the wrong type of resource.

David Hessing

And one thing that AWS lets you do is split that out, right? So if you can actually break out your storage and
put it in the right medium… It depends on your software, but typically, that’s going to be S3 or EFS. Then,
you can dramatically lower your costs, both on that storage directly, but more importantly, the gains are from
when you can then reduce your compute. So, that was one big pattern that emerged pretty quickly.

David Hessing

Another one that came out is database consolidation, right? So when we buy all these products, they all have
their own database. Some of them have many, many databases. We made an acquisition where the product had over
2000 databases. And one key insight is that, if you can take all those databases, and they’re all running on
some sort of container where it might be within RDS, it might be self-hosted, but you can take all of those and
consolidate them onto a huge server, you actually save money. Because when each one is running on its own in its
own dedicated compute and storage, typically, you need to leave something around 70% headroom to account for
periodic spikes on each of those individual databases.

David Hessing

But when you consolidate them onto a huge server… Big server, many cores, tons of memory… You only really
need to leave about 20% headroom. Because each individual spike that comes on each of those little databases,
first of all, on that huge server doesn’t need that 70%. It only needs a huge server relative that 20% is
enough. And also, those spikes typically don’t happen at the same time. This is something… The insight comes
from the Erlang distribution. And because those spikes don’t happen at the same time, you need less space, and
you can run all of those databases for dramatically less. We’ve seen savings of around 65 to 70% from doing that
sort of consolidation.

David Hessing

Well, I will just add on to that, and say, this is this DB consolidation is an interesting example. It’s right
on the border between the resource level of cost savings and a product level of change, right? Because
typically, a DB is going to be pretty well-defined from your product, but there is going to be code you’re going
to need to change. So it’s sort of the lightest level of where you’re going to have to go into your product,
probably change your DB configuration, depending on your product.. Hopefully, it won’t be too hard if your
product’s written if your code’s well-organized. But it’s kind of right at the level… Of that level of
division between those two cost areas. One last area, just, I think, worth mentioning, is, by the example of the
harder type of savings, is when you go all-in on an AWS service, even more right?

David Hessing

So one area that we have seen is where older products will have their own custom analytics reporting areas, will
have an area where they crunch the numbers. They may have written it entirely themselves, or maybe they are
leveraging an older type of analytic technology. And with some AWS services, like Glue, and Athena, backed by
S3, re-doing those analytics and that reporting on those technologies leads to huge, huge cost savings. Order of
magnitude cost savings.

David Hessing

I mean, running Athena and S3 is pennies, typically. For storing the data in S3, and then you’re charged per
query in Athena, and you can even get the UI with QuickSight. So that’s a very common stack we go to. But this
kind of savings requires a lot of coding, right? So we did do this on one product, and we had a team of coders
working on it for about a quarter, and they went through and just rewrote every single report that was
previously running on Hadoop, with a custom UI, and converted it over. And we saved a whole bunch of money
there.

Dionn Schaffner

And so, some of these larger cost optimization projects, like you just mentioned, like… We’re consolidating
databases, we’re rewriting how our product is accessing this information. What types of organizations, or maybe
it’s the size of the organizations… Does it make sense to start looking into these types of initiatives?
What’s a profile of an organization that should put this type of cost optimization at the top of their strategic
list.

David Hessing

Everybody should be exercising good cost hygiene, and be looking to save money. I recognize that not everybody
might want to make the investments to do the harder types of cost savings. Certainly, the easy stuff is
definitely worth doing. And the resource-type of optimizations are worth doing. In stuff that’s not used at all,
or barely used. And these are just easy. You should definitely go after these, and find these. I mean, whether
it’s your developers that [inaudible 00:25:48] up some test machines, and then forgot about them or data that’s
just accruing because of the way your SAS products was written, and you never bothered to worry about deleting
data. I mean, there’s just tons of easy stuff to go and get.

David Hessing

Actually, one thing we’re doing internally, which is on the roadmap for CloudFix’s little ways out, is cost
anomaly detection. So we’re building on an AWS service that is backed by machine learning. And AWS has a very
nice service that… You can turn on your [inaudible 00:26:19] accounts, and it will alert you if it detects a
cost anomaly. The problem is, you got to do something. Right? The problem is, you got to check it. So we built
on top of that service. And now, we are running this internally for a while, now. Where, if AWS fires off an
anomaly, if it’s above a certain threshold, we’re using something along with the order of $50 a day, but it’s
completely configurable, we will create a ticket, and somebody has to go investigate that and figure out what’s
going on.

David Hessing

I mean, it’s hard to count these savings because they’re preventative, right? They never actually were on your
bill. So it’s a few assumptions about how long that cost anomaly would have run and some other types of
decisions about how you want to count this money. But we have saved, already, millions just by doing this cost
anomaly detection. So back to your point, some of this stuff is so easy to go and get, and some of its just good
practice, that really, everybody should be doing it.

Dionn Schaffner

So how are you organizing your team? So, say I’m listening in on this conversation, and I’m like, “You know
what? We are not doing this. We need a team like David’s team, that’s going to focus on this, find those
anomalies.” And to your point, actually do something about it. What would be a best practice for resources, the
things that you said, attack the preventative opportunities? What does that team look like?

David Hessing

That’s a great question. So I can tell you what it looks like for us. It’s not a big team. It’s a team of about
five. This is all after CloudFix. This is assuming CloudFix already ran, and got you your five and 10%. So after
that, what we do is, we will go and look, account-wise, and look for certain types of resources across different
accounts. And we’ve got that down to a bit of a science. The most common types of resources that we see cost
spikes for, or just waste. Not just only spikes, just waste to go and get, and clean up.

David Hessing

We have sort of a standard work unit for going and checking, say, Elasticsearch/OpenSearch, right? We see where
people will go and configure a cluster. When it’s for dev, they’ll have a master node in there and you just
don’t need it, and you can just get rid of it. Bring the cluster down to one, or especially for dev or QA, the
same… It works for you just fine, for what you’re using it for. And you just cut costs by 30% or more. Right?
So, that’s just one example.

David Hessing

So we have a set of these work units that are oriented around different common resources, that have either waste
or poor usage patterns, and the team will go and run those. And they’ll go and look for those sorts of
resources, just on an ongoing basis. And then, the other thing we do is, we go and we find things at the product
level that become the code level cost reduction opportunities. But that typically requires almost an anatomy of
a product, right? So we’ll go look at a product, and we’ll say, “Hey, this product is spending $3 million
annually,” right? So we will go and do the anatomy of that product and say… Just to understand where that
spend is, how it’s broken down, and where those opportunities are for cutting that within the product.

Dionn Schaffner

What are your success metrics for this team? How are they evaluated?

David Hessing

In cost, it’s easy. Right? Because cost, it’s dollars. Right? It’s the easiest metric… We touched on the
one-year ROI that we shoot for. That’s not just on the team’s activities itself. That includes the engineering
time. So success right now is, basically, that we identify cost savings where… The cost of identifying the
cost savings, and the cost of getting and executing those cost savings is less than the cost savings. That’s our
success metric.

Dionn Schaffner

So ideally, this group is going to work themselves out of a job, if they do it right?

David Hessing

Absolutely. We are starting to see a plateau, right? The low-hanging fruit, or the easy stuff that we went and
got over the last year. But I will say, this has gone a lot longer than I expected. When I started this over a
year ago, I thought, “Hey, maybe we’ll go cut some waste for a quarter,” right? Or maybe two quarters. And we’re
still going strong. We have a backlog of work that extends into Q3. ESW has a very large portfolio of products,
but a lot of those products are maybe a bit smaller. They’re each individual product… Annual spend is less. So
in that long tail of products, it may not be cost-effective to get the cost savings, right? You might not hit
that on your ROI.

David Hessing

So we’re doing two things. We’re looking for ways to be more efficient ourselves, right? We’re looking for ways
to identify cost savings that cost less, streamline our own work, and make it more efficient. The other thing is
that we’re turning this into a product because we think that this is something that is very valuable for
external companies as well. So if we succeed in that, then we won’t be out of a job.

David Hessing

Right? And plus, as ESW, we’re always making new acquisitions as well.

Dionn Schaffner

And that seems in line, also, with the process of going through the internal pains and gains of figuring it out,
and then productizing that and turning it into something that our customers can use, because obviously, ESW is
not the only one running into these same problems. Right?

Dionn Schaffner

Awesome. All right. To wrap up, David, tell us the top two tips our listeners should know about cost
optimization.

David Hessing

The easiest cost optimization is to find stuff you’re not using. We just finished a spec… Again, this is an
internal spec. And I think, something like 60% of the savings it identified was just stuff that people left
running, because the advantage of the Cloud is it’s… Click a button and you turn something on. There’s no
friction, which is great. Number one. Tip number two, I think, is what Rahul said. Which is just, always be
looking for what AWS is rolling out, right? Because there are also easy savings to get there. Those two tips,
that’s just easy stuff. Get you a ton of savings to start with.

Dionn Schaffner

And Rahul, what do you think the top two ways to reduce the fear of the risk in an organization would be,
forgoing and doing some of these more challenging cost optimization initiatives?

Dionn Schaffner

The build-versus-buy challenge.

Dionn Schaffner

Awesome. Well, guys, this has been great. Thank you, David, for coming in and giving us a little more insight
into some of the other areas of cost optimization. We’ve talked high level, we’ve talked down to database
consolidation level. I think we hit all the angles, so I appreciate you coming out, David. Thanks for being
here.

David Hessing

Absolutely. It’s been a pleasure.

Dionn Schaffner

Thanks, everyone, for listening today. If you enjoyed our podcast, please be sure to rate, review and subscribe.
See you next time on AWS Insiders.