Ask Us Anything: Episode 5 – C7g, Step Functions, and more

Rahul

Good, I think a lot of excitement around the new instance type. That makes it a little holiday that we were planning over the weekend. I was juggling enjoying the beach, for us out here in India, on the east coast, and trying to do some quick experiments to see the GA part of the C7g and I think we’re going to be talking a lot more about that. But overall, I enjoyed some of the air conditioning, because it is insanely hot.

Rahul

That sounds interesting. We are going to be in Seattle, as you know, next week with family. And I really hope the weather can change because keeping kids indoors for long hours is going to be incredibly hard. So looking forward to Seattle weather changing and being here next week.

Rahul

Fingers crossed. Awesome

Rahul

Absolutely. Let’s dive right in.

Rahul

Yeah. So these are basically the custom-built in-house ARM-based processors that Amazon has designed. And they believe that the graviton series of processes is going to take up very soon over 25% of all of their data centers. So Amazon is making a big bet on this new architecture and these new types of processors, they are truly providing or proving to be incredibly powerful and energy-efficient at the same time, and are really the Next Generation of compute that you’re likely to see. Between all the big players, right now, there’s such a push towards getting these ARM-based processors out, because just as sure improvements you’re able to see in these compute processors. And invariably, everyone is getting onto system on chip kind of design, which, ARM, you know, allows you to build. It’s remarkable what you can do with these chips. And I really see these as a future for computing over the next few years.

Rahul

Yeah. So if you really look at it from a server compute standpoint, and this story has changed so dramatically, you never associated ARM-based processors for you know, seller compute. I mean, till very recently, the only ARM processors you think of was in your cell phone, or mobile phone, or when you bought a Raspberry Pi, I have an entire cluster of Raspberry Pi’s, you know, the averages for experiments and my home automation. But that was the extent to which you would think about using ARM just a few years ago, but the game has to change with these new architectures. Now, when you compare these ARM design processors with x86, here are basically what the trade-offs are for a server use case, right? Because you have a data center, every data center is looking to pack as much compute as possible onto every unit of a server, right?

Some of the racks I mentioned are measured in use, you have one you to use, and so on, you want to pack in as much density as you possibly can into a single rack. And that’s basically the math that it boils down to. When you look at the numerous N1 or v1 architecture, it’s truly a game-changer, because A, it allows for three sockets as against two. Now what I mean by socket is the number of places or how many CPUs you can actually put onto a single motherboard. And the fact that this architecture allows for three sockets, that means you can put in three CPUs on the motherboard.

Rahul

Right. So you have these on one single board. And the fact that each one of those CPUs has 64 cores, that’s a pretty large number. But interestingly, the architecture itself allows for up to 96 cores, if I’m not mistaken. So you could go all the way up to 96 cores. And you could push the clock speed for each one of these cores, up to 3.1 gigahertz, right? From everything that I’ve read so far, it looks like Amazon is not really pushing the envelope on these processors from all of our bench front data appear to be still operating at about 2.5 gigahertz. So they really have so much more room for performance improvements.

I can’t wait to see what they’ll do with the next generation of gravitons. Because they can go from 64 cores to 92 or 96 cores, I think, on a CPU basis, you already have three sockets per motherboard. And on top of that, they can go all the way from 2.5 gigahertz per core, all the way up to 3.1. So this is so much headroom for performance improvement. They literally just started cranking the wheels on this. In contrast, when you look at the likes of AMD, and Intel, the approach that they’re taking for higher performance is to increase the clock speeds, okay? Most of these processors are doing clock speeds of 5 gigahertz and more. That’s what they’re shooting for.

In fact, just last week, I was really excited because AMD had a whole bunch of new announcements, where they were pushing their clock speeds per core up to 5.5 gigahertz on a sustained basis. And that is pretty impressive in itself. Again, 64 cores per CPU with the next-gen processors. But the other trade-off that servers need to make is the TDP. TDP is basically how much heat is produced, or what is the thermal designed, you know, provision…that’s basically how much heat it gets produced by your processors. And when you look at the AMD processors, they are pushing nearly 300 Watts, which basically means you need very, very custom-made cooling mechanisms for these CPUs that AMD makes.

My guess is that, and again, these numbers haven’t been published for graviton, but just looking at the general trend of ARM-based processors, I’m guessing that the graviton processors produce less than half the TDP, they are less than half the TDP of these AMD processors that are currently ruling the roost. I wanna mention Dell because AMD is kind of, you know, really kicking Intel’s butt at this point of time. But that’s basically the difference between those two approaches. When it comes to servers, you want the most efficient cooling for all the CPUs, so that you can push them as hard as possible. And you want to pack in as many cores as you possibly can on the single rack, or the single motherboard. And ARM does so much better than any of the x86 architectures, that it just makes sense to pack in as many of those as possible in your data center.

Rahul

Great. And I think Apple kind of blew everyone away with the performance improvements that they got out of these custom chips that they built.

And like I said, the TDPs on those are so low that I’ve actually never felt my Mac Mini that runs on M1 has literally been running tens of browser windows 24/7, I have never seen it get even warm. And my old MacBook Pro. And in fact, even a Mac Pro if you remember the dustbin design or the trashcan design.

That one which basically was running 16 core, sorry, that was running a 32 core processor. That’s the one with 64 gigs of RAM. That one gets so insanely hot in comparison. And this tiny little M1 does so much better on performance and thermal efficiency. It’s just absolutely mind-boggling. In fact, most of these cheese graters, Mac Pros, even those have been obsoleted. Like I wonder even though Apple continues to sell them. I don’t know why someone would go buy one of these. The new MacBook Pro 16 inches or the other trollers[SP] are actually, they would completely destroy these cheese grater Mac Pros.

That’s the fundamental difference between the x86 architecture and the new ARM-based ones that these guys are building.

Rahul

True. But if you actually open it up, and I don’t know if you’ve had a chance to ever open up one of these Macs, the SOC that has everything built-in, is so tiny compared to the old motherboards with the CPUs and all the other peripherals added to it. It is remarkable how efficient they made it all. Yes, you had the constraint that you can upgrade it and do all the other stuff. But if you were purely looking at performance and efficiency, it is truly remarkable what they’ve done with these new ARM architectures.

Rahul

So, basically, there are four parameters that determine what kind of instance size you pick. So you have your CPU, you have your memory, you have your network iO, and your disk type, right? Those are your four parameters. And you have to determine your instance type based on the workload that you’re running. Now, let’s say you had some kind of compute-intensive workload where you just needed a ton of CPU, you would pick the C types that you have on AWS, the C types are your compute-intensive machines, where the ratio of compute to memory is significantly higher, then you have other workloads where, let’s say you are trying to run a database, right? When you’re trying to run a database, what you want is good iOS that you can load up all your database, you know, tables very quickly into memory.

So you want to read very quickly. And you want as much memory as possible to keep everything in memory. Because once you can do…if you can do queries in memory, that is the most efficient way to run a database. So it’s not really that compute-intensive, but it all hinges on being able to load up as much of your tables in memory as possible. And that’s where some of the R types and the N types of instances come in. The R types optimize on both iO and memory, rather than on CPU or network. You have other instance types, which optimize on your EBS volume attached to it.

So there are instance types like the Is, which you would use for file servers and things like just do tons and tons of writes, you know, onto disk. Like if you had something like an Elastic Search Cluster, they’re just pulling in a whole lot of data from different streams and just dumping it all into disk indexing it right away, you would possibly use something like the Is, as your instance types for those, those are EBS optimized. So it all depends on what you need out of your workload. There are some scenarios where network iO makes a very big difference. And in fact, the larger your instance, the more network iO you get up to 12 or 14, I think, yeah up to 12 Gbps connectivity on the network. But if you have smaller instance types, you probably don’t get that much.

Then there is another category, which is the burstable instances which are the Ts. So you’ll see the T3 instance types, yeah T4 and T5. T4g is available now as a Graviton1s, but you have the T3s and T4s. These basically are instances where you know they aren’t going to use these processors on a sustained basis. But there might be times when you need to burst your CPU load for a short period of time, and that’s when you would use these T types. So you have a wide variety, you really need to understand what the parameters of your workload are going to be. And based on that you pick the instance type.

Rahul

Yeah, and I think back to AWS’s core philosophy, you need to pick the right resource for the right kind of workload, whether it be databases, pick the right database for the kind of workload you have, pick the right kind of compute, or instance type for the, you know, compute requirements or memory requirements or iO requirements, they are providing a massive catalog to users to go pick whatever they want. Now, one thing I would say if you’re starting afresh, pick the latest generation, because the latest generation is always going to be cheaper and faster. The reason why you have this big explosion of the old, you know, you have 500-odd instance types, I would cut that down by you know, about at least 80% and only focus on the new instance type because they are faster and better. The old ones exist because AWS changed their strategy for how they provided cost benefits back to the customer.

So, in the early days, you know, when we first started, there was only one instance type or, you know, there were one of each type, there was one standard. And what they would do is any price reductions that they got, they will just straight away pass it on to the consumer. That means every year, even if you did nothing, you just stay on the same instance type, you will suddenly see your bill go up 10% because AWS will literally pass on those cost benefits to customers. But two or three years in, they kind of stopped doing that, they started creating new instance families and saying, “you want to stay on the old standards, they are just gonna get more expensive. You want the next generation, the faster, better ones, you have to make the effort of moving on to these new instance types.”

And that did, you know, require work on part of the customers to upgrade and migrate because in some cases, they were disruptive in certain senses. But largely not too hard. In the early days, you also had different types of hypervisors you had to care about, a lot of that problem has kind of disappeared. But back in the day, you had to decide, you know whether you had paravirtual instances or hypervisors. And then you had to figure out whether they were compatible and do a lot of restrictions on each. So you have to build your EMIs to be compatible with each. They were very complicated to kind of move. But these days, they’re all pretty much on the standard platform, I would say pick the latest family and latest generation of instances. So right now you’re going with the T4, C7, M6s, pick the latest generation and move to those always. They’re always cheaper and better.

Rahul

Yeah, but actually, you’d be surprised how many customers are still running the very, very old M1 instances, [crosstalk 00:33:42.890] there are still customers running it and you know, just continues to support them.

Rahul

Yeah. I mean, it’s both the good and the bad. Again, back to AWS Made Easy. Our recommendation is to get on the latest and greatest because it’s just cheaper and it is faster. Like there is…it’s a no-brainer trade-off on that front.

Rahul

I will just add a caveat to that. So if you are typically running an application that is based on a virtual machine, so languages that run on Java, Ruby on Rails kind of architectures, or Python, or, you know, Node.js, one that run on some kind of a virtual machine or interpretation, you should generally not have a problem switching from x86 to ARM. The only places where we could run into some complications are scenarios where you have some libraries that have a native core, you know, there are some Java libraries that are written in JNI. Basically, C++ implementations under a Java wrapper, scenarios like that will require you to recompile the native core part of it in, you know, so that it’s compatible with ARM processors. For the most part, you usually don’t encounter those kinds of scenarios anymore. So you should be able to relatively easily take your x86-based application and ported over to ARM pretty easily.

Rahul

Absolutely. I mean, the price-performance improvement is, you know, to the order of 40% or greater, depending on somewhere between 20% and 40%. But the other key thing, I think, for our audience to understand is that when we deal with AWS, we’re dealing in terms of vCPUs, right? And that’s the fundamental unit that you purchase. And one would think that these are apples-to-apples comparisons, but they really are not. The way vCPU is computed on ARM versus the way it’s computed on the x86 machines is fundamentally different. When you look at x86 architecture, each core is a hypothetic core, that means you can run two threads on each core.

So when you buy a vCPU, or when you resell a vCPU, on x86 architecture, what you’re effectively getting is half a core, or you’re getting one thread, if you get a two vCPU instance, you effectively get one core at that point. On ARM, given that ARM is single-threaded on a per-core basis, what you get is an entire core. So even if you had the state of the art, you know, processor, an x86 that’s running at near 5 gigahertz in effect, what you’re getting is only a single thread or two threads that are running on that core on x86. So it is whatever you get on that one thread. You also have the overheads of managing the threads in the pipelines there. Though Intel and AMD have gotten pretty efficient with managing those pipelines. On the ARM side of things, you will literally get the dedicated core that processes your single thread and that is your vCPU. So there is a fundamental difference in the units even though we’re talking about the CPUs, it is not an apples-to-apples comparison when you actually look at what you’re getting out of the processor.

Rahul

Yeah. And from our experience, working with the Graviton team at AWS, over the last few months, it’s remarkable, every little optimization that they’ve done and they’ve worked on is based on a real-world workload like you said, it is not hypothetical. They’re not randomly trying to optimize everything there. They’re, they’re literally looking at all the standard instructions that are basically going into the processor and they’re optimizing those pipelines. Every time they encounter a new scenario where they find something being slow they look at what kind of instructions are going in there optimizing those and making them available to customers. So they are, like you said, working on real-world workloads, and not hypothetical scenarios or made-up benchmarks that you invariably see in a lot of scenarios.

Rahul

No, get around as quickly as possible. This is the bet you want to make. You know, like, I said before, the runway for performance improvements on ARM are significantly greater than on the x86 architecture. At this point, AMD and to some extent, Intel, they are literally fighting physics, to the point where they are trying to figure out how not to melt the CPU, as they are kind of increasing the number of cores on every little CPU. And the TDP is kind of reaching insane level at those 5.5 Gigahertz clock speeds. You know, they’re hard to make trade-offs like decide what are going to be efficiency cores and decide what are going to be performance cores. Because if they made everything a performance core, they would literally melt the CPU. So they’re fighting physics at that particular place.

And then you look at the trade-offs that, you know, the likes of Amazon have to make with Graviton on the ARM side, there’s just so much headway. You can go up in 96 cores that the architecture supports, you can go from 2.5 Gigahertz and core up to 3.1 and get a massive boost just out of that, you already have three sockets per motherboard. So they have so much more room for performance improvements over the next few years compared to x86 that, that really is the bet you want to make. So please go ahead and try it out. See if fabrications work there, it’s more performance and saves you tons of money.

Rahul

By the way, these guys have an incredibly hard job interpreting all the technical jargon we throw at them. And I can’t even imagine, you know what they go through as they are doing this interpretation. So, thank you, Jacob and Lindsey.

Rahul

Those are awesome pictures.

Rahul

And looks like the image when he just finished and got it to boot.

Rahul

Absolutely.

Rahul

Remember, I was talking about the use or the rack space use. This one looks like a 4-U build. Like I can’t imagine how that fan worked. You know fit in less than 4-U. So, I mean, that’s the amount of space that just cooling would use up. If you were to kind of build a data center based on these kinds of processors, this is the heat, you have to figure out a way to cool and efficiently.

Rahul

So, I think the more I think about it, I would not go for the Intel. So it’s just a difference between choosing AMD. So yeah, choosing an AMD Epyc processor versus graviton. And at the end of the day, given the recent cloud workload, all you should care about is your price performance for your particular workload. And for some unique scenarios, AMD probably would be better. But I think largely when you look at a lot of general-purpose workloads, ARM is actually kind of winning that race at this point from a price-performance standpoint. And given that AWS’s overheads are going to be far less than custom building their own chips, and running them, and offering it to customers, my bet is on Graviton. Like, even if there were a few workloads that are optimized right now for AMD, if I look at the next two or three years, I expect that pretty much every workload would be running on graviton.

Rahul

Absolutely.

Rahul

Yeah. I think we spent way longer than we had anticipated on this particular topic. But again, it’s just such an interesting announcement, an interesting area, where today, customers actually have to make the bet. You have to decide whether you’re going to be on x86 or the ARM type of processors and knowing all this stuff is incredibly valuable.

Rahul

Absolutely.

Rahul

Stephen, you want to just quickly pull up the announcement.

Rahul

Great.

Rahul

Yeah. So you should think of step functions pretty much as a workflow, right? You have states, you have a state transition system, you have events that allow you to move from one state to the other. And you have a bunch of processors for each state. So typically, these processors are lambda functions, or some kind of, you know, processor that really might just compute. But in the simplest form, lambda functions is basically what is used to tie all these things together.

Now, historically, think of it as if a million transactions went through this entire workflow. debugging an issue is invariably very, very hard, because the way you’re number one, identifying an issue because these are all aggregates, you need to aggregate up stuff around state. So when you say that, in a particular state, things are failing repeatedly, you need to be able to jump on it. That kind of visibility was fairly workable. The other thing was, you’re dealing with millions, if not billions, of events that go through these workflows and being able to trace them down at the level of events to understand what event got triggered, or did not get triggered. How did the true state transition actually happen? What are the state changes that happened?

Getting into all of those details with every single trace, was incredibly hard to do. And therefore, any workflow that was beyond maybe five or six boxes, was, you know, a few thousand instances running on a per-day basis, would be incredibly hard to debug. And I think this particular announcement has made it way simpler. Now we are to see how it performs this experiment with a few use cases where we have millions of transactions going through this, but it definitely makes it way easier to dive in and figure out where things are wrong.

The one thing that I kinda miss and right now has to be done separately is aggregate up, you know, stats across different transactions that go to this. So for example, you want to understand how many instances failed at step three of your state, right? So that when it hits a certain threshold, you know to jump on it. And that’s on the observability side of things. That’s a little harder to do right now out of the box. You can take all the data events data, put it into an s3 bucket or into a Kinesis stream run Athena queries, create a quick side dashboard and alarms and all that stuff. It gets a little harder, or you can create CloudWatch metrics to do the same thing. It is just a little harder to do, but it’s definitely a step in the right direction towards observability. And being able to get right into the nitty-gritty details of what’s happening with each execution.

Rahul

Absolutely.

Rahul

No. I think we can move on to the next announcement.

Rahul

Yeah. So any of these Data syncs is actually a really neat service. If you haven’t tried it up already. It is a great way to move your data into AWS from wherever your data is. And you can set up a whole lot of rules and schedules and stuff like that. There’s tons of monitoring available to understand what’s being Synced and what’s out of Sync. And it’s a great way to keep different data stores on the same page. This announcement I find interesting and confusing at the same time. So this announcement seems largely driven by a lot of customers who are trying to push the multi-cloud story where they want the data not just in AWS but also replicated for disaster recovery purposes, on Azure as well as GCP.

Now there are…I’m still trying to digest that use case. Because when I really think about it, the first thought that goes to my head is the egress charges nightmare, because AWSes egress charges are so atrociously high, that if I think of taking lots of chunks of data that are very important to me and replicating it across different cloud providers, I am definitely going to see a massive bill. And I don’t know how this kind of a setup really helps, you know, or disaster recovery, because you could do the same thing across three regions, and within AWS itself, or across different availability zones as well.

Rahul

That’s a lot of concession they’re thrown your way.

Rahul

This will hurt a lot coming from experience, you know, I mean a simple thing like writing into s3 without using a VPC endpoint, right? I mean, that traffic goes over the internet. So if you have your computer within a VPC and you didn’t use the VPC endpoint to write even to s3, you suddenly see a massive bill, because stuff is going over the internet into s3. So it’s considered as an egress and back into s3, right? Which is why the first recommendation we tell everybody is for any, you know, a store that has a VPC endpoint, switch over to the VPC endpoint, you will save tons of money just out of those egress charges. This is just I don’t know this, this feels like there’s gonna be a massive bill that’s just waiting to come your way. And you’d be better off just using multiple ACs or multiple, you know, regions at most to replicate your data, rather than doing it across cloud providers. Yeah, this feels like a disaster waiting to happen to be honest. So, I have very mixed feelings about it. I love Data Sync as a service but just a word of caution to anyone trying to use this for multi-cloud. Be prepared for a massive bill depending on how much data you’re trying to move over.

Rahul

Ingress is usually free across most cloud providers, for example, AWS will not charge you anything for an ingress. They’d have to…they want you to put all your data in that. Egress is basically what gets charged double of what it should have been because they cover ingress cost by overcharging on the egress.

Rahul

Yeah. I have a good reason I would love to learn. Like I haven’t been able to wrap my head around a reason, which requires you to be multi-cloud in that way, versus multi-region within a single cloud provider. But I would love to learn. It’s something that I’ve been struggling to figure out a good use case for.

Rahul

I have a slightly different take on this one. I think this is a testament to the AWS Marketplace, where if you are a software vendor, you know that all of our customers are in the AWS Marketplace. AWS Marketplace has two fundamental advantages. Number one is that it allows you to discover customers very quickly. And if you sign up with the partner program, AWS encourages or will help you find customers for your products within their customer base. So that’s one advantage. But the second advantage also is for a lot of companies that are only now starting to move into AWS. And this is going to be you know, 90% of workloads that we’re talking about here. Over the next few years, there’s still in a very, very huge chunk of workloads that are yet to move in.

A lot of these organizations are still struggling with the old procurement model, where you have to go through a separate procurement engagement to buy software, right? But the Marketplace makes it really easy where with one click, you can go deploy a new product within your AWS account, and it shows up as a single line item in your AWS billing. So your organization has already signed up for AWS, makes it really easy to take your product and deploy it within an end…you know, within a customer base that’s already on AWS, and for them, it allows them to kind of override traditional procurement, which can in some cases be insanely painful to get through. But because it is all beautifully integrated within the AWS ecosystem, it makes it really easy for, you know, random teams to literally go click a button and get it deployed and start using. So that’s the advantage.

Rahul

So, I think, in this particular case, Elasticsearch has realized that it does need the AWS Marketplace because that’s where all of their potential customers are. And they have the advantage also the fact that while they open-sourced large chunks of Elasticsearch, which AWS basically took and turn into their, you know, Elasticsearch, or open-source service. Elasticsearch as a company still has the entire ELK stack that includes Kibana and LogStash and ELK as a whole is basically what a lot of customers want to leverage. The Kibana visualization beats anything that AWS has, and customers want. So it makes sense for Elasticsearch to leverage the marketplace to sell a whole stack as a whole with their terms and their pricing. Even though it’s on the AWS Marketplace, and have it be available to AWS customers. So rather than fight AWS on just the Elasticsearch part, they are now dealing with the entire stack, and are offering it in the marketplace. And that makes life a lot easier for both parties, they will fight a lot less. And I think customers end up benefiting the most out of this.

Rahul

Exactly.

Rahul

No. I think again, I go back to the first topic, please go ahead and look at the new C7gs, they are awesome. And go take a look at our benchmark that we published on awsmadeeasy.com We’d love your comments on that. And you know, we’d love to get your questions. We take a look, we get a lot of offline questions that we try to bake into this conversation. But feel free to post live as well during these sessions and we’d love to answer them live. Yeah, and we look forward to seeing you next week.

Rahul

That’s right. So the team is actually in Seattle next week, we are going to be working very closely with the AWS teams themselves. We have a lot of discussions and meetings across different product groups. And therefore, instead of doing this on Tuesday next week, we will be doing this on Thursday.

Rahul

Thanks, everyone.