Ask Us Anything: Episode 7

Rahul

Doing very well, had a pretty amazing weekend. We went up to the mountains, Mount Baker. And for us who come from the subcontinent where snow is, you know, one of those things you rarely ever get to see or witness, I think it was awesome. The kids had an amazing time. And, yeah, it was a fun, relaxed weekend with a bunch of hikes.

Rahul

I have some nightmare memories from Snoqualmie. I have a YouTube video that’s floating around somewhere of me trying to ski for the first time in my life and falling off or dismounting ungracefully from the ski lift and I hope no one finds that one.

Rahul

That’s good to hear. Awesome, I actually looking forward to the conversation today, there are a bunch of very interesting announcements that came up. And, yeah, looking forward to the discussions from the audience. Please feel free to ask questions. All yours, Stephen.

Rahul

Yeah, so I learned one other thing, which is even though machine learning and, you know, this stuff sounds very fancy and sounds, you know, very exciting and bleeding edge, about 90% of the time that is spent by data scientists or by people who are building these machine learning models is actually spent in grunt work. And that grunt work is taking loads and loads and loads of data, you know, cleaning it all up, taking out all the outliers, trying to figure out what features might influence or not influence, run a bunch of experiments to look at the standard deviation of your data, look for biases in the data.

It is literally…I don’t know, this is absolutely apt, this particular image you pulled up. It is 90% of the grunt work. And everyone thinks, “Oh, I’m gonna build a nice machine learning model and I’m gonna, you know, change the world.” But all these machine learning models are all about the data and the data, more often than not, is incredibly messy and dirty. And just to get it in shape so that the machine learning models can do something reasonable with them, just takes an inordinate amount of effort, an inordinate amount of time in the grand scheme of things. And Data Wrangler actually really helps you with tackling a bunch of that grunt work.

Rahul

Yeah.

Rahul

Yep. And the other thing that, you know, all the machine learning teams…I mean, machine learning is now such a massive org within AWS. But one thing that you see all of these teams kind of trying to do is make sure that all the various steps that you have in the pipeline are being made super easy. If you haven’t already, take a look at, you know, a lot of the functions that the Data Wrangler tool has, there are over 300 functions that you can use out of the box that help you your transform the data in a very, very simple, easy manner, create new features or vectors that you can use. They are trying to make it really easy for people to get through this painful data processing stage and data cleansing stage.

And yes, I think what you can expect to see is across all the different SageMaker services, SageMaker is now a portfolio of maybe 25-30 different services, you will start seeing a bunch of simplification of how you use it. There are a bunch of, you know, autopilot-like mechanisms where the tool runs, it finds a bunch of patterns for you, and then you get to make simple choices. If you haven’t looked at it…I think this all started back in 2019 when SageMaker Autopilot first launched, what we did…or what was done as part of Autopilot was you could actually run a whole bunch of different models, you know, just completely automatically without you having to figure out how to do you know, all the fine-tuning of all these models and do hyperparameter tuning and all of that stuff. You could just let the model run.

And the output of it, I thought was pretty amazing because the output told you for each of the models that were candidates, what each one cost in terms of time, what each one cost in terms of money, what is the cost of running all the compute behind it to create the inferences. And third is the latency requirements. So, depending on what your trade-offs are for your problems, you can decide between the accuracy, the cost, and the latency. You can pick the model that fits your use case the best. And most people don’t realize it, they always think in terms of accuracy. But accuracy isn’t always the only parameter you care about, you do care about the latency and the cost in a bunch of cases. Like there are cases where you want your model to return back an answer within, you know, 100 milliseconds because that’s the critical decision you’re making. You don’t care that the accuracy is 99% or the confidence score has been at 10%. Eighty percent is good enough, but you just need that super-fast decision-making.

Rahul

Exactly.

Rahul

No, I just love Data Wrangler. You guys should try it, use it, give us feedback, we’d love to know what your experience with Data Wrangler has been. From my old world of not having tools like Data Wrangler, this has been a pretty big game-changer, it simplifies so many things.

Rahul

The versioning is actually a very, very good point to bring up. Because I think as you do a lot of this work, you realize that it’s all about experimentation. You run hundreds of these experiments, you don’t know what outcome you’re gonna get, and sometimes you have to roll back, trace back to the previous step, do something different. And it’s all these crazy parts you take down, a spider web of, you know, experiments, that being able to version it and go back to a particular version, being able to trace, you know, why certain changes…you know, why you’re going down a particular path, is incredibly valuable. And back to what you said, I have had infinite number of final-final-final versions of my dataset where, you know, we’ve just literally replicated and copied that dataset over and over and over again and then completely lost track of, you know, where we came from during those experiments. So, yeah, an incredibly valuable tool to have.

Rahul

Yeah, I agree.

Rahul

Yeah, back to the simplicity that the SageMaker team is bringing to the product. They’re making it easier and easier with very few clicks. This is almost like a WYSIWYG version, you know, if you remember Visual Basic from back in the day, you know, the WYSIWYG editors and stuff like that. They’re gonna bring WYSIWYG, of sort, to SageMaker Canvas. And it is a pretty interesting tool where if you just want to play around with some datasets that you already have, just get going without having to set up a bunch of, you know, super big, heavy-duty pipelines, SageMaker Canvas is pretty neat in, you know, getting you started: understanding the basic concepts, getting some basic models out, starting with ML. And as you build more complex data sets, as you need to, you know, kind of bring in data from different data sources, build a data lake, do all of that stuff, you can advance to more complicated scenarios, but SageMaker Canvas is actually a great way to get started.

Rahul

Yeah, and actually having a bunch of this predetermined or pre-available data set for you to get started is incredibly valuable because like we were talking earlier, all machine learning models are about data and you can spend so much time just grappling…and I think that’s why they call it Data Wrangler. The amount of time that you can spend just grappling with the data and giving up because it’s so damn hard is…I mean, it just makes sense to have some of these datasets be available so you can get started with, you know, some of the machine learning stuff, get started with something, and then sometimes you can even figure out what kind of data you want to have from the existing datasets. Like, you don’t have to learn and experiment with modeling from scratch.

If you know a certain kind of data set works well with certain kinds of features or vectors, then you’ll know that wherever you pull data from for your use case, you want to have a similar vector set, at least make that your starting point. It may not always work but it’s a reasonably good starting point. And then from that point on, you can run the models again, train it, figure out whether the accuracy or the confidence scores, you know, continue to remain good, if not, then you can start looking for what’s different. And I think that approach is a lot easier than trying to literally reinvent or invent your machine learning model from scratch.

Rahul

I completely agree. By the way, I’m not a big fan of CloudFormation in general because, you know, if anything that’s non-trivial that you want to build, CloudFormation just, you know, makes your life hell and I’m just being very blunt about it. It helps to have the next-gen tools like CDK in the CloudFormation under the hood for you but that’s kind of why it’s neat that it addresses both of these for everyone so you can get started without any roadblocks. It would be interesting to look at the script to see how easy it is to modify and kind of tweak it to your own needs, whether it be the instance types and sizes, or where it gets deployed or, you know, what kind of permission schemes you would need to give the cluster or the setup for your data lake and, you know, wherever your data resides. So, that would be an interesting exercise, which I’m probably going to do right after this live stream. But like we were talking earlier, it’s great that you have something to get started with. I think the barriers to get started are being reduced every day at AWS, and the fact that you have a template to get started with is awesome.

Rahul

Yeah, I wouldn’t know how to get started with CloudFormation from scratch anymore. Like, I would probably give up.

Rahul

Yeah, let’s go ahead.

Rahul

Yeah, go ahead.

Rahul

I mean, these days, the demand for ML specialists or people with some ML experience is so great. One of the ways that a lot of organizations are recruiting for these people is via a lot of competitions that you see online, Kaggleism is a great example of it. AWS has their own…you know, everything from DeepRacer competitions to just a bunch of other machine learning contests that you’ll have online. So, I’d say start with a lot of these competitions, start participating in them, because they make datasets available to you and then it boils down to honing your skills and how you do feature development for those datasets and how you build out those models. You can actually be on a pretty accelerated curve using that mechanism. Competitions will be my advice, go start participating in a bunch of these online contests and start building your skills that way because everyone that I know of is recruiting out of those competitions. You show up on the list there, you will get hired.

Rahul

I think I completely agree with that. One of the things that AWS has done in general is kind of democratized all of these otherwise very niche, you know, high-tech kind of operations. Like, machine learning 10 years ago was not really in the realm of, you know, a student in a dorm or, you know, anyone wanting to get into machine learning. Today, the fact that data sets are available in an s3 dump, which costs literally nothing per GB to compute that’s available by the R, the only thing that you need to really invest in is time and the effort, and the rest of the resources actually become incredibly cheap to get started with.

So, if you can spend the time, it’s actually really easy to learn fast, there’s tons of resources out there. I would also say reach out to some of the AWS heroes or the AWS community builders around machine learning. Reach out to us, you know, we have a bunch of AWS community builders within our own organization, we’d love to, you know, put you in touch with them, and give you a helping hand in getting started. But, yeah, there’s an amazing community of, you know, machine learning specialists who are more than happy to share their experience, to share their learnings, and help you get started. So, yeah, reach out to us and we’d love to, you know, connect you to all the right people.

Rahul

Yeah. In fact, I think outside of the U.S. in general, in most places around the world, people are multilingual. People usually know at least two, if not three languages. Where I come from in India, I think on an average, folks speak maybe three languages. In my family, I have folks who can speak up to seven languages. So, it’s very common to have people in multilingual setups. For our organization in particular, so more on the work side of things, we have folks from 130 different countries.

And it’s not just the dominant language that matters, but it’s also the accents that matter when you’re talking, you know, when your native language is something else other than English, being able to transcribe all of that knowledge and all of those conversations becomes quite a challenge. Like, there is…I mean, I know AWS has a few models, like, you know, Indian English or they had these hybrid ones where…we do end up using a lot of our Hindi or, you know, other regional languages as part of our English conversation. So, the fact that you can now automatically detect all the different languages because you can have more than one speaker in a conversation and be able to detect all of those words that people use from different languages, that’s pretty neat.

Rahul

Yeah, actually, the natural language detection and the transcription is becoming refined with every iteration. And, for us, one of the big initiatives that we’re doing internally is we have folks in 130 different countries who all collaborate across our group and bring in all of their perspectives into problem-solving, which is the thing that I love the most about my job working with all these amazing people. The thing that we’ve always lacked is, you know, we do hundreds of meetings in a day across all the different teams, but then we don’t have a mechanism of capturing that knowledge, that essence of those conversations.

So, one of the big pipelines that we built behind the scenes is whenever we do a Chime or a Zoom call, we stream all of that data, we run it through transcription, and then we save it as documents that then move into Kendra for indexing. And then you can ask a natural language question like, “Is there a conversation where they were talking about Product X, you know, and something else happened, you know, during the conversation?” Or, you know, “Were there these three specific people in a meeting and they talked about Product X?” Or if you want to remember a meeting, you’re like, “I wonder if I can pull up the transcripts for a meeting where I was there, Uznek [SP] was there, and then we had discussed this particular concept or this particular requirement from the customer.”

Those are the kinds of very, very interesting use cases around organization knowledge bases that Transcribe really enables you to kind of go after now. And Kendra, again, is another one where, to be able to take all of this transcript data…again, this is the NLP space, to be able to draw semantics out of it, to understand what it means at least in some sense, to be able to then answer natural language-based queries, is incredibly powerful. And like I’ve always said, AWS services, you should not think of them as just one service by itself. You should think of it as a whole set of building blocks that are made available to you and then it’s all about how you stitch these together to solve your particular problem. So, Transcribe for me is just one piece of a larger puzzle. It would be very interesting to see how folks are using Transcribe in different places.

Rahul

True. Okay.

Rahul

Let’s go to the next one.

Rahul

Yeah, so role-based access control, you know, is a more user-centric view of the world from an access control standpoint, like you basically define a role for a person and then that person gets, you know, those kinds of permissions. And this is very central to how IAM manages permission schemes today in AWS. It is all about what user you are, what roles you are part of, and what permissions do the roles have. Attribute-based access control is actually very interesting in the sense that you can now have tags for your resources, so you can carve out…or let me say it back. AWS has their way of seeing the world, AWS’s way of seeing the world is around resources they have and, you know, things at that level.

However, your business and the way you think about a particular pool of resources or a particular setpoint maybe completely different. For example, for some of our teams, they look at a project as their entire worldview of…you know, their entire worldviews around the project and the project has budgetary implications, the project has staffing implications, the project has implications around security and all other aspects of it. But the language that everyone speaks is the project, it could be a business unit for someone else, it could be something else.

But what the attribute-based access control allows you to do is it allows you to set tags for all of these resources or for a set of resources and users and whatever else. And you can then start defining permission schemes based on that rather than setting the permission scheme based on a role. A person might span across multiple different, you know, organizational units or BUs or projects, but you can define what permissions that they have at a time based on the tags that you’ve defined on the resources. So, I think it just gives everyone a lot more flexibility and a lot more ability to kind of move other levers from their perspective rather than stick to the one way everyone sees the world and that’s what’s incredible about ABAC.

Rahul

Exactly. And I think you’ll start seeing ABAC kind of become ubiquitous across AWS services. There are a bunch of standard patterns that AWS, you know, works on and you’ll start seeing this become a theme across the board. There was a time when VPC endpoints was the thing, you know, where everything was public initially, and then everything had to be brought into VPCs. Serverless is another big trend that’s currently ongoing, like every team is trying to turn their services into serverless because this makes it easy for customers. There was a time when IAM had just been launched back in 2010, I think, where teams were starting to adopt IAM and that was a thing. I think ABAC will become one of those things where, you know, leveraging tags to create your permission schemes is actually a very neat way of organizing things. So, I think we’ll see a lot more of this going.

Rahul

Absolutely. We’ve all had our IAM horror stories and nightmares. So, yeah, absolutely. I’m not sure that the complexity is going to completely go away. I think it just aligns a little better, so you have to kind of contort your worldview a little less. So, you can directly say, “This project is a permission scheme,” and get done with it. Right now, you have to figure out, “Oh, this project, I need a separate set of roles,” and then this is how it’s going to…you know, there’s a bunch of mappings that need to kind of go in place, this might actually make it easier. But I don’t think all the complexity will go away.

Rahul

Possibly. So, we learn more as more services implement ABAC and see how that pans out. But for now, definitely a welcome change.

Rahul

Yeah.

Rahul

Yeah, I think these are the kinds of instances you want to use for large data processing databases. So, there are a lot of databases where you have tons of stored procs, you have tons of, you know, local processing you need to do, whether that be for data analytics, like you’re building OLAP cubes in your instance. For things like that…or you’re running a Redshift cluster. For those kinds of operations where you need tons of memory, the ratio of memory to CPU is needed to be really high because you want to load up as much data in memory as possible.

You need a really good CPU because you’re processing a ton of that data, whether it’s building, you know, cubes in dimension and then being able to access it, or literally just running tons of stored procs because those are processing your data, you know, right then and there within the tables that you have in your DB. You know, in some cases, for example, if you’re running massive Elasticsearch clusters, these instances might be pretty interesting to run Elasticsearch on those. I actually see, you know, these instances being used for that. I was surprised to see another Intel variant come up this soon and for the price improvements, I was beginning to write off Intel, but it looks like they’re coming back pretty aggressively with some of these.

I don’t know how much further they can push these in…sorry, push on the performance, 3.5 gigahertz across all cores is pretty impressive. I don’t know how far you can push that. There was a recent…I was watching a little review recently, someone’s been able to take one of these processes and, you know, use liquid nitrogen, cooled it to a point where they’re able to, you know, eke out about six or a little over six gigahertz out of those processes but again, it’s not consistent. You know, it’s the peak performance boundary that you can push these cores. Yeah, I don’t know how far you can push this. You know, by looking at the TDPs for these kinds of processes, you’re basically building a heat engine at that point.

Rahul

Yeah, you could be contributing pretty significantly to global warming with a processor, you know, running kind of that way. But, yeah, I see the need for workloads like that and these numbers look pretty impressive, by the way, just to be clear.

Rahul

Yeah. I think the only ones that might actually beat this one…could you pull up the X1e 32 Excels? Both the X1 32 XL and the X1e 32XL, those are also 128-core CPUs with two and four terabytes of RAMs.

Rahul

Yeah, but it has four terabyte RAMs.

Rahul

Yeah, half the SSD, but four times the RAM.

Rahul

Correct. And what about the X1 32XL? Not the X for me, but can you look up the X1 32XL?

Rahul

Yeah, I think this was about 14 Gbps, if I’m not mistaken, on the X1. X1 did move to the Nitro some time ago, or maybe that would be X1e. I need to go look this one up again, it’s been a while. But, yeah, I mean, these are the kind of use cases. You know, like the X1s are custom made for SAP HANA kind of workloads where you have a lot of, you know, in-memory processing that you want to do. you want tons and tons of memory, you want CPUs to kind of be able to process all of that data quickly and these are perfect use cases for, you know, workloads like that. And there isn’t much difference between the 10 bucks on the new R6id and X1.32. So, it’d be interesting to see where you make the trade-off. Does it say what the CPU type is for the X1.32XL

Rahul

Yeah, these are about two generations old, they are Xeons, E7s. And E7s have been out for actually over four years now, if I’m not mistaken, four or five years now. So, these are much older processes, so they probably pull…does it say how many gigahertz they pull on a per-core basis?

Rahul

Yeah, so these are 2.3 and the new R6ids are pulling 3.5.

Rahul

Yeah, these are old, these are very old.

Rahul

Correct. I think these processes are…these Xeons, the Ice Lake processes that R6id is running on are the latest gen. 3.5 gigahertz is very impressive, so you’ll get way better CPU performance on these instances over even the x1.32xl. Unless your requirement really was insane memory…

Rahul

I think they just…so, you basically get 64 cores in the underlying…

Rahul

Hyperthreaded cores. So, you get 64 hyperthreaded cores in the underlying processor. So, I think that’s what you get, and in one case, you’ll get 64 2.52 gigahertz…or 2.3, I forget how much that was. But you get 64 of those and in the R6ids, you’ll get 64 of the 3.5 gigahertz cores.

Rahul

Yeah, it looks like 50% faster.

Rahul

Yep. So, one of the things where…these are interesting use cases where you have tons of data sitting in a data lake and you need to run massive MapReduce kind of operations where the data is the same but you kind of operating in little chunks and pieces, you have a lot of network transfer or data transfer that happens in these kinds of workloads. And for that, the instance type you want to pick…so the job themselves might be tiny, the jobs might be small, but the data transfer of the output of the job might be actually pretty large. In all those cases, you really need amazing network connectivity where the performance that you get over the network almost feels like it’s local disk performance. And when you have use cases like that, using these Nitro-based instances is incredibly valuable. That’s where the N series of processors comes in. And I think it just goes to show that…

Rahul

Yeah, here’s the new R5ns. I mean, just from a network performance standpoint, it’s mind-blowing that you could get 100 Gbps, you know, connectivity to whatever data you have. You couldn’t build your own data center with that kind of connectivity, you know, and sustained the throughput.

Rahul

Exactly. I mean, that’s just an insane engineering challenge and AWS has done an amazing job of doing that. Again, back to the point. AWS is doing a lot of the thinking for us about what is the most optimized resource service or pattern to use for different kinds of workloads. Every time they come up with something, I have to think about, “Where would I use this?” We are so used to using general-purpose compute because that’s our general-purpose resources because the overhead of thinking through and creating absolutely custom stuff is just too high.

But when you’re building building blocks, like what AWS is, it just completely makes sense. You can then have the freedom of picking just the right resource, just the right service that does that one thing for you optimally. So, it moves you from being, you know, these generalist products and generalist, you know, kind of services that you can bake internally to building an absolutely kick-ass world-class product or service based on very pointed services. And that’s what’s remarkable about the set of services that AWS has.

Rahul

Exactly.

Rahul

Absolutely. For the audience, please feel free to ask questions, post them offline, we’ll definitely bring them up in our next episode, and look forward to seeing you all soon.