Ask Us Anything: Episode 10

Rahul

I know.

Rahul

Yeah. Feels like quite an achievement and I think we should celebrate. Yeah, getting to 10 episodes is pretty awesome. When we started this series, you know, we were like, let’s just do it for fun. Let’s see, you know, how it goes. And we are already 10 episodes in, which is pretty neat.

Rahul

Awesome. So, how’s the week been for you guys or the weekend?

Rahul

Awesome. Badri, what about you?

Rahul

I had my taste of baklava. So I just returned back on Sunday from a five-week trip working remotely. So we spent three weeks in Seattle, a week in Hawaii, and then last week I was in Dubai. So yeah, I got my taste of baklava there as well, apart from all the other Middle Eastern sweets. And then yeah, just got back after five weeks. It’s awesome to be able to travel like that, you know, be completely remote and travel and work from anywhere and then come back, but it also then feels even better to be back. And I apologize in advance for a little bit of a humming that you might hear from some of the servo motors that are running on my printer. I have a 3D print going and it’s been hours, I think almost seven hours since this printer has been running. And it says another 20 minutes to go. So I feel scared at this point to turn it off or pause it.

Rahul

I am actually printing a case, a wall mounting case for an iPad for my home assistant dashboard. It’s been long time coming. And while I was away, I had a lot of my automation run. Amazingly, I have a lot of plants up in my terrace, and I built this, you know, drip irrigation system, which is all custom-controlled via my app that I wrote. And all my plants survived six weeks or five and a half weeks that we were away. So I count that as an accomplishment here. Just my final test run was about two or three days before we left. And then to have this thing run seamlessly without either the pipes bursting or, you know, something going wrong with the solenoids or the relays was quite an achievement given that everything was built from scratch. So, yeah.

Rahul

Yeah. Have a separate tag in there for all the home fixers and cloud fixers. Awesome.

Rahul

Oh yeah. Chennai went up to like 44 degrees centigrade or something like that in the middle.

Rahul

Yeah, 150? Yeah, it’s 115 plus. And it gets dry here. So you know, it was…It’s a miracle that all of it survived. I was monitoring it from the cameras while I was there. And I had to augment a little bit of the automation with extra doses of water being given on some days, but yeah, otherwise… And I was just very pleasantly surprised that it all just worked.

Rahul

Yeah, we have about 15 lemons that have kind of budded, so that was a nice surprise coming back in…

Rahul

We should have to do a completely separate segment by the way, on how we produce all of these videos and create the segments and then put them up on LinkedIn and our entire workflow and pipeline that’s also on AWS to do all of this stuff. I think that [crosstalk 00:07:24] fun conversation.

Rahul

Yeah. So this particular one, where you have heterogeneous clusters for Amazon SageMaker model training, definitely interesting. I think it just boils down to the fact that training these models starts getting so expensive, that everyone is looking for new ways to save money. And the fact that you can actually have different kinds of, you know, instances in a pool depending on, you know, what your savings plan look like, your CRIs or your RIs look like, you might be able to leverage some of those instances in some of your training models. So instead of just being stuck to one instance type, you could leverage any spare capacity you have lying elsewhere to run all of these models and thus save some money.

For me though, you know, honestly, I just wish they had spot instances, but they probably don’t have spot instances for two reasons. One is either they don’t have capacity. If they don’t have spare capacity, which I mean, given the amount of training and inferencing and stuff that happens right now in AWS and the fact that pretty much for everyone, the very first workloads that they move to AWS is machine learning related, I can imagine a situation where they don’t have any spare capacity left to allow them in spot. It would be a good, you know, experiment to run, to see if you can acquire any of the P2…the P instances, or the P family instances on the spot.

But that could be reason number one. Reason number two is it’s just not been implemented as a feature yet. And I don’t quite know what the complications might be because spots have been part of the compute platform for a very, very long time now. So it should…I had assumed be pretty standard as part of the underlying infrastructure to bring into any other service that leverages a bunch of compute. But Badri, thoughts?

Rahul

Yeah, I think it’s the same code. It’s almost like an autoscaling. It’s basically like your spot fleet or your, you know, autoscaling fleet that you can have where it’s just a bunch of different resources. Some of those training models on smaller machines will run longer. Some will run faster.

Rahul

Correct.

Rahul

Yeah. I think I was reviewing some of the pricing for these clusters, and it was either on-demand or you could do reserved instances, but there was no mechanism of doing spot in there. There wasn’t spot pricing available, but something I need to go look into it, but I didn’t see spot being available for these clusters.

Rahul

I’ll take a look at it for sure.

Rahul

Yeah. I mean, there used to be a system…there used to be a piece of software called… I think it was called Globus if I’m not mistaken. And this is back in 2003, we had actually an eight-machine cluster that we had set up to do this, you know, distributed processing. It was an optimization problem that we were working on. And yeah, we had to manage every aspect of that distributed compute that was happening then. And now that you can kind of just build these resilient systems that just assume that machines will go down like that’s the base assumption that the machine will go down. You don’t have to start worrying about, you know, a scenario where, you know, you have to take care of an event, like a machine going on. So [crosstalk 00:13:44] everything is different since then.

Rahul

We definitely want spot. I think having the spot fleet, if you could run spot fleet in this SageMaker model training, I think that would be a game-changer. If your costs were that low, whether just a fraction, my… And I’m gonna run an experiment and let’s talk about it next week. I want to run an experiment to see what spot capacity AWS has.

Rahul

The last time I ran an experiment to test AWS’s spot capacity, I got a call from them because at that time I was running X132 Excels to understand what spot capacity they had on that. I was trying to launch 100 of them. And I think I got a panic call from someone in AWS fleet planning or capacity planning. So…

Rahul

Absolutely.

Rahul

Badri, you wanna start with this one?

Rahul

Yeah. I’m actually very surprised by the gap in the savings between what the paper was claiming and what AWS’s numbers are. So there probably is something fundamentally different because when they came to… The paper, I think says something like the eventual cost was just about 7%. There’s still a lot left on the table, and I wonder why. Either the approach is different and there’s gonna be more fine-tuning coming in, or it’s also possible that the… Yeah, I guess since you’re operating in kind of three different… You’re trying to basically figure out what your best answers could be in terms of latency of your inferences, the model that gives you the faster latencies.

The size of the model, of course, because it depends on where you want to install the model. And then the third part of it is the cost of actually training and the cost of inference. And it basically becomes a trade-off at the end of the day once you have a selection of these models. Are they doing something different to be able to get better at those three dimensions and getting models and answers across those three dimensions, which is why it’s taking them to only the 50% mark instead of the 90%?

Rahul

Correct. Model size would be interesting to figure out, you know, what… Because that is one of the trade-off parameters that you have to come up with. So, it’ll be interesting to see. The dataset sizes are different. And the larger your dataset size, your improvements are smaller, which to me is a little counterintuitive.

Rahul

No, this is actually just hyper-parameter tuning, right? It’s not model selection. So this is primarily just hyperparameter tuning.

Rahul

Yeah. Okay. Let’s do a takeaway. We will reach out to the AWS team and get an answer on the comparison between that paper and to really understand what they did.

Rahul

I think this is a very, very new announcement. We haven’t gotten down to circling back with AWS on this, but yeah, we’ll definitely want to bring this back.

Rahul

So I think for the vast majority… So I put it in two categories… Or actually, yeah, the three categories are one is you have the super-advanced users who are building their own models from scratch and doing every aspect of it, controlling every parameter of it. The second one is ones who are using autopilot to build models. And the third one is your vertically AI services like recognition, comprehend, forecast, personalize, all of those. Is that your question? Is that your third category?

Rahul

Okay. So, I actually think the vast majority of machine learning customers, especially the ones… I mean, I’ve spoken to a bunch of them and I get the impression that they rely a lot on hand-building their machine learning models. They use the compute, they use SageMaker notebooks, they use all of that stuff, but instead of letting something like autopilot run and finances, the people who are actually doing this work know what they want to do and therefore just run the models themselves more often than not. So I’d say that has the highest use. People who are building end-user applications and want something that’s a cursory feature or I’d say a peripheral feature of their application tend to use their vertical or a lot more of it.

So for example, if, you know, providing not safe for work is a peripheral feature of your product, not something that’s core and central to it. Like it’s feature number, you know, 75 in the list of features, then they tend to kind of take this as an approach to kind of, you know, using recognition or something like that to do it. I have not known of very many products that have started using these services as their first choice for their core and fundamental feature. Like, I don’t know if anyone… If you look at a service like transcription, if you look at what Otter is doing, you look at you know, a bunch of other providers who are doing transcription, you find that none of them is just packaging up comprehend… Oh, sorry. None of them is packaging transcribe and comprehend to do all those basic things. They are actually building custom models, right?

And that is a little surprising. I haven’t come across any product, which just built a very thin layer over these AWS services. Though I think they should. I think it makes perfect sense to do that. So yeah, I don’t think that many, that very many people are actually using those services. And I think a bunch of newbies who are getting into the ML world, still trying to discover it, still trying to figure out what it can do for them, are starting to use Autopilot. So, people who have all the data but haven’t really, like, they don’t know how to get started with ML in general, they’re, like, okay, here’s an algorithm. Let me see what you can… You know, here’s what I care about. Let me see what it can produce.

Rahul

Yeah. And I think the reason why I like Autopilot is because it actually gives you…it makes you take those trade-offs, right? It says your three dimensions are accuracy, latency of your inferences, and the size of the model. Right? It literally gives you those three parameters and it gives you the best models in those three categories, and you go pick the one that meets your business requirement. And I think using automaker forces you to make that trade-off across those dimensions. I don’t think most data scientists even think about it as they are… You know, unless it is governed, unless some constraints are governed by hardware, like model sizes, I don’t think most data scientists even think about where the trade-off line is gonna be made between accuracy and model size and latency. Yeah. They’re trying to get the best results and they’ll go endlessly in a number of cases to the point of diminishing returns. Yeah.

Rahul

We’ve all lived through scenarios like that.

Rahul

Correct. Yeah. So I think this is for anyone who is developing applications that also need, you know, clients or, you know, need something specific built for the Mac ecosystem. You wanna put something on the app store, you know, either for your mobile apps or your iPad apps or your desktop apps or even for that matter, if you are doing Safari plugins and so on, you kind of require Xcode to run, to build them, to compile them. And this, I think they’re literally going right after the likes of MacStadium who have provided this service kind of for years historically. And anyone, you know, if you wanted the latest version of Mac with a particular OS on it and wanted to test out all your, you know, builds and setups, you would basically do this.

You would, you know, provision an instance, run all your Xcode stuff on it, build it, deploy it from there. We’ve had Macs in the AWS ecosystem as one of the instance sites for a while. Now that the new ARM-based ones have come, they’ve actually, you know, created a big problem because now you had these ARM-based processes. You want to make sure that all of your stuff just worked as is on this completely different architecture. There was support for, you know, a setup, X86 apps could still work, but they wanted you to recompile all the stuff. They made it really easy for you to recompile stuff for the next code and generate the ARM binaries. But you needed to do it on an M1 machine. And now that this is available, I think dev teams, in general, can do this.

The thing that I usually dislike about, or I had historically disliked about the Apple ecosystem, was it created an even bigger onus on, it has to build on your local machine for you to create a binary and deploy to production, right? And for the first time… I mean, MacStadium did some of it, but it was a pretty manual process at that. For the first time, you can actually build an entire automated CI/CD pipeline that processes all of your builds in the cloud on standardized, clean, or pristine environments and can, you know, process and create your builds and deploy stuff into your app store. So that I think is pretty neat. And these days, everyone is building, you know, iPhone apps and iPad apps for their products. So yeah, there’s a huge demand for it.

Rahul

And the thing is that there’s always going be demand for something like this, right? I mean the M2s are already out. The M1 Macs and Ultras, though they’re the same architecture, the SoC is kind of different. The Ultra is literally two of these fused together. And so I don’t know, I haven’t come across any scenarios yet, and it’d be interesting to see if people need to do specific testing for those specific environments as well.

So yeah, it is gonna be pretty interesting to see how people are using these. I think it’s really required for headless CI/CD setups. Like you do not want to build locally. Like, I struggle to set up, anytime I’m doing a new project on Xcode, there’s just so much other junk on my machine that it messes something or the other up. And clean environments is I think the bane of most developers around the world.

Rahul

Yeah, let’s do that.

Rahul

So, before we jump into the other serious stuff, I have to make a comment about the audio. What audio? And that a really funky audio.

Rahul

You guys should check it all out…

Rahul

But yeah, going back to this particular one, yeah, DevSpaces is pretty neat. It gives you all these amazing clean environments to work with. Like, I’ve stopped doing local development, to be honest, on my machine. And I switched over to the iPad because this shows up in a browser and I can work out of pretty much anywhere. And with the last five weeks of travel, it was awesome. But coming back to this, so the ARM flavor that AWS runs is Graviton, right? It’s you’ve got the graviton 2s. Now, the Graviton 3 is available and GA for the C or the compute family, the C7Gs are now available in GA.

So if you wanted to develop something for the C7G, even the Mac for that matter is even though it’s ARM architecture, there are different flavors of the ARM architecture. You don’t exactly know if everything will continue to work exactly the same way once you get it into Graviton. So in an ideal scenario, you would want to develop on a Graviton instance. And what DevSpaces does for you is it gives you those clean environments with Graviton machines. So…

Rahul

Yeah. My favorite experience out of all of this is actually doing PR reviews. So when I get a PR review, I get… Because usually, you know, developers are sending me a bunch of them. I literally get a click on a link from the DevSpaces plugin that we have, literally allows me to go from the PR request, click a button. It brings up an entire environment with the PR showing up with all the Devs. I get to review it. And then I run the entire tests right there without it, you know, completely hemorrhaging my machine, or, you know, putting it into a stall state. I get to review the PR and I approve it right there, and then I kill the entire environment. So I never bring anything locally onto my machine. I get to test it and run it.

In fact, earlier, you know, I used to be so frustrated with trying to build it and get it all working on my local machine. It would just take so long, I’d just do a cursory read-only of the code. And I think in my… I actually always felt like I was doing a shoddy job of reviewing those PRs. And now that I can not only review it, review the text and the content, but I can also run a test or make sure all the tests are running, do all of those, you know, or even just run the code, run the app, make sure it’s all working as I expect. And once I’m happy with that, I just kill the entire environment. And so I can run through PRs in a matter of like 10, 15 minutes now as against…

Rahul

Correct. I mean, the amount of Git stashes I used to do before, you had to be working on a particular branch, someone’s PR came in, stash it all, then come back here and pop it out. And then it…my local Gits used to be an absolute mess, not to mention the environments. Like, God help you if someone brought in a dependency, it would just completely derail my workflow, it would derail my entire work and my day for that matter.

Rahul

Correct.

Rahul

Absolutely.

Rahul

Okay. So, Badri, you wanna take a first pass at this announcement?

Rahul

Yeah. I think they do a pretty good job of, you know, looking at the patterns and giving you all the best practices. Like if you go to the foundational security best practices standards… Why don’t we pull that up, Stephen? I think they go to the level of literally giving you, you know, step-by-step guidance. If you just go down to that, just go down one level. Go to the… Yeah, that one. The best practices controls. So if you’ll just scroll down a bit, they will give you step by step…

Rahul

Okay, great. So they tell you what the problem is, they give you all the recommendations, they tell you what you should do or shouldn’t do…and if you scrolled on just a little bit, they’ll give you the remediation method. Literally, point by point, they will say, do this, do this, do this, do this. And…

Rahul

I really wish that there was one simple automation or a script that they just turned on for you. Like, this is a security best practice. I wonder if they ever have a conversation where they go to a customer and say, “Yeah, we know you probably don’t want the security best practice, but we’ve spent all this hard work doing.” Like who here is… It is like when Bezos talks about his in variance, like, customers always want more, you know, faster delivery and lower cost. And he’s like, I have never met a customer that I go to and says… Who comes in and says, “I want my stuff to be more expensive.” Right? This is one of those things. Like, who goes to AWS and says, “No, I want, you know, this security to be loose.” Or, you know, “I want all these security vulnerabilities to be there in my system.” Like why aren’t these just turned on by default?

I thought one of the things that AWS did a few years ago, I think actually quite a few years ago now, they made IAM policies or IAM, yeah, IAM policies you know, forbid everything by default. And then you had to kind of build up from there because can’t imagine anyone wanting to start with, “Hey, let me open up everything to the world. And then let me start figuring out what I need to, you know, shut off.” So this is similar, like, why isn’t that a standard across the board? That’s something that I don’t understand. And why isn’t this just getting implemented as is?

Rahul

No, but things like this, like there are already remediation. So you have SSM for example, which will allow you to log into your instance, even if it is inside a VPC, it’ll allow you to play around with it and do whatever you need to do without violating any of those controls. So the need for something like this to exist, I don’t really get, because anything that’s inside of VPC should be controlled by the policies and the rules of the VPC. And if you need to, as an individual, access a particular instance, for whatever reason, the last thing that you should do is attach a public IP address to it. It just uses SSM. So you should use SSM. You know, just log in using SSM and…

Rahul

…do whatever you need to do. So I went through… I spent some time going through a bunch of these and I was like, these should be turned on by default. Like there’s literally no excuse to keep these on.

Rahul

Correct. So these controls are basically just, hey, just red flagging this thing for you, you know, which is interesting and neat, but someone going in and fixing this, like looking…an average account these days has 500 resources, right? And across these services, you’re gonna have about 500 resources. Who’s gonna go down, and there are what, seven or eight steps per remediation, who’s gotta do 500 into eight remediations, you know, one by one by one? It’s, no one’s gonna care about it.

That’s where AWS falls short of, you know, helping customers actually fix these problems. They’re highlighting them and saying, we’ve done our job, now it’s your problem. And I think that’s where… You know, that’s kind of the whole reason why CloudFix came to be, and we just gotta take all of these and put them in CloudFix. It’s, we hope customers will benefit from having all of these standard hygiene rules built and auto remediated for them.

Rahul

Yeah.

Rahul

Yeah. Security app just says, “Hey, here’s a dashboard that tells you all the things that is wrong with your account from a security standpoint. And then you basically have the link that’ll take you to this article. Then you go to the article and then you go one by one, follow the steps. It’s impractical, in my view.

Rahul

Yeah. In fact, that brings up an interesting idea. I think we are gonna start an open… I’m thinking of, we should probably start an open-source project and invite other contributors. We will go ahead and create a whole series of change manager templates. Let’s do that. Let’s create change manager templates for all of these and get AWS to certify them.

Rahul

Right. I mean, these are AWS recommendations, so it’s like all of the other ones, let’s get AWS to certify these. And I’m pretty confident the config manager, the change manager team has gotta be pretty excited about it.

Rahul

Yeah, let’s talk about IAM.

Rahul

Huh, where do I start? So, IAM for me is I think one of the most painful of the AWS services because it adds too much friction in the way it is designed and it operates today to getting stuff done in general. I struggle [crosstalk 00:53:51] this one for two reasons… For this announcement for two reasons, number one is this doesn’t take away the problem of… For me, IAM authorization, of course, is one aspect of it. But…sorry, authentication is one aspect of it, authorization is the other piece. And this doesn’t solve the problem of centralizing authorization in IAM. When you have applications, this basically just gives you the certificate and that digital certificate will be shared with the application. Your access controller authorization is still gonna be managed within the applications as I understand it. I don’t know how you would manage that within IAM itself. I…

Rahul

Yeah. Let’s say something like Salesforce. I mean, you are using Salesforce that is outside AWS. And lets say Salesforce allow you to use digital certificate, the X 509 certificate. Then you could actually use your users certificate in IAM or get temporary credentials, which is basically this X 509 certificate and use that to sign into Salesforce. So that can be done. But then the access control, what data will you have access to, that is managed and maintained within that system and not within IAM roles and policies. So the more I think about it, and Badri and I were just having a little bit of a chat before this, if you were to take IAM and extend it into authorization as well, what would you do?

That’s a really hard problem because it’s the applications that define your roles. It’s the applications that define resources. And if you were to do something like RBAC, you need to be able to somehow pass those roles and resources back to IAM and have IAM recognize them so that you can model them there and have some way of IAM then pass that back to the application or have the application call into IAM to actually do authorization resolution or RBAC resolution. So I don’t know, it seems to get very complicated and messy, and I don’t know what prompted AWS or what use cases prompted AWS to create this. Badri, any thoughts?

Rahul

Yeah. I think the one thing that certainly is interesting is the fact that this has a valid use case for temporary credentials. Like if you make a request, you get your temporary credentials, and then you use it over there. I guess the secrets manager stuff would be a little bit of a workaround in that you have some mechanism that constantly resets the password at whatever regular intervals or whatever. And this kind of flips it and says, you get a temporary credential, it has an expiration in the certificate. So as soon as, you know, that particular time elapse, you have authorization for five minutes to do whatever you need to do. And I guess that use case makes sense the more I think about it. But that difference is very, very subtle. For a lot of external systems, like how many systems would you need temporary access for a certain period of time?

Rahul

And for a lot of the third party applications also, the more of the two-factor routes that starts taking, you know, playing a factor in your authentication part of it, like what would an app… If I were a third party app, not in the AWS ecosystem, there’s a big mandate to make sure that you have two-factorial authentication working over here. How does that work with IAM?

Rahul

I mean, no one can help you if you’re doing this in plain text. No amount of other, you know, checks and balance are gonna help you. But assuming that you were using something like secrets manager, right. I mean, this adds very limited value on top of it.

Rahul

No, two hours a day is not even remotely close to being able to keep up with all of these conversations, but this is awesome. So we have three takeaways that we are gonna follow up on. The first one is we are gonna run an experiment on trying to figure out spot instance availability for the P-class instances and see if they actually have any capacity and where that might be. That would make a good conversation. The second one…remind me what the second one was. The third one, of course, was we were going to start doing some SSM templates, so SSM documents that try and take a lot of these security hub policies and hopefully kick start an open source project for them.

Rahul

Thanks, everyone.