Ask Us Anything: Episode 3 – AWS EKS Anywhere and more

Rahul Subramaniam

Doing very well. Looking forward to the discussion today.

Rahul Subramaniam

Really good. For the audience. Both Stephen and I are big lego fans. We just spent the last four months working on that Bugatti Chiron and that is an insanely complex piece. It took us about four months to build it. What really amazed that how much concentration it gives the kids, they sat for hours at the end trying to put together these pieces following hundreds of pages of instruction so that it’s really, really interesting learning experience for all of us.

Rahul Subramaniam

I think so, I think these have gotten really, really, seriously complicated. These ones have, you have to build the engine with all the camshafts, the transmissions, the differentials, steering mechanisms, all of that actually works on discussing, they’ve tried to make it very complex and as realistic as possible.

Rahul Subramaniam

No, it’s fast and it’s great exposure for the kids as well um, to have to learn all these to go through these experiences um definitely very very enriching.

Rahul Subramaniam

Pretty much. We had a number of showers over the last two or three days, but now it’s back to being hot, sultry and uh yeah, we’re gonna live through that for the next few weeks.

Rahul Subramaniam

Absolutely.

Rahul Subramaniam

Yeah. So um trusted, You don’t think of trusted advisor as your place to go for a lot of the AWS this practice recommendations for which they have built a bunch of finders and that’s the place where they bring in all of these recommendations. They have seven core services of categories under which a lot of these um uh insights or advisors are coming in the services, primary R. S. Three. You’ll see security groups, you will see I am. You will see multi factor authentication again along your permission schemes you’ll see some recommendations on RDS E B S uh service limits and so on. Um broadly I think the vast majority of recommendations that exist today fall under the security category. So even though they are across all of these services, I think the first ones they have focused on and rightly so are all the security ones where developers um or customers end up leaving some loopholes in their accounts. Whether those we don’t log in with your root account kind of recommendations using I enroll for your services um you know use tokens that expired, you know on regular use and use refresh tokens to operate your services. Um things like that, don’t you know, open up um everything to a c idea of 00 slash zero kind of set up for your resources. Those are kind of best practices that they put into a trusted advisor now over a period of time. What trusted advisor has been doing has been drawing from a bunch of other tools that AWS has built and computer optimizer happens to be one of them. And where do the computer computer optimizer help is they bring in certain insights that are related typically to cost and appropriate resource utilization. Um They usually started off with Recommendations around ec two instances where they would take a look at your CPU utilization memory, um network I. O and disk IO. And make a few options as recommendations. And you could pick the one once you did that. It helps you run a few. What if scenarios to understand what might happen if you chose one of the three or four options that they provide and that had to make better decisions. Now, just to be clear these are advisories where you still need to do a bunch of work. You still need to go look at those options where those options make tradeoffs as necessary. Um You also need to make sure that you’ve covered some of the fundamental requirements. So for example please I request request everyone install the SSN agent in your on all your instances. It doesn’t happen by default install it. That you can actually start capturing memory metrics that you can make better optimization decisions without simple metrics like that. It’s really hard to make decisions around what the right size for your instance is going to be. The next thing that I recommend everyone does is really think about all the valuables in making a change on these instances because you have to think about what is actually running in there. Is it really stateless. Is there stuff that’s in memory that’s not being persisted on this because when you change your instance type, you’re going to have to take a snapshot, turn down your machine. Start up the new instance and then you start using the new instance. So if there’s stuff that’s still in memory that’s not really persistent, you might end up losing it. So be cautious about all of those things. But if you actually get to the right resource utilization, I mean, you you suddenly find that everything is far more performant, you’ve balanced, you know, all of your trade offs between different IO’s I apps, CPU utilization and the utilization. Everything just works seamlessly. So there’s a bunch of that you have to do, the other area where we’ve actually found computer optimized to be incredibly valuable as landowners. Um really trying to figure out how much computer and memory you need for lander functions is incredibly hard because it’s serve earless. You don’t have control over the silver and you have to basically look at executions over a vast number of lambda calls. And what computer optimizer does for you is it actually does all that for you. That’s all the hard work, collect all of that information about how much CPU got used, how much memory got used, does all the matter for you and tells you what the right size for cores and memory should be and they just announced uh memory size over provisioning. Um as one of the insights out of computer optimizer, which is awesome. They were doing CPU earlier now it’s memory as well. So you get to optimize your CPU and memory and your lambda costs quite significantly? That’s that’s okay.

Rahul Subramaniam

So this is an interesting one i in my fear that I actually find that um developers either operate in powers of two or powers of 10, so when they provisioned ebs volumes, it’s either eight gigs, 16 gigs, 64 gigs, you know, they either go in those, you know, powers of two or they would do 10 gigs, 100 gigs, uh you know, tv kind of volume provisioning. And to be honest, I don’t think developers spend enough time thinking about how much this space you actually need on the face of it, it actually looks really cheap To, you know, take up, you know, one gig or 10 gig or 10 gig at 100 gig um ebs volume and just slap it onto your instance. But over a period of time, as you do a lot more of these, you suddenly find that you’re wasting so much space on those volumes, it’s just, you know, you really never need that much and in most cases I found volumes to be provisions in excess of 10 X. Of what they achieved needs. So um there’s tons of room for right sizing them and I think having computer optimizer do that for you or the box is pretty neat, It’s actually fairly straightforward also to resize your ebs volumes. So if they are small and you need to resize them at a later stage, you can actually do that with a very simple a vehicle. So that’s pretty neat, so it makes sense to right size them at regular intervals and make sure that you have them running optimally.

Rahul Subramaniam

So I think this set of recommendations are disjointed from what loud fix does not think focuses primarily on cost advisories from AWS that are completely nondestructive. Uh So with literally a couple of clicks you can start saving money without worrying about you know, restarting instances of something getting lost or you know, disruption to your service. These ones are disruptive because they either need an instance, we started talking about instances, ebs volume resizing um depends on the scenario which way you’re going, you may or may not require you to detach the volume, work on it and then reattach it and then lambdas when you resize them may cause a little bit of destruction as it’s kind of turning over unless you manage um you know the mechanism by which these changes are being made. So I would say um potentially disruptive And not, you know, 100% said that you do need to make some tradeoffs at times. So the Venn diagram stands disjointed at this point between what computer optimizer is doing and what cloud fixed up.

Rahul Subramaniam

Correct. And by the way, one other thing is there isn’t one answer to these computer optimized recommendations. Like if you go to the ec 21, they typically will list three or four options with different profiles. Do you care about you know, finishing the work load faster? In which case if you find that it’s multi terry, they bump up the cpus they will, you can probably recite to, you know, compute optimized instances or if you don’t want to reduce your cost, it might be memory optimized instances. If your network I. O. And stuff are the primary drivers afraid you might decide to go with certain other kinds of IO optimized our ebs optimized volumes that go on those instances. So there are different trade offs you have to make depending on what your motivation is for that work, Lord. And that’s not something that a machine can make um you know, just out of the box. So given that you as a human has to go in, you have to look at these options, make tradeoffs, make a decision and then execute on God.

Rahul Subramaniam

That’s correct.

Rahul Subramaniam

Yeah, So I think um the way you should think about its computing is there are certain scenarios where the, the cloud compute infrastructure has to kind of extend to the edge because of constraints around bandwidth and latency. Um you have to have considerations around um where your um, what the workload really is and how much data are you actually ending up processing? Um There are times when the sheer volume of data that you’re processing is so large that there’s no way, no matter how large pipes you have connecting you to the cloud, there’s no way you can go through all of that data to give you an example. And that’s kind of a lot of the work that panorama is doing um for more details, by the way, there’s a amazing podcast that I did with Omar’s Arca, the Gm of AWS panorama just a few weeks ago, um as he talks about where panorama, you know excels in its compute and the kind of use cases that they’ve been working on, but I’ll try and cover some of that over here.

Rahul Subramaniam

Yeah, so um some of the examples that Omar was talking about um in that particular episode was um they’ve got you know, warehouses, they’ve got you know, big supply chain logistics partners, where what you have on premise is all of these CCTV cameras that are literally monitoring and managing everything, but that data has not been made use of Apple and with this um edge appliance that they have, you can actually take machine learning models, deploy them on the edge remotely from your interviews console, you can deploy them on the edge and you can now literally just connect your your cameras, your I. P. Cameras, you don’t need to change any of the infrastructure, you don’t need to change of your hardware, just add this new appliance to your rack and you know, start um put them in the same network as your cameras. Start, you know, getting data from data streams from your cameras and then you can start applying machine learning models to whatever is happening over there. Some use cases where pretty amazing in certain hospitals, they’re looking at cases where um you know, patient care. Um some use cases on patient care, like making sure patients don’t fall off their beds or predicting if a patient is likely to fall off their bed while they are in the hospital. Things like that are really remarkable. Use cases I think tomorrow is talking about another case where on remote farms, sometimes the people who are actually inflicting cruelty to the animals that are, you know, the the humans are responsible for inflicting cruelty to animals. But to be able to detect that remotely and using machine learning models and act on it immediately is something very remarkable to understand, you know, what’s happening around security logistics where every packages are, where shipments and containers are moving in massive dockyards. Those are all use cases that Panorama is now enabling and having this new device that’s now available at a lower price point is pretty amazing. Um to the original AWS device that they came up with the AWS has was based on the NVIDIA Jetson, a G X Xavier processor, um, which I think you have that over there. So that’s the Xavier NX um, the A G X is what goes into the AWS build um device and that has a lot more throughput and not more bandwidth to to basically. And if I’m not mistaken, instead of up to um 8 to 20 cameras, but it also depends on what kind of models you’re running. So how much CPU you will actually end up utilizing for machine learning models. But these other ones um, the new ones that have just been launched by um the novel are pretty neat too. This is some powerful, powerful kit here. It’s a 384 and video course 48 tensor cores. The sixth corps, we’ve got the end video arm and then also it’s their hardened devices. If you look at this photo there there fanless design, their industrial application, you said could be used in a loud, busy factory floor or farming applications and have been able to have that image recognition right there. And also you mentioned medical, right? This can do image recognition or voice recognition where the data isn’t leaving the premises, which is really important in some of these regulated environments. But you can still take advantage of AWS scale in developing these large, large models. Um that’s pretty neat. I wish I personally had an application for this where I could uh I think for me, a raspberry pi and Arduino is probably gonna be sufficient, but if I did have a factory, I would definitely want one of these. So I actually ended up getting one of these Jetson nano’s. And uh I’m still working on the machine learning model around the security in my house. I have 11 cameras around my place and uh I’m looking forward to experimenting with the Jetson nano to see what kind of models I can come up with.

Rahul Subramaniam

There’s actually some really interesting work being done in defect detection um where There are some like if you look at lookout for vision as one of the aws services, I think if you give it something like 10 images of just for training of defective and non defective pieces, it does a pretty neat job of identifying um, anomalies or defects for all future, you know, pieces and parts that come through the line. So check out AWS lookout for vision as well. It’s a very, very mean service in the manufacturing and industrial space. If you have a use case, we’d love to know.

Rahul Subramaniam

So I think the question says in what situation would you recommend using an auto scaling easy to volume in a cost reduction perspective. So I’m a little confused. Um to be honest about this particular question, I’m wondering if it is ebs volume or are you talking about auto scaling easy to because those are due different things, but let me, let me talk about using order scaling ec two instances, um, I think that’s the right way to do it because the whole point of leveraging the cloud is that you have elastic workloads or you have elastic compute being leveraged to perform processing on your workloads, so as long as you’re you’re processing is going to be elastic where it scales up and down depending on how much workload is coming in, you actually optimizing costs. So auto scaling groups, leveraging them should be kind of done by default. It requires you to rethink your core architecture, you can’t state fullness doesn’t really work without a scaling. It’s got to be fairly stateless uh to operate. So if you have state full workloads, state full processing happening within your processes. That’s when we are tricky, you have to figure out how to extract that out and make it stateless because once you do that, then you can order scale up and down and not have to worry about um you know, managing state, there are some workarounds, we’re gonna talk about EFS shortly one of the ways to manage some of the state fullness that exists in file systems, but in general, I think statelessness is where you need to be in the long run and yeah, and then E B S we don’t really think of ebs volumes as auto scaling, you can resize ebs volumes volumes as and when you need it, which is actually really handy thing. So, my only advice there is have regular checks set thresholds for your ebs volume limits. So for example, as soon as your head, You know, you want your EBS volume to operate between 60 and 80% of socialization or you know, space. So haven’t triggered at 60 have a trigger at 80 when it you know, approaches 80 or just as soon as it crosses 80% has threshold, triggering KPI it re sizes your volume to go up to a point when it comes back to 60. If it drops below 60 have a trigger that will resize it down so that the utilization is 80 again and you can do this with a very simple lambda function that called the P I to rule to do the research.

Rahul Subramaniam

Mm hmm. Yeah. So just for some background context, those not familiar with the fsc. Fsc, I mean think of it as it reduces version of providing some service like NFS And there are two areas where you end up needing something like this. The first one like I said was you want ephemeral compute where the compute is, um, you know, short lived and for example, you want to, you, you need tons and tons of compute to do something but you don’t want to spend an insane amount doing that. So you want to use and leverage as much of sport instances as possible. But the drawback of spot instances is that they can be taken away at any point of time, but spot instances could be 90% cheaper than your on demand instances, right? But then it comes to state, where do you store state when you’re processing tons and tons of workload, you either store state in a database, but a lot of older applications towards ST in the file system, like you took some data, you did some processing and you dumped it back in your file system. Right? So the file system was one of the mechanisms of storing state. Now when you don’t have something like NFS, you only have ebs volumes. Your file system is local to your machine, your instance, right? The only way you can take that data and put it somewhere else is either detached the volume and attach it somewhere else or actually try and do some kind of life transfer um, you know, from one instance to another and when everything is ephemeral orchestrating, that is an absolute nightmare. So E F S comes as an amazing bridge that helps you resolve that gap in being able to deal with some of the older workloads that still don’t use things like object store, um are you required to use the file system? I’ll give you an example of a place that we started using NFS very extensively. Um today we manage about, you know, across the, across the entire portfolio, we have about 2, 2.5 billion lines of code that we manage and they are all engaged repositories. We have, you know, I think nearly 100,000 git repositories if not more, I’ve forgotten how many there are now. But one of the things that we do as part of our analysis whenever we acquire a company, whenever we are really trying to understand what’s happening with the court basis is we always want to know how we got here. Typically the companies that we acquired our 10 2030 years old and they have tens of thousands of comments, if not hundreds of thousands of commits across different branches. Um, so one of the most important things we try to identify is how did we get to the state of the core base that we acquired? So we literally start analyzing the code from commit number one and then go down coming after coming to understand what changes got made, what technical debt got added at what step, why did it get added? What was the evolution of the different features? And so you have to do this commit by commit across hundreds of thousands of code bases. So you want ephemeral compute to do all of this Unfortunately, because this is all in debt, we couldn’t use, we couldn’t use it as three as a plus same mechanism because get operational file system, You can’t run, get on S three. So we were limited by the fact that each commit is very beautifully packaged in your GIT repository, but it always runs on the file system. So we had we needed some sort of some sort of a file system to operate on and cbs wouldn’t work because the average get repository that we have is about Somewhere around 5- six gigs. So every time if you needed to take the six gig repository and split it out across 1000 compute units, you’re copying six gigs 1000 times and that is not cheap and that is not fast. And so you can have this, that’s incredible.

Rahul Subramaniam

Right, where you said you needed a lot of files, correct? We need tons and tons of files that we write to because those, so it’s all, it’s, you should think of it literally as a pipeline of processes, um they’re just running, you know, step up a step. And so every step, writes back to E F s with some of its analysis and then other processes pick it up because they pick up a committee to pick up the previous analysis because independently you need to do a diff between your previous analysis on the previous commit and the current analysis that you just got on your existing carpet and then you do a diff, you write that as you know, um analysis for the next step and so on. So locks started becoming an integral part of what we have to watch for. We actually ended up even though we were using lambdas and we’re using spot instances for managing our computer. We were limited by the number of locks we had because that was literally the number of Landers you could you could run but that was the number of spot instances you could run um exactly jobs that you could run at the same time because all of these cases is holding 100 locks. And if you increase the number of if you grew your computer beyond that limit, you will basically start um you know, throttling you start receiving tons of letters and you have to throttle your entire pipeline.

Rahul Subramaniam

Absolutely. We could do always infuriate times faster when you acquire a company, which is awesome because right now it takes us a few days to finish. Plus saying, you know, in some cases nearly millions of comments, you know,

Rahul Subramaniam

Absolutely, we’d love to do it

Rahul Subramaniam

Now actually E F S given, you know, the old literature of NFS um has is one of the services that has so much room for improvement in a cloud native environment, like you can start rethinking what it means to build, storage is accessible across multiple computers with multiple rights happening at the bank, you can truly build it in a collaborative manner. So there’s tons of head room to grow.

Rahul Subramaniam

Yeah so I’ll answer this very quickly. Lex is actually one of those services that I absolutely love. Um if our audience hasn’t looked at it um I’ll break it down very simply lex has this concept of an intent and intent is basically what you intend to do. Uh And an intent is constructed of basically the set of phrases that you might say ah and a bunch of slots. The stink of slots as valuables, right? So for example you might ask um you might want to ask a question like Um you know, book me a ticket from L. A. to New York on the 25th of december, right? If you had a phrase like that, that’s your intent booking a ticket is your intent. The utterances are book me a ticket or you have to three different kinds of ways of saying it. Um and then your slots are your origin, your destination and your date, right? Those are your variables and the job of resolving that.

Rahul Subramaniam

Absolutely. And actually one of the things that I really love about lex and I wish more people would do this is whenever you’re building an application, you invariably get inhibited by the Ui and the U. S. You have to build to figure out how customers can access a set of api that you build. Engineers are really good at building A. P. I. Because they are to the point, the function of the api call does one thing, it has a set of variables that you pass it as parameters and it produces an output that you can consume right, but for an api to be leveraged to build this whole massive UI and the UX around it to make sure that people find that aPI call, they know how to get it, they know how to pass, you know, do the right things to get the right parameters passed to it and then make sure that the data is presented back to them in a manner that is digestible, consumable and all of that stuff. The conversational parts that you can build with legs are amazing because you can literally build it to do one thing and one thing really well which is do the job that the API does for you. So you can ask a question that question maps directly to a single Api call the slots in your intent get passed in as parameters to the api call, whatever the output is goes directly in your chat, you know, client and you get the answer then and there, it’s probably the easiest way with the least amount of friction that you see in us and different kinds of us so that people have built over the years. It’s probably the easiest way to get to what you want to do with a whole bunch of applications out there, especially when it comes to data analytics, data analysis, um you know, slicing and dicing stuff using a chat board to do? It could potentially be revolutionary.

Rahul Subramaniam

Correct.

Rahul Subramaniam

So, um, kubernetes is, do you think of communities more as a framework? Um, it’s basically an an orchestration framework that manages the life cycles of your containment. Right, But there is so much more that you need um, there are other packages that you need to install for either network layers or different kinds of resources that you want to kind of mount onto that kubernetes cluster or you know, have services like lower balances. There are a bunch of different proxies and sidecars that you might want. So there’s just so many things that usually go into that that kubernetes ecosystem, it’s never just, oh, I started kubernetes and I’m ready to go, you need a lot of other stuff that goes with it and with different versions and different variants of how kubernetes gets deployed, it’s hard to get stuff working right from the get go, Like you have to be a master at tinkering around with all those packages and the way the installations happen, there are different mechanism of installing these things, so you really need to be a master at being able to do all of that and that’s hard. What interviews has done here is they’ve curated a set of packages for you and I’m sure that this list is going to keep growing um you know, over a period of time, but the ones that they have just installed out of the box, they could click you say, I want this deployed, it’s very easy to make it really easy to get all this stuff working without having to be an expert at kubernetes Jamel files dealing with these particular packages.

Rahul Subramaniam

I think that’s absolutely right. About 90% of workloads today are still on premise and there is a shift towards the cloud, but that’s again gonna be gradual, this people, there’s a lot of hurdles to overcome. So the more services that allow you to start at the edge and then quickly bring you onto the cloud are always welcome. Let’s help more about yours. Perfect.

Rahul Subramaniam

Yeah, thanks,

Rahul Subramaniam

Likewise, see all last week by All right, thanks