Who Owns What? Security Ownership and Responsibility at AWS

Clarke Rodgers (00:08):
Customers today are trying to figure out the best way to balance speed and security. Sharing security ownership across the business may be the answer. I’m Clarke Rodgers, Director of Enterprise Strategy at AWS and your guide for a series of conversations with AWS security leaders here on Executive Insights.

Today we’re talking with Laura Grit, Vice President and Distinguished Engineer at AWS. As an AWS leader with an engineering background, Laura understands how excellence in engineering contributes to better security and optimal business outcomes.

We hope you enjoy.

Clarke Rodgers (00:55):
Laura, thanks so much for joining us today.

Laura Grit (00:58):
Yeah, thank you for having me.

Clarke Rodgers (00:59):
If you'd be so kind, could you tell us a little bit about your background and what brought you to AWS?

Laura Grit (01:04):
Sure. I joined Amazon almost 15 years ago after getting my PhD in computer science. And I actually joined on the store side of Amazon. One of the early things that I did was I ran Amazon's adoption of AWS-

Clarke Rodgers (01:18):
Oh wow.

Laura Grit (01:19):
Around the early days of AWS, and so got to experience what our customers experience when we look at cloud migrations from the Amazon perspective. From there, I joined our infrastructure organization. I worked on network scaling and availability. And then several years ago I came over to the services part of AWS.

I am a Distinguished Engineer, which is an individual contributor at the Vice President level.

Clarke Rodgers (01:45):
Okay.

Laura Grit (01:46):
This is the senior most engineering role that we have at Amazon and there's actually only a handful of us. And, as a result, all of us are very unique in the different skills and what we bring to our role. Something that we all have in common is we are all looking across the company to see how we can influence and drive the technical roadmap based on what we observe from working with different teams across the board, and ultimately, we're looking at how we can best deliver across the company for our customers.

Clarke Rodgers (02:19):
Got it. So interestingly, I didn't hear security anywhere in your title, and this is the AWS Security Leaders video series.

Laura Grit (02:28):
Sure.

Clarke Rodgers (02:29):
And the reason you're here today is to really talk about how we spread security responsibility out throughout the organization. So, you're not a security badge person yet you do have quite a bit of security responsibility. Could you talk a little bit about that and how that works at AWS?

Laura Grit (02:48):
Happy to. Security is our number one priority, and so as a Distinguished Engineer, I am working with security and ensuring that we are secure for our customers. I do a lot in collaboration with our security organization, especially as we look at the types of programs and other work that we do that cross our engineering teams.

One of the things that we do at Amazon is our engineering teams act very distributed. They prioritize their own roadmaps based on the needs of their customers. And so there's work that we do that really spreads across all of our service teams. And so I partner with security and the different services that we have for how do we optimize that work, how do we automate where possible? And so work collaboratively between the different organizations.

Clarke Rodgers (03:44):
Got it. So, if we take a service for example, how would you influence their adoption of a certain security principle or maybe an engineering best practice that you've seen in another service for example?

Laura Grit (04:00):
One of the things that we do is security is part of all aspects of our development. And so, while we have a separate security organization for AWS, the security engineers are partnering with our software development engineers from the design and architecture to put those security best practices into the design.

So, as we learn about best practices from another team, that then is helped communicated by our security organization and the other documentation they have for how they build. And because they're partners as we're developing, as we're reviewing designs, that then is able to get from the get-go, as opposed to having to do it at the very end when the feature is ready to go.

All of our features and services go through a security review as process of part of launch, and then as we're operating our services as well, we are partnering with security for that. Ultimately, it is our services teams that own the security of their service. We don't hand that responsibility off to the security organization — instead they're partners with us.

Clarke Rodgers (05:10):
That's fantastic, that's fantastic. So culturally, I speak to a lot of different types of customers and different industries and there's often a conflict or almost headbutting between the security team and the engineering or the development teams.

It's relatively easy for a security person to understand how important security is for the overall health of the business, so to speak. But oftentimes I run into customers where the developer, the development staff is focused on, "Well, I need to get this feature out the door, that’s why you hired me."

What is done at AWS, and perhaps the larger Amazon to make sure that there's a certain level of ownership for that developer around the securitization of that particular service or feature that they're working on?

Laura Grit (06:03):
Yeah, great question. This really starts with our CEO and talking about security as our top priority in all-hands, in other communication that they're doing. This also feeds then into our Vice Presidents, how they're talking to their organization. Part of our onboarding training talks about the importance of security, why it's such a high priority for us. And this crosses all roles, which I think is another important thing. So it's not just our engineers that are getting that message, it's our product managers, it's our managers as well that this is all of our job and it's not just the job of security, but it's the job of all of us to ensure that our services are secure.

Clarke Rodgers (06:48):
Got it. So again, if I'm a developer and I'm putting something through the CSE pipeline to get to that next stage and I get something kicked back because it didn't meet a certain security standard, I imagine it's viewed more as a learning opportunity and an opportunity to make the software better versus, “Clarke, you’re a terrible developer and can't secure.”

Laura Grit (07:12):
Correct, and this is really where the partnership comes in. Having security be part of our processes early on in development and not just late helps prevent some of those last-minute surprises. But also, as we're going through when we are scoping or planning the work of how long it's going to take, we put in that time to be able to remediate anything that security finds as they're going through the review before we launch. And so part of that is also just upfront planning to be able to put that in so that reduces some of that tension that I think others can have of really pushing something to get to launch when you have that as part of an expectation of what it means to release a feature.

Clarke Rodgers (07:58):
So, in that model, how are conflicts managed when a feature's supposed to go out tomorrow and then a security vulnerability has been found that maybe takes two weeks to fix? How does that conversation happen within the service teams?

Laura Grit (08:14):
For us, that isn't a conflict. That is a conversation, that is figuring out what we need to do to do the right thing for our customers. And if we find a security vulnerability that needs to be addressed — it needs to be addressed. And we then have that conversation with the executives, whoever needs to be involved to explain what is happening, what is the risk and why that's being pushed out. So, part of security being that top priority is being able to have these conversations at all levels as we get surprised or as things need to be prioritized.

Clarke Rodgers (08:49):
And then the executives get it, the product owners get it and it's again, it's not a fist fight at the last minute.

Laura Grit (08:54):
Correct. We are engaged, we know that this is an important thing for our products to be secure. And so it really becomes part of that definition of what it means to release a feature or service.

Clarke Rodgers (09:07):
So Laura, as you know, I meet with a lot of customers throughout their digital transformation journey. Some are starting off and they're just pointing and clicking in the AWS Console. Some have fully built out CIC pipelines and don't even know how to spell “console” right, because they're hardly ever in it.

Integrating security and sort of a strong engineering culture can be a challenge depending on where you are in those different stages. If you look at the sort of “crawl, walk, run, and sprint” of cloud adoption, do you have any sort of tips or tricks from an organizational and management perspective to sort of drive these behaviors that you're looking for?

Laura Grit (09:46):
One of the things that I did early on in my time at Amazon was work on Amazon's adoption of AWS. And I've continued to be part of many migrations that we've done internally. And something that we do that's been really helpful is when we are looking at a migration or transformation, we take a few different types of use cases.

A couple are ones that we think are going to be easier, ones that we can get early wins to show success. And then another one we'll choose is one of the hardest ones that we think we need to accomplish. And so, we were able then to go to the other teams that felt they had a hard use case and say, “but we got them over the line.” And then that showed them, “Oh, well we can't just say that we're too hard because you actually got one of the hardest use cases over the line.”

Clarke Rodgers (10:38):
Sure.

Laura Grit (10:39):
But then we also worked with some of those ones that were more straightforward so that we could get early wins and publicize what we were doing into the company of, "Hey, check this out, this is really cool, anyone want to partner with us?" And really kind of build that momentum. This helped us figure out what tooling, documentation, those types of things we needed to do while we were also working on the harder transformation.

Another thing that I really focus on is how I can get engineering engagement and how I can rally our software developers to get excited about the work that we're doing. And part of this is doing talks or having good whitepapers or other conversations, whatever your company uses for that type of communication.

I've really found a lot of great folks across the company that want to partner with what I'm doing, either because they're really excited about it or because they know that they could really help shape what we're doing and they want to be part of that shaping. And so finding ways to find those people throughout the company can also really help you because that gets the word out and they can really help spread what's happening better than a single person or a team of people might be able to do that are more centralized.

Clarke Rodgers (11:55):
So, almost having guilds of people in particular areas of expertise and they just help of coach each other it sounds like, right?

Laura Grit (12:02):
Yeah, because they are working on their particular service, which spread across the company, but they'll provide feedback or "Hey, we're thinking about doing this” or “This is a challenge that we're running into." And having those folks that are sounding boards that provide their insights of what that would mean for their service or what that would mean for their space can really also help accelerate and ensure that you are building the right thing for a larger set of services than you might be the couple early adopters that you're working closely with.

Clarke Rodgers (12:34):
mSo, in that sort of transition phase, for lack of a better word, where do you start introducing that security early in the development cycle?

Laura Grit (12:45):
What I have found with the work that I do is it is incredibly important to engage security early and to work with them as partners on that transformation. Making sure that you have senior security engineers as part of that core working group is really important, and because they can then look at things from that angle, which might be different than your particular perspective, and help you not have to make changes after the fact, and instead build those security best practices in from day one can really then help you accelerate and also not have to run another migration or something later because you didn't engage them early in the process.

Clarke Rodgers (13:24):
So, as customers are trying to build out their security organization and a strong engineering culture as well, how can strong engineering actually support the security team or the overall security mission? How does being a strong engineer, knowing everything about the libraries I'm using and things like that, how does that benefit the overall security mission of the company?

Laura Grit (13:49):
Having engineers that are thinking security from the start and how we build our tools and how we build our processes encompassing security is really important for us. Part of it is bringing in security engineers themselves into our conversations as we're building our services, as we're architecting, and also as we're building tooling, one of the things that we look for is ways that we can do secure practices such as OS patching, software patching automatically where possible.

And so, we've built a couple systems internally that are able to do that automatic patching. We also have tooling that allows us to know what software is being used in different locations. This then helps our engineers know what they need to patch, what they have on their box, what that looks like, and helps them have all of the information that they need to do the security work for their service.

Despite having this de-centralized structure, we do use a common code base, common deployment tools, common ticketing system, common dashboards. And that then enables us to be able to communicate in a common way to the service teams. And that's a big part of our partnership that we have with the security organization.

So they are looking to provide us those dashboards, that visibility so that our engineering teams can then go to those dashboards and see what they need to do. For example, understand their patching status. This is the work that is required in order to maintain the security of our services.

Clarke Rodgers (15:32):
So then it sounds like that discipline around what we're building, how we're building it, what we're using to build it helps when a new vulnerability comes out. You're not wasting a lot of time with, “Do we even use this piece of software?”

Laura Grit (15:45):
Exactly.

Clarke Rodgers (15:46):
It can just be boom, “Of course we do. We use this version and this rev and we know whether we need to patch it or not.”

Laura Grit (15:52):
And what host it's on and therefore what service is using it. And we know how to contact that service to get to the engineers who will then take action on that.

Clarke Rodgers (16:02):
So, it's the discipline aspect of the engineering that helps the overall security mission?

Laura Grit (16:07):
Exactly. And really looking for those opportunities to continue to provide good insights and quality data to our service teams, and then also ultimately tooling to help it be easy to do this work.

Clarke Rodgers (16:20):
That's awesome. So, let's sort of shift to a real-world example. So, the world got to experience Log4J and Log4Shell and all of the “fun” that went along with that.

Can you sort of walk us through how that was handled at AWS? From a responsibility perspective since we've touched on that a lot today, what did, let's just say AWS Security do and then what did the service teams do to address that issue?

Laura Grit (16:50):
Log4J, due to the nature of the vulnerability that it didn't require sophistication in order to exploit, was a very, very high priority for us to remediate as quickly as possible. And so, we put additional mechanisms in place to get the right prioritization on this.

What happened is the initial engagement involved some of our most senior engineers, most senior folks in security, talking to each other and then did engagement with our Senior Vice Presidents and CEO around what this was, education.

Clarke Rodgers (17:32):
So this is not just a run-of-the-mill vulnerability.

Laura Grit (17:35):
Correct. This is different. We need prioritization, this is what it means. And that created that top-most alignment at the highest levels in Amazon for what needed to happen.

Clarke Rodgers (17:45):
Okay.

Laura Grit (17:46):
At the same time, they also engaged our senior engineers because those are the folks that are going to be working with our engineers and our service teams to really make sure the right prioritization was happening, while at the same time we're talking to the executives around alignment. So, that we found to be really important to not just talk to management, but also to engage our senior engineers in the conversation as well.

The other thing we did was engage the central tooling we have. So we have host scans that help us know what packages are deployed to what host, and also we know how to contact the team that owns and operates that service.

Clarke Rodgers (18:26):
Okay.

Laura Grit (18:27):
And so, we were able to leverage that central tooling that we use for other patching mechanisms in order to cut high severity tickets through this common ticketing system to our engineers.

One of the other things that we have found over time that really is important when we have something that requires engagement from this many teams is we first have some engineers run through the documentation and instructions that we're going to be sending out. Because you want to get those tickets out as soon as you can, right? But taking that extra half hour, that extra hour to really review and ensure that it's clear can help save some of the questions, confusion, and ultimately get things out faster even though it feels upfront like you're going a little bit slower.

Clarke Rodgers (19:13):
So, the work doesn't start until that engineer receives the ticket. So you have to make sure that the ticket itself is as clear as possible so there's no back and forth.

Laura Grit (19:22):
Exactly. You want to make sure that the engineer that receives the ticket has all of the information that they need. The other thing we did is we set up chat rooms as well as calls that our engineers could join 24/7 to be able to get the information they needed to remain unblocked.

We're a global company and so it's important that we had these opportunities for conversation, for engagement, for clarification, available to engineers all of the time that they're working. We also continued to engage with our Vice Presidents, with our General Managers around how their teams were progressing. And so we had regular calls with them throughout the day, being able to update them as to where we were so that they could then enable that driving of the patching within their service teams.

So really helping both our managers and our engineers get access to as much information as we could and being very clear and crisp. We also then had status reporting up to the Senior Vice President/CEO level so that we kept them informed as to what our status was as well.

Clarke Rodgers (20:28):
So, that's pretty incredible. So, it sounds like there was a strong partnership between AWS Security and all the service teams and the engineering teams to actually get this done. And it kind of sounds like the actual security team was more of an “observe and report the status of the Log4J fixes,” while the engineering and development teams were actually doing the “work” of patching, et cetera.

Laura Grit (20:58):
Exactly. Log4J was an example of the great partnership that we have between the services part of AWS and the security organization. We really were in all of this together. So the conversations and the calls that we had with our executives were being driven by both the security organization and the services team so that we were reporting where we were and addressing anything that we needed their help with prioritization or things like that.

We were getting our status reports, the data from the security organization, partnering with them on the tickets that needed to get cut, the documentation. And then really on the services side, doing the deployment, doing the actual patching, and making sure that our software development engineers had what they needed. So, a great example of us coming together in order to really make this change very rapidly.

Clarke Rodgers (21:48):
Yeah, so that's great to hear because a lot of the customers I speak to, they think, “Well, in order to have a strong security organization, we need to 10 x the population on the security team.” When, in fact, you have your extended security team in your engineering organization. It's just making sure that the partnership and the culture supports all that stuff.

Laura Grit (22:05):
Exactly. Really creating that partnership — really helping engineers, managers, product managers understand how critical security is for your business — really helps create that joint work that we're able to do together.

Clarke Rodgers (22:20):
Well Laura, thanks so much for spending some time with me today.

Laura Grit (22:22):
Thank you so much for inviting me. It was great chatting.