AWS Cloud Security: How it Started, Where it’s Headed

Clarke Rodgers (00:05):
The larger your business gets, the more complex your security concerns become. Building a truly secure business as it scales actually involves simplifying the operating model in order to clarify the priorities and focus of your team. I'm Clarke Rogers, director of Enterprise Strategy at aws and your guide for a series of conversations with AWS security leaders. Here on Executive Insights. Today we're talking with CJ Moses, Chief Information Security Officer, and Vice President of Security Engineering. We hope this conversation provides useful insights for you and your team as you continue your own cloud transformation journey. Thanks for joining us.

Clarke Rodgers (00:54):
CJ, thanks for joining me today.

CJ Moses (00:55):
Thanks for having me.

Clarke Rodgers (00:56):
Would love to hear a little bit more about your background and what brought you to AWS. I know you've been with us for quite some time. But what got you interested in AWS and what brought you in the door?

CJ Moses (01:06):
It's been just about 15 years, or coming up on that. Myself and a few others that are here at Amazon, were working at the FBI. We had a mission that required us to do big data before there was big data. Essentially it was in support of counter-terrorism. Every piece of digital media the US government got, we got a copy of it. We got a copy of it in order to cross correlate it against everything else we knew, in order to figure out and to find that needle in the needle stack that was going to keep bad things from happening to good people. That mission is the definition of what would keep you up at night. Because if you'd fail to do that, people could truly die.

We spent a long time building lights out, fully virtualized data centers with state-of-the-art everything that we knew or could buy. In looking for the newer technologies and things like that, a few people went out and talked to different companies. One of which, EC2, had just been launched. Literally was brand new. Elastic Compute Cloud. That sounded like something that we really would want to have. Or the idea of using a thousand computers for an hour versus one computer for a thousand hours, time to mission, time to keeping bad things from happening to good people, much, much different model.

So, we took the opportunity. Jeff Barr, our chief evangelist these days, our only evangelist back in those days, had an open calendar set-up. A few of our people decided they were going to get on his schedule when he was coming to DC and talk to him about things. And we did. That conversation led to, ultimately, realizing that ... Because EC2 was brand new, the user ID and password that you used to buy books at the bookstore was the same thing you logged into EC2 with, or AWS as a whole at the time. Not a security model that the US government was going to sign on for counter-terrorism information. That's 15 years ago.

But that discussion, although it took six months or so for us to come around from being the customer that had the need, to being potentially employees that were going to try to fulfill that need, took us six or eight months, but in the end, December of 2007, through a little prodding from Jeff, as well as up to Andy, that the only way we're ever going to get to that position is to have people like yourself, say, come on and show us, and assist building the enterprise in a way that's going to meet those types of needs, which will also make us enterprise ready. That discussion turned into a trip out here to Seattle, where we interviewed. We thought we were coming out to discuss and it turned into interviews.

Clarke Rodgers (03:51):
Surprise.

CJ Moses (03:51):
Yeah, surprise. They were interviews as we normally have at Amazon. A couple weeks later, three of us started. Dragged beanbag chairs into a data center in Virginia. We started on our mission of trying to build the infrastructure that we always wanted to have. Not only for US government or otherwise, but just after years of being a computer crime investigator in my career, I'd always had to deal with the issues of the internet.

To now have the opportunity at a company to start from scratch, essentially, and to build an infrastructure, to build an environment that you could operate in, that would be truly secure from the core, was an opportunity we couldn't turn down. We were like, "Okay, this is what we've always been waiting for. No more wild wild west. We're going to turn it into something that we can actually do. Or we can actually manage and have security, true security in." Not security theater, which is a lot of what you see these days out there on the internet.

We didn't come in as a security team, myself and a few others, including our Chief Security Officer at Amazon, Steve Schmidt. We came in and we ultimately created, after we figured out what we were really here for. But we, from there, figured out, "What is it that would make enterprises feel more comfortable with us?" AWS that is.

Clarke Rodgers (05:16):
Sure.

CJ Moses (05:16):
We looked back to our previous mission, and said, "Okay, if we were in our past shoes, we have that experience, that's why they really kind of want us here, what are the things that we could do to make government entities, enterprises as a whole, your most highly regulated enterprises in finance and others, healthcare, feel comfortable being able to operate in a cloud?"

Natural distrust in the intelligence community is a good thing. It's not trust, but verify, it's verify, then trust.

Clarke Rodgers (05:43):
Sure.

CJ Moses (05:44):
The environment that we had, we had to figure out a way to try to get them to better trust. So, we created logically isolated networks using VLANs and things like that, that had the theater perspective of being secure, but we knew, at scale or under heavy attack, wouldn't work in this environment. That was the beginnings for what we ultimately became — the Virtual Private Cloud Team, creating the novice overlay network onto commodity networking equipment. That allowed us then to create Virtual Private Cloud. To do logical isolation within the cloud, provide that capability out for all of our customers.

Early days, we were bolting stuff together. At first, we were trying to do it on pre-existing protocols and things like that, It just wouldn't work, it was all bolt-ons. So, we took that greenfield opportunity that we were given and said, "Okay, we're going to start from scratch." Which even at Amazon was kind of a lofty goal. "What do you mean you're not going to run normal networking protocols? You're going to create your own and then you're going to operate the web fleet for amazon.com on it at some point?" One of our leadership principles is “Think Big.” And we did.

Clarke Rodgers (07:02):
That's pretty big. Yeah.

CJ Moses (07:05):
That was 2008. By 2009, VPC was launched. By fourth quarter of 2010, the entire amazon.com web fleet was running on Virtual Private Cloud infrastructure, logically isolated from each and every node, which allowed us to scale test, truly, how VPC would operate. Learned things in that process, but didn't have any failures that took us off the air, thank God. Jeff would've been upset. But then iterate just like normal. Continue to work backwards from the customer. I know it's a long intro, but that's kind of the story. There's more to it, believe it or not.

Clarke Rodgers (07:41):
One thing that stuck out to me in that was that you were not hired as a security team. I'm going to assume you were hired as an engineering infrastructure type team.

CJ Moses (07:51):
Yeah.

Clarke Rodgers (07:52):
That really makes sense today because, when we look at the larger AWS internal organization, you run AWS security, and then there's multiple engineering and service teams that are out there. One of the principles that we operate under is security, for the most part, is an expertise organization, and an observe and report organization, while the engineering and development teams are the sort of the doers and the patchers. So, because you came in so early, were you able to set that culture then, that engineering owns security and owns the security of their products?

CJ Moses (08:33):
Our culture at Amazon as a whole is one of ownership. So that “Ownership” leadership principle and the culture actually sets the bar or sets the base for security teams owning the security of their services.

Going back to the early days, we were a Virtual Private Cloud service team, so we were in that same boat. We didn't have an AWS security. We had an Amazon-wide security engineering organization. It wasn't even a security org per se, it was security engineering. So, they would help to engineer things, if you brought it to them, and kind of set some policies and things.

But it became very clear when we were getting ready to launch Virtual Private Cloud, and then subsequently did, that customers would have a lot of questions about how we did security, and that we would need to be able to articulate that, as you would expect customers would want to, especially fullness of time, US government or international governments wanting to ask questions. They have really deep questions.

The “got you” was that security organization that Amazon had was very focused on the Amazon retail business, the consumer, and now stores business. That is quite different than having AWS, which is service oriented, selling to customers. So, we kind of needed a change there. It was clear that the mentality and the model for Amazon security was one that we didn't talk about security externally hardly at all, intentionally.

Clarke Rodgers (09:58):
Sure.

CJ Moses (09: 59):
We did our things and it was very strong, but it wasn't one that we were going to talk about. In AWS, we needed to have that, and we need to be enablers to the business. Coming from running a service team, knowing the things that we needed, and what the customer questions were. And then subsequently in 2010-ish, thereabouts, I think AWS security was established. Then establishing the security team, because of those needs from customer perspective and service team needs, allowed us to build that infrastructure and the culture from scratch.

It was Andy Jassy who decided we need to have our own security team. It wasn't Steve or I. Quite honestly, when it was first determined, Steve nor I were really in the running or had talked about doing it. We were running Virtual Private Cloud, had just got past that growth curve of just starting, getting the startup going and everything moving well. We helped interview for a potential CISO to sit out here in Seattle with Andy. We interviewed and really didn't find anybody that kind of fit our company culture, as well as knew what we needed to build.

We needed a security team that was an enabler to the business…

Clarke Rodgers (11:04):
Right.

CJ Moses (11:05):
Not one that was a land of “no.” Steve spent a lot of time out here, as did many of us, getting things set up. But in the end, the reason why we have the kind of model that we have today is because we started out as a service team, understood the needs and the friction — because quite honestly, there is friction. But at the same time, we needed to be, as a security team, needed to be an enabler to the business, and to make sure that the service team owners understood that the ownership model of success and failure, profit and loss — security was included within that.

Taking that kind of forward to, "Okay, working back from the customer, I had a paradigm of technology that I was limited to, the vendors that would work with me. Now I have all of, what I called at the time, "This alien technology within Amazon,” that allows us to be able to do things that we never thought we could do before. And the innovation mindset and culture that we ran into, that opened us up. Opened up to think Greenfield, and say "Okay, if you're able to rebuild the internet and have it within your walled garden, to some extent, what would you do?" Well, first, we get rid of the walled garden because we don't want it to be a walled garden. We want it to be right there on the internet.

Clarke Rodgers (12:15):
Right.

CJ Moses (12:15):
This gets into the “unfabric” or the idea of data-centric security, not walled gardens of networks; moving your way towards that. That started many, many years ago when we first joined. And that's where the technologies that we implemented from VPC kind of forward have allowed us to continue that. That innovation and the builder culture, in security as well as just in everyday life, is I think one of the things that I look at as enabling Amazon to continue innovating the way that it has for all these years. Normally you see companies that innovate for a while and then they get caught up with all of the keeping-the-lights-on work.

Clarke Rodgers (12:57):
Yep.

CJ Moses (12:59):
That is a challenge for everyone. It's a challenge for us as well. But the innovative spirit in leadership principle/ownership model allows us to continue to innovate on behalf of customers, working backwards from them.  

Clarke Rodgers (13:13):
So let's pull on that humanity of security for a little bit. Every customer I talk to is, "I can't find qualified security people." Or, "I can't afford them," or something like that. "I want my security team larger." The good thing is they realized there's a need for stronger security and better security at their organizations. Can you talk a little bit about what you look for in security hires?

And just to set the stage, AWS security, you're responsible not only for traditional security functions, but there's also compliance functions and regulatory functions. All sorts of different things that fall under that umbrella. What are you looking for for professionals coming into your org?

CJ Moses (13:58):
So the number one thing that we look for — obviously, security needs a lot of different types of people doing a lot of different jobs and responsibilities. But when a lot of companies are focused on hiring security engineers or people that have security expertise, I think our focus has been not as much hiring or focused only on security engineers, it's focused on builders. People that can find things that we're doing that need to be automated and automating them.

Those builders can learn security or do naturally. Especially with, now, we're at a point where we have a lot of security engineers. Having the security engineers that we have, that our builders help to mentor and bring forward other builders is really important. Because from the standpoint of if you hire only security engineers that don't have the builder capability, you're going to need to hire many more security engineers because you're going to manually scale.

Clarke Rodgers (14:57):
Right.

CJ Moses (14:58):
If you hire builders, AKA software development engineers, or security engineers that can code, to some extent, in those cases. Not only code, but other ways of mechanizing things. Then you don't need to hire as many going forward. Because they're going to take the opportunity, when they see things that are repetitive, that are boring to them, engineers don't like to redo the same thing many times, they're going to create the scripts. They're going to create the tooling. They're going to create the services that will make a path of least resistance, again, make security easier for them. And you won't need to hire 10 people to go through logs because they've created a logging infrastructure that is able to be reviewed quickly, and at scale, and pull out the right pieces of information. And then have one human look over the logs for the things that the system has told them is important based upon their own parameters that they put into it, versus the 10 people that would have to just fumble through logs trying to find, by luck, the things using grep. That's not the way to scale.

So, the number one thing is hire builders that can help you automate within that space. Build on the expertise that you have doing things like that. I think we also ... the opportunity exists nowadays, looking at the adversaries that we face, the adversaries we face are very diverse. They're diverse on so many different levels. Everything from where they come from, cultures, backgrounds, or otherwise, technologically. In AWS, we're defending against everything from script kiddies from around the world, all the way to nation state actors that are the actual spies. That is where, when you're hiring, in our case, we want to create an environment of inclusion. What I mean by inclusion is having that team esprit de corps, if you will, coming from my military days. Where the team is all focused on one goal, in this case, defending and securing AWS and our customers.

But then take that even a step further. Once you have that kind of team environment, make sure that team environment is open to all builders, to all of our security people. You can go down the whole list of ... It's not just a security engineer focused culture, or it's not just this type of person focus, it's open and inclusive of everyone. By doing that, guess what you've just done? You've opened up the aperture for those that you can hire, or that you should hire.

Clarke Rodgers (17:28):
Right.

CJ Moses (17:29):
Because your adversaries are very diverse, and that normally begets a little bit of a diverse thinking and thought process. If you are defending against those other people to begin with, you should have people that represent and think that way in order to create that.

I think we get lost a lot of times in the security space, in a lot of spaces in the technology world, thinking that computers are attacking the computers.

Clarke Rodgers (17:51):
Right.

CJ Moses (17:52):
It's not the case. The computers are the new tool, using humans to attack other humans. How are they attacking them? They're attacking them through the computers to get their money. Theft has been here since there's been money or anything someone else wanted that you had. Those are the types of things that we have to think of. Bringing that diversity of thought into play, hiring from a broader swath of people to begin with, allows you to have that diversity of thinking. That allows us to be better at protecting our customers data, and ultimately, creates a better, inclusive environment for us all to work in.

Clarke Rodgers (18:31):
That's awesome. It's well known, you're a veteran, I'm a veteran. We've both done very well here at AWS. Sort of taking that military background and bringing it to the AWS security apparatus, for lack of a better word. How do you see veterans contributing to not only the AWS security team, but the larger AWS footprint?

CJ Moses (18:58):
No, absolutely. I mean, years ago, Amazon-wide, they committed to hiring 25,000 veterans within a certain period of time. That turned into 25,000 a year. Without even having that specific goal, it's more than that these days. I have been very fortunate with military background to “Forest Gump” my way into various roles over these days. That's my career plan, "Follow Forrest Gump. He seemed to work it out."

One of the things the military does is teaches people how to think in adverse situations. In security, that's a lot of what you need. You need people to remain calm, to not be worrying about those things they can't control, but being preparing for those things they can. That's what the military teaches you. Going back to my past life — the FBI, prior to that, Air Force OSI — I told you that the things that really kept me up at night was the fact that people were going to die on my watch.

Clarke Rodgers (19:57):
That sense of mission. Yep.

CJ Moses (19:57):
Yeah, that sense of mission. Normally, that may not be the case today, but that also sets me up to be able to deal with the things we do have. As well as those that are on the AWS security team, or on AWS as a whole, or Amazon as a whole. Hiring from an environment where we already know that that training and that mindset is already there, allows those people to fit into our culture, which happens to align very well with a lot of military cultures.

Clarke Rodgers (20:22):
Right.

CJ Moses (20:23):
I know in AWS security, it seems to be the continuing thing. It's not because I'm forcing that. It's, the people that have that experience fit in very well. I don't mean fit in from a, "Yes, boss," type of mentality. Quite the opposite. It's having the experience and the cool-under-pressure capability to be able to move the ball forward, and protect our customers and not get frazzled. Because there is a zero day, like a Log4j type of issue. Maintaining calm under pressure and getting it done.

We continue to recruit from military space. We work with the Amazon Warriors program. I'm an amateur race car driver. My race team, we actually work with Operation Motor Sports, which is a nonprofit that's designed to take medically retiring veterans from the military and help them transition by making them part of the race team.

Clarke Rodgers (21:23):
Oh, wow.

CJ Moses (21:23):
So, from a mentality perspective, they go from one team to another. Helps them transition. It's been really good for me, from the standpoint of being able to give back and make sure people can transition.

I had a weird transition from Air Force to FBI to Amazon. Whereas in some cases, people retire from the military and they're like, "What's next? Where's my team?" And if you don't have that team, you kind of have a loss of self-worth. Adding people to the race team, and also making them integral parts to championships and things like that, like we've done, helps to build that personal confidence back in themselves. Every year, as we have new people come through, new veterans, it's more ... I feel personally that I get more benefit out of it than they do. But they're very thankful in moving on to be able to do things.

Clarke Rodgers (22:19):
Awesome. Well, CJ, thank you so much for your time today. I really appreciate your insights.

CJ Moses (22:23):
Oh, thank you.