No Such Thing as “Done”: The Role of Security in Product Development

Clarke Rodgers (00:01):
Customers today are seeking opportunities to be more intentional about their security culture — how to build it, how to measure it, how to adapt it to their unique business. I’m Clarke Rodgers, Director of Enterprise Strategy at AWS and your guide for a series of conversations with AWS security leaders here on Executive Insights.

Today, we’re talking with Bill Shinn, Senior Principal in the Office of the CISO at AWS. Bill is an important voice for AWS customers, advocating on their behalf to ensure that the right questions are being addressed when it comes to security.

We hope this conversation provides useful insights for you and your team as you continue your own cloud transformation journey. Thanks for joining us.

Clarke Rodgers (00:51):
Bill, thanks so much for joining me today.

Bill Shinn (00:53):
Sure. Anytime. Good to see you again.

Clarke Rodgers (00:55):
Nice to see you. So tell me a little bit more about your background and how you came to AWS and what your current role is?

Bill Shinn (01:04):
Sure. Yeah. It's 10 years next week.

Clarke Rodgers (01:06):
Oh, wow.

Bill Shinn (01:06):
I came here from banking mostly. About 12 years in pretty large banks, some are now customers as well. I was the security person for this pretty significant financial institution. And I recall now that security wasn't necessarily a rigorous professional discipline. I mean, there were still education and training and certifications a little bit, but it was pretty early in the industry. So I just got to do everything. I was dealing with the NCUA audits, federal audits, security awareness training, running the firewalls and switches and all the log management, threat detection, all the things.

And security was a little wild west back then, I think. There were things that we didn't even know potentially could happen. So, I learned a lot on that job and then I moved into larger banks, progressively. And then a bunch of people I've worked with over the years in various capacities at various jobs, a couple of them landed at AWS and said "This is pretty interesting." And at the time I worked at a pretty large financial institution and it was really challenging to get security work done because of just the procurement model, mostly, and the long tail of capital expenditure. So, you buy hardware for three years, it depreciates, so you have to make a very, very rigorous buying decision. There wasn't really a lot of opportunity for experimentation. There was very little room for failure because you're buying millions of dollars of expense and if you get it wrong, you're kind of stuck with it.

The other part of that was security, the landscape changes so quickly. The threat landscape, the regulatory landscape, and the business you're supporting or the mission. And you have to stay current, you have to stay on the most modern ways of approaching security problems. And so, if you do that with three-year-old hardware, there's software updates, but you're still kind of stuck with where you are when you buy it.

And so I saw cloud, particularly around log management and storage, for retention, for analytics — there was a lot of potential. I think AWS may have had MapReduce or something similar to it, or at least could run a dupe on EC2. So I said, "Well, maybe we should do this on the cloud." And that was not really going to happen with legal and procurement, and AWS wasn't where it is today. So, I joined the cloud and said, "Hey, let's bring US financial system into a place where they can modernize safely. They can experiment, they can benefit from the innovation that the cloud provider AWS provides."

And with that goal, in around 2012, we took a lot to get the financial services customers to adopt the cloud. We needed to change the way we did business. We needed to understand their needs more deeply. And I had the background, I think, to help AWS do that.

And so, to be closer to that work, I moved into AWS Security to really understand our security operations processes, our application security processes, how we think about culture and enforcing security properties and getting people to want to do security right — it is the top priority here. And I've seen a lot of mechanisms and observed a lot of things now having spent about five or six years inside of AWS Security that I think our customers want to hear about. And they wanted to hear about it to trust us. But I think the conversation significantly changed at about the five year mark. It just became much less about trusting the cloud. Although that's still I think first principles, but we have so many assurance programs. We've done so much work to be really clinical and precise about how we secure the cloud with our customers that now the conversation is very much about, “How do I succeed on the cloud with my security program and how do I benefit from the cloud?”

Clarke Rodgers (04:29):
And I would agree with you a hundred percent that even in the time I've been in AWS — not quite as long as you have — the initial conversations were about, “Is the cloud secure?” And now the conversations are, “How fast can I adopt it?”

So, you mentioned trends in the industry, from cloud adoption and I guess specifically with the security risk or compliance bent, what are you seeing? What are the customers coming to you these days, more often than not...?

Bill Shinn (05:00):
You know, it always kind of goes back to the basics, I think. How do you do the same things better and how do you innovate in incremental ways? Ransomware and other trends that are in the media — it's really hygiene. It’s typically an opportunity for better security awareness, which is usually the vector that ransomware…it’s kind of a buzzword trend.

Clarke Rodgers (05:20):
Sure.

Bill Shinn (05:21):
It's a combination of security awareness opportunities to improve, as well as hygiene for patching systems and detective controls that would help you identify issues. Those fundamentals really haven't changed. I think the speed has always been a big part of the conversation. How do you move faster as a security program without compromising your risk management decisions and without making trade-offs, but keeping the business moving? That is a constant trend that customers want to hear about.

Helping customers move fast, I think part of it is having security in the sprint planning, having them at the table in design decisions, distributed security as well. I think you'll hear a lot of Amazonians talk about that, where a security team can't own all the security. Our top priority is security and it is everyone's job. And there's mechanisms here that make sure that's true, always.

Most of the service teams have security engineers or security guardians at least as they grow because they know that they can't continue to ship code if they're distracted by security issues. And so, we want to find everything we possibly can before we ship a service. But it's not a project. You're not starting and finishing. You own that service once it's live. And you need to plan for how you're going to maintain it, how you're going to patch it, how you're going to deal with emerging vulnerabilities and defects. And you have to be able to do it quickly.

Bill Shinn (06:43):
I find a lot of times the blocker for customers is not a lack of skill or a lack of technology. It's often the financial structures and the staffing models that are needed to move to a modern way of doing security along with a DevOps and Agile and continuous integration — lots of deployments every day — that you kind of have to plan for the unplanned.

Clarke Rodgers (07:06):
Ok.

Bill Shinn (07:06):
And so, how do you justify headcount in a service that's going to last forever, right? So as the service grows, it becomes more complicated and you need to do more hygiene, you need to retire technical debt. And it's not just always “build new features, build new features.” It's make it more robust. Make it more resilient. Find ways to improve the efficiency and the cost of that. But also, how do you keep the hygiene? Because security atrophies over time and defects are discovered. Ciphers break, security researchers discover things that are not always in your control — it could be an open source package or something.

So you have to have a way, and staff, and you have to plan for that. And so sometimes if they don't have either a model for the application security teams and the security functions to scale along with the growth of the development functions, then they're at a loss for resources or the teams that are now responsible for security, those teams need to pick up some of the resource responsibility for patching, for scaling, for retention, for a lot of the things that I think traditional, maybe finance models didn't always plan for.

Especially when software was built and delivered, often as a project. And then maybe you'd have a small operations team that maintained it, but there often wasn't a plan for the longevity of that application.

Clarke Rodgers (08:18):
So, a lot of our customers, they're all over the map from a cloud adoption and maturity perspective. We have some that are just starting out and they're applying security in the console. And then we have the far other end, where infrastructure and security as code and everything is going through a pipeline. If we look at the latter as being the goal for many customers, right, to get to that level of automation, what kind of advice do you give customers who are just starting out to see what that end result could be — without making it too intimidating — and get them adopting step by step so they're actually moving that quickly?

Bill Shinn (08:59):
Let's run that in a couple ways. The first is, I think just recently, in the last few years, I think security organizations industry-wide are finally understanding what it takes to build and deliver and ship software in a rigorous way, rather than just taking software packaging and clicking through an installation in a console — being able to do infrastructure automation, being able to deploy through continuous integration and deployment.

Most of the people in the security industry did not come up through software development and that part, that factory has really evolved. So understanding, specifically to security people, how would they go from doing something simple in a single account in a console to a mutable infrastructure that does thousands of deployments to 24 regions a day, right? You have to bring people along in their learning. You have to invest in training and education.

But I think showing it, showing them what it looks like, it's “walk the factory floor” if you want, whatever you want to call it. But having the CISO, if they're not familiar with modern software development, or people that are responsible for making risk decisions about “go, no go,” they should sit with the engineering teams and watch them do deployments for a little while. Watch them do sprint planning, watch them do their standups. Think about, look at how they prioritize their backlog, look what they have to do both with automation or with manual steps to just kind of walk a mile, I think. That opens a lot of doors. It sets off a lot of light bulbs of the security team then understanding how they can fit into that.

When you're starting out, there is a point at which using the console and document is just a natural immersive learning experience, workshops and getting started guides and there's that learning and lab experimentation environment. I think one of the things that to hold the line on when you talk to your business — who're the people paying for the development — is you have to build things into their expectations, right? You can't say, “We're going to deliver these features,” if you don't also accommodate the things like rigor, like testing and coverage and scaling and certainly security.

There's a whole bunch of properties in there that you need to make sure are accounted for as a product manager or the interface to the business. And if they don't understand that, it's your job as a CISO to explain the importance of these things so they fund it. And you have to do it in a way in their terms. So speaking about, not “CVEs” and “bugs” and “firewall rules” and all the things that we talk about internally, but you have to talk about what it means to not ship a widget or to have their brand, and what that specifically means. And sometimes it's hard to predict revenue, but if you talk in their terms, those are things that are tangible, just real-world tangible examples of why you need to build those things in.

You don't want to use scare tactics or FUD. I think being precise and clinical and talking about things that tend to make the news. They may have heard of Log4j or Heartbleed or some of these large-scale ransomware attacks affecting a lot of municipalities. But using those, not in alarmist terms, but talking about what investments needed to be made to avoid that from happening and to prevent that. A lot of it is going back to the basics about hygiene.

And I think customers ask me a lot, "What's the one thing you can do for security?" Or "What's the best tool?" One of the best strategies for security is modernization. Get your code modern. Measure with specificity and detail how fresh the software packages are. How often do you import from the main branch of code? How often do you get a new package and import it? And you're on the latest version. Because if something comes out that's a defect, a Zero Day, and then there's a patch available, how many versions do you have to go through and test and potentially rebuild a build process before you can get on that version that's patched?

Clarke Rodgers (12:52):
And the business risk that's involved.

Bill Shinn (12:53):
And the business risk that's involved. So that time — and you could learn from past issues similar to that, they will happen again — and being better prepared to weather that storm, if you will, I think is important. But that's kind of explaining that to the business with real tangible examples and not coming back to the well when the crisis is happening, right?

Clarke Rodgers (13:13):
Bill, thanks so much for joining me today.

Bill Shinn (13:14):
Sure, thanks a lot, Clarke. Good to see you.