EP 12 — Rajat Bhargava: How Stripe Built a Highly Scalable AppSec Program

Rajat: Thanks, Harshil, for having me. I'm excited as well.

Harshil: All right, so let's get into it. But before we go too deep into it, we just talked about some very exciting list of topics, but before we start them, it would be great if you can introduce yourself to our audience really quick and just talk about where you started your career and how it transitioned over different companies, different roles over a period of time.

Rajat: Yeah, definitely. As you mentioned, my name is Rajat Bhargava, and I'm a security leader currently working at Stripe. I lead the scale security assessment team at Stripe. The mission of the team at Stripe is to give confidence to developers in the security of the products that they're building. And we do that by preventing and finding security issues. But the main thing is also maintaining their velocity, right? So that's what I do at Stripe. My journey, actually, I started as a Java developer, and when I was developing code, there was an internal pentest that actually happened off our team code, and some security issue was found. And I was like, I'm very intrigued of like, what is this? How can people actually exploit that? And that's where my security journey started, because I started teaching myself how to do web application pen testing. And then finally, after a couple of years, I got an opportunity to be a web application pentester at eBay, and that's where my security journey started. So I started doing web application pen testing, and then application security just sucked me in. And then after that, I did all aspects of security — security reviews, doing tooling. That's where my developer side of things came in. Then I did architecture reviews, designing systems. And that journey took me from a lot of companies, from eBay, to Walmart, to Netflix. And then most recently, before Stripe I was at Twitter, I was the engineering manager of the application security team there, which I led in group.

Harshil: Phenomenal. So when you saw your pentest report as a developer, I'm guessing you were not the one who said it's a feature, not a bug. Haha.

Rajat: Definitely. I was like, this is something super interesting. Like somebody can actually do… Actually, I don't even remember what the actual issue was, but it was just interesting to know certain things that people can do, which are not intended for the code, and how they can exploit that.

Harshil: Yeah. So I love talking to especially my guests who come from a software development background because they just have such a unique perspective on security and the relationship with developers. Since you've been on both sides of the role, can you elaborate or can you give me some tips on as a security person, if you want to generate this intrigue and interest from developers about security, how have you seen it being done well? What are your suggestions on how to get that engagement from developers?

Rajat: Yeah, that's a very interesting question. One of the things that actually came easy to me was empathy for the developers, right? Because I was already a developer and it was easy to be in their shoes. So from a security profession, I think always having, when we talk about security, having security from their perspective. They're building a product, why do they care about security, right? And talk more about the quality of their code. Like security could just be a quality issue. Like increasing the quality of the code could be just better access, there is no access, that's just code quality. So thinking about what they actually care about and then praising the security team that you would want them to do in their perspective. And also keeping priority in mind, like they have so many things going on. So when we have from a security world, so many things that we would want them to do, have a list of priorities, have a list of ranks. What would we want them to do first, instead of just like, sending everything their way in one go. And also making sure that not everything is a fire. If security professionals always have everything that we want the developer to do like a fire, like, “fix it right now”, then you know, that social capital that we have is very limited with them. That gets depleted really fast. So I would always say keep not just developers, the people that you're working with, the stakeholders, it could be program managers, product managers, legal teams, keep them in mind when you're working with them, and also when developing your program. When you're developing your AppSec program or even security program, work with their perspective. Like, how would they actually take it? Because it's not in isolation that your program is going to be working in, it's going to be working with other people.

Harshil: Right. So that's an interesting point because we talked about multiple different personas who have a stake in this, whether direct stake or indirect stake. Whether it's developers or the development leads, or the managers, or the architects, or product managers, legal compliance and obviously application security, product security teams. So that is also challenging because if you have to keep these multiple personas aligned in terms of application security, but at the same time as application security people, we have a lot of sources of data. We have multiple different scanning systems, we have our own risk assessments, architecture review results, bug bounty results, pentesting results, and things that we just know of. There’s this multiple inputs, multiple sources of data, and they're all silos within themselves. And then there's multiple people who need to look at some sort of data. It may not be the exact level of detail but at different levels of abstraction, different levels of detail, different parts of that data. How do you manage all of these multiple inputs and multiple personas who are looking at the outputs?

Rajat: Yeah, that's actually a really interesting question because one of the challenges that AppSec or even just security has right now, there are so many tools that we run, so many results that are coming in, and because of the results, we're passing it to the developers, maybe without context, without action, right? So one of the projects that I'm working on currently in Stripe for our team is the project of having this one infrastructure, like a security infrastructure platform that helps consolidate all these tools into this one common place. All the results go in, and also hydrate those results with context. Like, for example, who the owner of a particular code path would be, or maybe a domain that's coming out of like, a dynamic analysis project, and adding priority to that result, right? Using the information that you have. For example, what service is this issue in, this vulnerability? Is it external, internal? Is it like PCI scoped? You know, adding that priority and then giving it to a developer. So one of my team members, Alex Garbutt, he has this thing that he promotes that everything should have context, should be found in a timely manner, and it should also be actionable, right? So we always try to have that. So use that common entry priority, add like some kind of hydration, and then provide those tickets to the developers with the priority that is company specific. Because these tools have their own severity. Like, dependent on you might say that this is the most critical thing that your package is fixed, but we might not be using it as an external, we might be using an internal.

Harshil: So how do you do all that? Have you built an internal system for aggregating all this, and tuning, and triaging and prioritizing? Right, okay, so I see you nodding your head. What is the level of investment that you have to make in terms of whether you have to hire developers? How do you maintain all of that stuff?

Rajat: Yeah, one interesting thing that we know now that application security is going through is that first we were hiring people that were just security people. Like they purely knew security. Then we started hiring people who knew some little bit of development and application security, and obviously they were really hard to find, right? Now, what companies are doing is that the team that we lead is mixed. We have security professionals, we have security professionals who know a little bit of development, and then we have software engineers, right? And I think that's a whole good collaboration to have because the tools that we're building right now for example this sole priority is we have a team of security professionals that help with the architecture, and then these software engineers take it to another level where they have a distributed system and they think about like, how to deploy it in a proper way, in a scalable way. And yeah, so that's why that's a good synergy of like software engineers and security professionals.

Harshil: Phenomenal. So since you've been at many really good companies and with solid security teams, have you seen a similar pattern being done before, or is this something new that you’ve worked on?

Rajat: So the interesting thing is, with all the companies that I work for, there are certain patterns, but because of the company and the context of the company application security changes. It's really hard to just like… or it's not even advisable just to take what worked in one company and then just put it into your other company, right? So depending on where you are at in your application security program, or where the product is at, or where we think the security should go in the company, is what we want to deal with, like maturity. So for example, with prioritization, currently we are at a maturity where we want this to happen. We have the tools running, we have the results coming in, but we want this prioritization and these things for developers. Like I mentioned, we have limited social capital and we want to make sure that we use it in a proper way.

Harshil: Right. Yeah, It's interesting because a lot of the security teams that I talked to, they don't typically have enough resources to build these types of systems internally. I mean, hiring security engineers is hard enough, there's almost never enough of them. And at the same time in a lot of cases they also don't have support from the other aspects of engineering organizations for them to be able to build something sophisticated like this and maintain it and deploy it. I think the challenges are also around writing all these integrations and maintaining data pipelines and maintaining this data architecture. It sounds simple, but it gets very sophisticated.

Rajat: Definitely. Yeah.

Harshil: But it's awesome. Kudos to you guys for having built all of this internally. You should definitely write a blog about it if you can. You should definitely distribute that knowledge within the community. One other related topic that I wanted to talk about was what do you think about the current state of application security? I mean, you've been in this space for a long time, working for some of the best companies. How do you see it different from what it is today as compared to what we've all been seeing?

Rajat: So over the past 10-15 years, I feel like right now it's very encouraging actually to be in application security because just the fact that there's more awareness in the industry about security and even application security, So the developers are more aware of the common vulnerability. Like, they know about OWASP top 10, Mobile top 10, right? Like when I started, we had to give education about cross site scripting, what cross site scripting was and why we need input validation, right? So right now it's like all the developers already know about the most common problems. And also the leadership, right? The company and the leadership consider security as a high priority. And they also recognize it as an existential necessity, right? They know that this is important. Sometimes companies can just go poof because of security. So that's very encouraging that the security industry is growing, and also there's more training available. If you look at Twitter, or if you look at any of the things on YouTube, there's so many free resources available for developers, or even for security engineers that are helpful. And also in the past, like there is more emphasis on security by default. I worked at Netflix and Jason Chan, he always talked about paved roads, making it easier for developers. Security by default is coming in so developers don't have to think too much about security. And also bug bounty. I think bug bounty in the couple of years that came in has changed application security a lot too. Because of this bug bounty, more people are interested in seeing how things are being done. And also I think it's elevating the security teams internally as well, to evolve the application security program and also the bug bounty program.

Harshil: Yeah, I agree, I 100% see that as well because earlier we used to have a lot of application security engineers do pen testing and triaging and just running tools and scanners and things like that, which are good, they're fundamental things, but they're not exactly leveraging your security expertise. And now that you can give it out to people or tools or systems or automations that can do it at a much better scale, in a much more effective way, a much more cost efficient way as well. That helps security engineers and security teams focus on more sophisticated problems that can be solved. So your tools are getting smarter so they can find more things without having to spend more manual effort, and security people can provide their security expertise to dev teams, work on building more interesting solutions, solving interesting problems, rather than doing manual ditch digging work.

Rajat: Yeah. And also as you mentioned before, earlier on, we had to build our own tools internally because those tools weren't available. Now there are tools that you can just add to your program, and you get that speciality, you get that expertise, and you can concentrate on the work that you're doing to build your program instead of just like “This is the tool that I want”. And having said that, there are certain situations with companies specific things that you actually just have to write. Like we do that all the time, where Stripe specific, we want to do something that the tool might not do so we write our own thing, which you'll still have to do, but the environment is much better now.

Harshil: Yeah. Can you give me an example of, like you just mentioned, tools coming out of the box with certain capabilities. Maybe give an example of that and an example of something that you had to write specifically for a particular business.

Rajat: Yeah. So I think the perfect example is Semgrep. We use Semgrep for static analysis, and Semgrep comes out of the box with a lot of these community written things, and they have a lot of patterns and stuff. But when you come into the context of, let's say, when we use it at Stripe, there are certain things that we had to write on our own. For example, for cloud security, we had to write a pattern that is specific to Stripe that are in configuration that we had to write on Semgrep pattern. Also, like, for example, we have configuration for egress proxies. So when an egress proxy happens, somebody can add rules, allow lists, but we wanted to make sure that people have business justification for every allow list. So we wrote a Semgrep rule for that, where now if somebody doesn't have a business justification configuration, a Semgrep rule shows up and then it's a blocker for that, right? So that's a tool that we have to write on our own.

Harshil: That's phenomenal, I love that idea. So in the old model of doing things, traditionally, we would run some sort of a scanner or whatever to say, “Hey, we found this AWS configuration that allows egress or whatever it is, or this particular configuration exists, but it shouldn't exist”. Now you go file a ticket, now you bug a developer to actually go fix that thing, and you track it, you escalate it, you file SLA tickets, and then eventually somebody closes it, and then you have to follow up and validate whether the fix is actually in place. So now you're short circuiting the entire loop by writing a simple rule in Semgrep that checks for the business justification being present, I'm guessing as some sort of text regex pattern that you're writing in Semgrep that checks your config file in Git, right?

Rajat: Right.

Harshil: Phenomenal. And if you don't have that business justification, your PR is not allowed to merge or what happens?

Rajat: Yeah, so it basically blocks it, you just need a business justification. And that way we can always have a paper trail of, like, why did we add this domain to our proxy, right? Why are we doing a legal proxy to the domain? But the other thing that came to my mind while you were talking is letting them know about this issue in a timely manner is really important. But also, what we want to do further what our future looks like is instead of somebody actually fixing it, we want to provide a PR that is like a suggestion saying, “Hey, there is an issue that we found, and here's a fix for it and we want to give you a suggestion”, right? Like, automatic PR. So we want to get there eventually and some of our teams in cloud security are actually doing that where they find like a configuration issue and they’ve already fixed it for the developers and checked in the code.

Harshil: That's amazing. Yeah I think that's the next generation of things. I mean somebody will still have to review that PR and merge it. But you're doing a lot of the work in creating that PR. Awesome. So I think we were talking earlier a little bit about some evolutions of security, and I think you had mentioned something about security guarantee that you were providing to the business. I think this will be an interesting topic for the audience to hear in terms of what exactly does your team do in providing security guarantees, and first of all, what actually does security guarantee mean?.

Rajat: Yeah so it's a very interesting concept of security guarantee. As security engineers, when we do security reviews, we look at the tech stack, like how the tech stack is looking like, how's the security of the tech stack, all those kind of systems. But we don't really look into the products and provide like, think about what security guarantees does the product provide? For example, I worked on a tool which is like an audio meet up tool where there's audio and then people join in and then they can see how many people are joined in. Now as a product, do we give a security guarantee by saying that the people that we see in that conversation are those the only people who can listen to the conversation, or can there be somebody else who's listening to the conversation but doesn't show up there? So we have to be very conscious when we're building a product and think about those security guarantees that we want to provide our users. So a lot of times when a bug bounty issue comes in and especially with the permission there's like an admin permission and there is like a normal user permission and a bug bounty issue comes in, a lot of times sometimes the security teams are not sure if this permission is actually a feature, is it an expected thing, or if it's an issue, right? So that's why we have to think consciously saying that when we think about it from a security guarantee perspective, when you're building the product, these are other permissions an admin should have, and these are the permissions users should have, and it's very clear cut. And when we are building more permissions, we think about security guarantees that we want to provide and then add those permissions in. And the way we work for security guarantees changes too. We have to work with product managers now. Think about when a product is being created, we want to give the user a security flow. So it's very interesting. We've done quite a bit there and still there is more to do.

Harshil: Amazing. Yeah, I mean it will be interesting to see how that space particularly evolves, especially if there are easy ways to provide security guarantees as a service to multiple development teams within the organization. But that's definitely an interesting concept. I think one of the other trends that I've seen increase quite a bit is the evolution of what some people call a security engineering team. Sometimes it's actually within the CISO organization, sometimes it's under the CTO organization. But what they typically do is build security services for the consumption of developers. Whether you're providing authentication authorization services, or crypto services, or secrets management services, or what have you, right? So some sort of an internal service that makes adoption of security controls easy for the development teams.

Rajat: Yeah, that's definitely happening and we do that at Stripe as well.

Harshil: Fantastic. I think we had sort of talked about too many sources of data, too many outbound things, and that obviously results into multiple security issues, whether it's vulnerabilities or compliance controls or just security best practices not being followed or what have you. That obviously leads to the problem of what should the developers focus on, what is really a priority? Because let's be honest, I mean developers' primary job is to ship features, right? That's their primary objective, that's what they are measured against. Security is one of the things that they should be focusing on. But that's not the primary job. So how do you make it easy for developers to really understand and act on things that really matter to them or to you from a risk perspective?

Rajat: Yeah, there are a couple of things in there. One thing is, as I mentioned, make it actionable for the developer. Like when an issue shows up for them, give them the full context and give them the full action of what it should be, what they actually need to do. And so it shouldn't be something like just say cross site scripting, they have to go figure out what cross site scripting is and go search in Google like “how to fix cross site scripting”. Basically tell them how to fix cross site scripting in the context of the company. Like, there might be different functions that you provide. So make it very actionable. Again, it’s easy to say it should be prioritized, but it's a very challenging thing. Like, how do you prioritize those things? So one of the things that I like is having a flexible program. Start with something and keep on growing on it. Like you don't have to have the most perfect thing right away. So the thing that we're trying to do with the prioritization is that we say take a few rules saying that if the service that an issue is found is an external service, or internal service, prioritize it in a different way, right? Or maybe just start with like, is it in a PCI scope or not in a PCI scope, then it's a high priority. Start with these few rules and start building upon it then, right? The other concept there is also like, is this issue found on a domain that is being attacked by attackers all the time, right? If it is, then it's a higher priority, than something that attackers don't really attack that much, right? So that prioritization grows with time, but starts somewhere with the priority so that developers know exactly which ones to tackle first.

Harshil: Have you ran into the problem of operationalizing this in terms of even things like whether it's in scope for PCI or not, whether it's being attacked, and whether you're seeing any indicators of attack or not on those things? So those datasets live in completely different things. How do you bring it back to the application security world and say, “Hey, we found this dependency, it lives in a server that we are seeing being attacked in our runtime environment, so hence you should prioritize this”. How do you connect all of that together?

Rajat: Yeah, it's very challenging and that's exactly why we're working on it, because these are challenging problems and it's interesting to work on these, and this is something that we need, and we're currently working on those. So working here is like we have multiple things. Like, for example, for the attack site, we need to work with a data scientist who can actually take all that data and make it into something that's consumable. And then the whole ownership, we have a tool that kind of finds owners for all the attacks, for all the paths, using that one system that I was talking about, previously alluding to. Like we have that security infrastructure platform, using that and building upon it to hydrate that information, to give context, add context, that's really important. And again, it's challenging, it's not easy. Even with the prioritization, coming up with what priority input should be, how should we consider it as a high priority is challenging. But again, when you're building that program, when you're building these things, include the developer team, include the other stakeholders to help you build those inputs, and that will make it easy.

Harshil: Phenomenal. Yeah, these are super exciting problems that you all are solving. I'm quite looking forward to how it evolves over a period of time. These are all definitely the unsolved challenges that plague the application security industry in general. So before we close any pieces of advice, any three pieces of advice that you might have for up and coming security professionals, application security professionals who are looking to grow in their career.

Rajat: Yeah, as I alluded before to this, I think the most important thing for a security professional is having empathy for the people they work with, the developers, the stakeholders, the legal team, the product managers, and not having the mindset of us versus them. Let's work together. They are experts in something different, we are experts in security, but let's collaborate to get to the same thing that we want. We want a successful company, right? And working together we can do those things. So I think having that empathy, having that mindset of like, “Let's work together” is really important.

Harshil: Fantastic. How do you embody that empathy? Is there an example of something that your team might have done or any concrete example that you can share on that topic?

Rajat: Yeah, even a simple thing like vulnerability management. When you're building that vulnerability management where we say that when an issue is found, we have to triage to this team, and this is how we're going to escalate. Instead of just building in isolation where only the security team is thinking about, engage the development team, think about it from their perspective. Or like when we were trying to do mobile security and we wanted to get tools, we worked with the mobile team to say, “Hey, this is a tool, this is something that we want to do. Is it even useful for you? Would you even use it? Like you have so many things going on, is this something that will add value to your product?”. And they were like, “Definitely, yes”. And then we went through evaluations and they liked the results that were coming out. The worst thing that could happen is you purchase a tool that you think would be good for the developers, and good for the team, company, and in the end, no one is using it because they don't like the results or they don't see it as helpful for what they're working on,

Harshil: Right, that's an interesting topic. And just one parting thought before we leave, there was recently a conference called LocoMocoSec, it happened in Hawaii. And I believe there was somebody from Netflix, if I'm not wrong, Patrick Thomas from Netflix who gave a talk on productizing security. And that's exactly the topic that was discussed. Which is when you're building security capabilities internally, think of it as a product and think of the rest of your organization as your customer. So how do you embody that product management mindset when you're building security features, capabilities, and tools internally, so you can truly provide a good service to the rest of the engineering organization, and to really up level and scale your AppSec team.

Rajat: Exactly. Like when we build our products for security, we have personas that we are catering to. Like, just as a product manager, you're creating a new product for your company. That's what we do for our security as well. We have personas thinking about what use cases, what are the things that we want to think about. So that's exactly how we want to build security.

Harshil: Fantastic. Rajat, it has been a pleasure talking to you, and I'm looking forward to having you again sometime soon.

Rajat: Same here, Harhsil. Thank you so much.

Harshil: Thanks for taking the time.