Share

Security Unlocked

The Microsoft Security Podcast

Security Unlocked explores the technology and people powering Microsoft's Security solutions. In each episode, Microsoft Security evangelists Nic Fillingham and Natalia Godyla take a closer look at the latest innovations
Latest Episode1/20/2021

Under the Hood: Ensuring Firmware Integrity

Ep. 11
How do weensure firmware integrityandsecurity?JoinhostsNic Fillingham andNatalia Godylaand guestNazmus Sakib,aPrincipalLeadProgram ManageratMicrosoft, to dive deeper and assess the complexities and challenges that come along with securingfirmware - bootstraps and all! MegamindBhavna Soman,a SeniorSecurity Research Lead, joins us later in the showand we learn about her journey inoptimizing AI and MLto improve efficiency in securityand give the humans a break. In This Episode, You Will Learn:   How Microsoftensuresfirmware integrity and securityHow firmware is making it harder for attackersWhere AI and ML will take threat intelligencein the near future Some Questions We Ask:  What isfirmware?Do we know where firmware attacks begin?Whatdoes the threat landscape look like for firmware?What part of ML should be automated better so that humans can shift to other tasks? Resources Microsoft Digital Defense Report:  https://www.microsoft.com/en-us/security/business/security-intelligence-report  Nazmus’sLinkedIn https://www.linkedin.com/in/nazmus-sakib-5aa8a6123/ Bhavna’s LinkedIn https://www.linkedin.com/in/bhavna-soman-3004b613/Nic’s LinkedIn https://www.linkedin.com/in/nicfill/  Natalia’s LinkedIn https://www.linkedin.com/in/nataliagodyla/  Microsoft Security Blog:  https://www.microsoft.com/security/blog/ Transcript(Full transcript can be found athttp://aka.ms/SecurityUnlockedEp11)Nic Fillingham:Hello, and welcome to Security Unlocked, a new podcast from Microsoft where we unlock insights from the latest in news and research from across Microsoft security, engineering, and operations teams. I'm Nic Fillingham.Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft Security, deep dive into the newest threat intel, research, and data science.Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft Security. If you enjoy the podcast, have a request for a topic you'd like covered, or have some feedback on how we can make the podcast better-Natalia Godyla:Please contact us at securityunlocked@microsoft.com or via Microsoft Security on Twitter. We'd love to hear from you. Natalia Godyla:Welcome to the latest episode of Security Unlocked. Welcome back to civilization, Nic. I'm hearing that Seattle had a pretty bad windstorm. Glad you're okay.Nic Fillingham:Thank you. Yes we did. We were out of power and internet for best part of two days. That was fun. But yes, we're back online. We have power. We have internet. We're back in the 21st century. How about you, Natalia? Any insane weather events up in the northeast? You guys get ice storms and cats and dogs and locusts falling from the sky, don't you?Natalia Godyla:None this weekend though. I did almost freeze going camping and I had a close call with an attack over the weekend.Nic Fillingham:Oh my gosh, that sounds crazy. What happened?Natalia Godyla:I mean, it happened in Outward. I feel like I probably should have started with that. But Outward, the game.Nic Fillingham:Oh, okay. Phew. I feel like you would have mentioned that to me in advance of recording this podcast had you actually been attacked in real life. What's this game? What's the game you're playing?Natalia Godyla:It's an RPG game where you try to quest through this... Gosh, I don't remember a single name of any of the locations, the cities, or the mountains. I'm not paying attention. I'm really focused on the battles that you have to fight.Nic Fillingham:What are you battling? Can you give something away or is it a spoiler? Is it humans? Is it animals? Is it zombies? Is it aliens?Natalia Godyla:It's a mix. There are bandit camps and then there are troglodyte caves. I think I've taken on a whole lot of the troglodytes at this point though. So I don't know if they're still in existence.Nic Fillingham:Let's take 30 seconds to look up Outward. You said troglodyte, and I really feel like troglodyte is an established word that means something. Oh, okay. So troglodyte is from the Greek, troglodytae, which literally means cave goers. Is that right? Do they live in caves?Natalia Godyla:They do live in caves.Nic Fillingham:Oh, there you go. Okay.Natalia Godyla:This game must have done its research.Nic Fillingham:They're cave goers, but they're also your enemies. Is that right?Natalia Godyla:Yes, but I guess in theory, I brought it upon myself. I mean, I kind of wanted to loot the cave.Nic Fillingham:So you actually went into their territory and were like, "I'm going to smash this jar and get this green jewel out of it." And they were like, "Hey."Natalia Godyla:Yeah. I mean, that's a moral gray area because they saw me and immediately attacked but it was their cave.Nic Fillingham:So you're the bad guy. Nice. All right. We're going to play this. We're going to play Outward. Wonder if we can get all of the Security Unlocked peeps into a single game. That'd be fun.Natalia Godyla:Oh, yes. And I think with that, we can intro our guests. Yeah, there's no connection point here.Nic Fillingham:Speaking of cave-Natalia Godyla:Looting.Nic Fillingham:Looting? No.Natalia Godyla:How do you stop looting from happening?Nic Fillingham:Oh, got it. I got it. If only those troglodytes had better security, people like Natalia Godyla wouldn't just come wandering in to ransack the place looking for leather and iron ore to craft rudimentary weapons. Speaking of better security, today on Security Unlocked, we talk with Nazmus Sakib who is going to spend a bit of time talking to us about firmware and the challenges associated with ensuring firmware integrity and the integrity of device security all up starting with firmware. This is going to be the first of three conversations that we'll have over a number of episodes where we better understand the security of devices from the firmware up. And then after that segment, Natalia, who do we speak with?Natalia Godyla:After that, we speak with Bhavna Soman who is a senior security research lead at Microsoft. And she shares how she got into security, which was really a role that she played in her family. She was the de facto support for network and security issues like antivirus. And as she continued in that role, she got more and more curious and tried to understand what technicians were changing or why something might be affecting her computer. And that role and responsibility just made her that much more interested in the security space and eventually led her here to Microsoft where she works on understanding and getting insights from the data that we have in order to better inform our defender products. Onto the podcast.Nic Fillingham:Onto the pod.Nic Fillingham:Welcome to the Security Unlocked Podcast, Nazmus Sakib or Sakib as we'll call you in this podcast. Thank you so much for joining us.Nazmus Sakib:Thanks, Nic. Thanks Natalia for having me. It's a pleasure to be on here.Nic Fillingham:So two things. We love to start up these deep dives with an introduction. Would you mind explaining? I introduced you as Nazmus Sakib, which is your name. We're going to call you Sakib. Just anything you want to sort of say about that, but also, what do you do at Microsoft? Tell us about your role, the team you're in, the team's mission? What is your day-to-day like?Nazmus Sakib:Yeah. I'm Nazmus Sakib. I go by Sakib. It's usually a sign on the team that you've met me where I get to clarify that growing up, everyone just called me by my last name. I'm originally from Bangladesh and Sakib is just more common as a first name in Bangladesh, which is what most people... My family ended up calling me. There's a famous cricketer by the name of Shakib Al Hasan who some listeners may be familiar with, but this is my first foray into fame.Nic Fillingham:I am familiar with famous Bangladeshi cricketers. Thank you very much.Nazmus Sakib:He's finally back after an unfortunate ban, but I think it's great to have him back on the team. Super excited for the prospects of the Tigers.Nic Fillingham:Do you play cricket? We're going to do this. We're going to take the little party.Nazmus Sakib:Yeah. Let's go down fully on that rabbit hole. So I played a lot when I was younger. I've been in America mostly since 2008, is when I first came for college. But prior to that, like most I think kids in Bangladesh, we play cricket. And usually, I grew up in Dhaka, which is the capital. So it was all improvised for the longest time. We had a little space on our roof. So it was like this flat essentially. And so it was probably about maybe 10 feet by 10 feet or not even. And so me and my cousins be a team of like two or three kids and we'd split it up. Someone would bat, someone would ball. You'd make up the rules in terms of how the runs would work. And same thing with if you find a little space in a back alley, or in any small sort of field or space that you'd get, you'd find a way to make it a cricket field. So good memories from back there. So it was kind of informal, but a lot of fun, especially now that the years have sort of gone on and I'm in a much different place where you just don't do that. It's pretty cool memories.Nic Fillingham:Bring us back to your role here at Microsoft and sort of what you do. Can we think of a good cricketing segue? Is there any famous cricketers that have moved into the cybersecurity field? What's a hard left turn?Nazmus Sakib:I think Satya is obviously-Nic Fillingham:Oh, yes, Satya loves cricket. He's a big cricket fan.Nazmus Sakib:Satya loves cricket, yeah. So I guess he's the most famous former cricketer turned tech luminary that I can think of.Natalia Godyla:10 points for the connection there.Nazmus Sakib:So yes. It is a well worn path, cricket to Microsoft. And I'm just one more traveler on that road. But my day-to-day, I've been at Microsoft for a little over eight years now, actually right out of college. I work as a PM in one of the many security teams at Microsoft. My team currently is in the Azure Edge and platform team. Our team is responsible for the operating systems that we ship as part of Microsoft, and also that operating systems that our customers use on platforms like Azure. So our team has been responsible for building the security that goes into Windows for a long time. Been a part of that team since I started at Microsoft.Nazmus Sakib:And then with the way to serve our customers on Azure, we want to meet them where they're at. And we have a lot of Linux customers on Azure as well. And so increasingly, our team is not just doing Windows work. We're also investing in Linux security technologies to help ensure that if you're a customer coming into Microsoft, if you're using Azure, whether it's on Windows or Linux, really bringing that platform, that operating systems' expertise to help secure whatever it is that you're you're trying to do. Nic Fillingham:Awesome, thank you. I'm really excited for this conversation we're about to have. It's going to be one of sort of three. I won't call them introductory, but it's certainly a little trinity of conversations over the next few months where we're going to talk about firmware. We're going to talk about firmware integrity, the challenges of that, and how you go about ensuring and securing firmware integrity. We're going to follow that up in a future episode talking about the Microsoft Pluton announcement. I'm sure that'll come up at some point in our conversation today. You're joining us today, Sakib, to help us sort of come back to basics a little bit. Can you help orient us in this world of BIOS, UEFI firmware, all the various sort of synonyms for this stuff? We're going to talk about firmware. Let's talk about what is firmware. Let's talk about these acronyms. If you would, just sort of re-educate us so we can start the conversation.Nazmus Sakib:Right. So the easy way to think about firmware is it's the first piece of code that runs on your hardware, right? So it's easy to sort of visualize that when you have a device, it's a desktop, or a PC, or a phone, any kind of computing device, you have the actual hardware, right? You've got the CPU, the motherboard, the power button that you use to turn the whole thing on, you have the hardware. The firmware is really essentially software that's typically baked in to the hardware. So it ships typically as part of the hardware. There's usually some read-only memory chip that's dedicated to storing that firmware just so that when a customer hits the power on button, the hardware knows how to turn everything on essentially. It's the firmware, that piece of software that actually goes and coordinates how devices are being made available to all the other things that run after the firmware, which is the operating system, and then the applications that you use on top of the OS.Nazmus Sakib:So if you were to think about from the point that you turn on a device to the point where you're using an application, whether it's your browser, whether it's Teams or Zoom because it's COVID, usually a very simple workflow for that is you're turning on the hardware. The firmware is the first piece of software that runs on the hardware platform. It bootstraps the operating system. So it could be Windows, it could be Linux. And then after that, once you have the operating system running, you can run applications like your browser, Teams, Zoom on top of that operating system platform. Nazmus Sakib:So the second part of your question, what is BIOS or UEFI? They're essentially flavors of firmware. BIOS has been around for the longest time, I think, in many ways with the history of the IBM PC. The BIOS was what you'd call essentially the firmware that ran on an IBM PC platform. A few years ago now, I think, essentially, the industry got together to revamp the firmware standards. So it's both a specification and an implementation of that specification. So EUFI, you can think about it as the modern BIOS, but because historically, people called firmware BIOS for the longest time, they're almost essentially synonyms. But typically, BIOS and UEFI both refer to the firmware that runs on any particular platform. And in general, they're perhaps used synonymously if we're speaking loosely. But most modern systems today use some implementation of the UEFI specification as the platform firmware.Natalia Godyla:Can you provide some security context around firmware? What does the threat landscape look like for BIOS or the broader term firmware? What's been the history of attacks? What's more or less prevalent for firmwares compared to applications that are at risk?Nazmus Sakib:Right, right. So much work has gone in to so many different parts of the technology stack, right? You think about the work that we've done at Microsoft and across the industry around things like antivirus solutions. You look at modern platforms like Microsoft ATP, Advanced Threat Protection, where you have just a view of the health of your operating system across many devices that's customized for your enterprise. All of those things, in many ways, have already made it harder and are increasingly making it harder for attackers to do things that they would have maybe gotten away with in the past for attacks in the operating system.Nazmus Sakib:And so naturally, when you make one thing harder, you incentivize attackers to go elsewhere, right? And so what we saw as a trend and one of the places where this was really sort of evident to us in a way that felt it wasn't just us looking at it, it was also externally reported is if you look at the NIST which is the American standards body, essentially, the National Institutes of Standards and Technologies, I think, I'll have to go verify that, but they actually maintain the National Vulnerability Database. So if you think about vulnerabilities that get reported, you see in the news and they often have some numbers associated with it. That's actually all the numbers in the National Vulnerability Database. Nazmus Sakib:And so one of the things that you saw in the research that's being done in the industry, this is where all the security researchers report issues. It's like the aggregate. This is how the industry keeps track of all the vulnerabilities that are happening across all technologies. There was a large spike in firmware. If you just go to the NIST website and you go type into firmware, it went from a handful of firmware vulnerabilities being reported in, I think, 2016/2017 to hundreds being reported in the last year or two. And so a huge spike beyond exponential. And that really is because we're making it harder to do the things that perhaps attackers would be able to do in the past and the operating system. And so people are naturally moving elsewhere. And so they're gravitating towards firmware as an avenue. So that's one reason.Nazmus Sakib:The other reason is coming back to what I was talking about in terms of how a platform boots. Firmware, because it's the first thing that runs on your hardware, because it needs to, just by its very nature, set up your hardware in the right configurations, it actually bootstraps a lot of the security on your system. Right? And so it's almost like a double whammy. Attackers are moving to a place where a lot of the problems that have been solved in the operating system from a security perspective, they're trying to work around those protections. And then in firmware, they actually see that you have this highly privileged environment firmware typically has almost usually when it starts up, almost unrestricted access to all the hardware and the data that's on your hardware. And so that's really where we're seeing this trend where attackers are... the security researchers suggesting that attackers are going to be moving there. Nazmus Sakib:And one very recent practical example of a threat where these trends are bearing out is just, I think, last week, there was a report that TrickBot which is almost like a modular malware that's being used in a lot of other ransomware attacks, it's actually added firmware capabilities. So it's using other longstanding well-known vulnerabilities in the operating system, but because of the trends I've just described, we're seeing TrickBot add new firmware attack capabilities as well.Nic Fillingham:Sakib, do we know when firmware attacks begin? Is there a defining moment in time when firmware became an actual viable target? Or has it sort of always been there and it's just recently evolved?Nazmus Sakib:It's always been there. I mean, firmware is always run with high privileges in a way that it may be difficult for operating system software, including security tools, to tell what's going on in firmware. It's easy for firmware malware to hide what it's doing. But if I were to think of a tipping point, if you will, a couple years ago, we saw that at least one example of what's typically associated with a particular nation state threat actor. There were targeted attacks a couple years ago that were using a firmware vulnerability. So in some ways, that was a very clear signal that not only is the security research headed that way, but there's at least that first example. It's almost like the canary in the coal mine, if you will, where we saw an example of an attack that tried to do exactly what I described, is use for a very targeted attack, use firmware to circumvent a lot of the security tools, and find a way to persist. Nazmus Sakib:And with developments like what I talked about for TrickBot, which is generally often used by many different actors trying to orchestrate different ransomware attacks like Ryuk and Conti, we expect to see that trend sort of increase. And so if I were to think about that first tipping point where attacks start to become real, the LoJax attack is, I think, what it's typically referred to as maybe the one I can think of where it really sort of became not just a trend we're seeing in the research, but a really practical attack. Nazmus Sakib:By its very nature, firmware is complex. There's tens of thousands or millions of lines of code running if you think about all the firmware that runs on your system. So if you just think about the basic security principle of trying to reduce your attack surface, trying to have lease privileges, what you really want to be able to get to is that your trust is not necessarily fully dependent on all the firmware being written totally correctly and totally secure and not vulnerable to an attack. Ideally, you want to not trust that huge infrastructure. You want to be able to go do that trust of fewer set of things. And that's sort of the journey that we've been on recently with our OEM partners as well with secured-core PCs is to do that evolution. A UEFI secure boot doesn't go away. It's still an important technology. But we want to be able to start layering on additional capabilities that can start to protect important security properties or security capabilities even from firmware compromise as that's really where the trends are going from an attacker perspective.Natalia Godyla:So your team has done a lot of great work around secured-core PCs. What would it take for an attacker to actually break into one? Is it possible? What do they have to overcome?Nic Fillingham:Without obviously giving away some operational security here, but just like in Bizarro fictional land with infinite compute power and physical access to the device, what are the monumental challenges that would need to be overcome?Nazmus Sakib:There are a couple places that I think are interesting that we're definitely thinking about. Security is not a static thing. It's always dynamic. We do something and then so do attackers. And so if you think about... It comes back to maybe the foundation analogy. We are building a lot of our security promises on things like the TPM. We want to be able to securely record the firmware that's running so that we can actually tell that it's the firmware that we expected. Right? So that's an area that we're thinking hard about and it's part of the motivation for Pluton. I'll leave it up to you all to interrogate Peter around what the effects are, but I think that's one place where a lot of our security promise is built around that.Nazmus Sakib:We spend a lot of time thinking about TPM attacks. And it's a big part of the motivation for why we're adding another choice to the Windows ecosystem around using Pluton, is just being able to continue to raise that bar against attackers. So I'll leave it to you, Nic and Natalia, to interrogate Peter as to how Pluton will help with the security of future Windows systems.Nic Fillingham:We'll absolutely do that. So Sakib, thank you so much for your time. As always, we will have some notes. We'll have some links in the follow-up show notes. And I'm not sure we've actually offered this to listeners before, but if you do have questions about securing firmware, anything that Sakib talked about, contact us on the Twitters. You can send us an email, securityunlocked@microsoft.com, and we'll do our best to point you in the right direction. Thank you much, Sakib.Nazmus Sakib:Yeah, no. Definitely thank you for having me on here. It's just a great competition. I enjoyed it. And I second what you just said. We'd love to hear from listeners around things that we can do a better job of communicating or feedback folks have on how well we're doing in terms of meeting their needs. Nic Fillingham:Sakib, thanks so much for your time, mate.Natalia Godyla:And now, let's meet an expert from the Microsoft security team to learn more about the diverse backgrounds and experiences of the humans creating AI and tech at Microsoft. Today, we have Bhavna Soman on the episode. Thank you for joining us.Bhavna Soman:Thanks for having me, Natalia and Nic. I'm very excited to be here right now.Natalia Godyla:We're excited to have you. So love for our audience to get to know you a little bit more. What is your role at Microsoft? What does your day-to-day look like?Bhavna Soman:Yeah, absolutely. So my official title is senior security research lead. But like it often happens in big organizations, it kind of doesn't accurately reflect what I do. I lead a team of security researchers and data scientists who use machine learning and AI to fight threats on the Microsoft Defender platform. And that kind of reflects my own background as well, which has been checkered with experience in security research and machine learning. So to me, that's a very good fit even though I can't get them to include all of it in my title.Nic Fillingham:Bhavna, we've spoken to a few of your colleagues on the podcast already: Holly Stewart, Jeff McDonald recently, Karen Lavi. How would you describe what you do? What is different about your role and your team compared to maybe Jeff's team or Karen's team, et cetera, et cetera?Bhavna Soman:Yeah, absolutely. So the focus for my team is on using AI and ML on building intelligence and context for our enterprise customers. So when you look at how you want to apply machine learning in data science, I think it all really boils down to how can you reduce the dependency on human beings who have the security expertise? How can you bring in AI to help enterprise customers better defend themselves in this field that has a scarcity of talent, to be honest? And so what they do is look for clean or malware files. Whereas my team is focused on providing, for example, information about emerging campaigns or information about, what are the attacks that are linked to each other and form one incident so that an organization can address them together as a whole and therefore get efficiencies from that analyst as well?Bhavna Soman:So these are just a couple of examples of what I mean when I say like we provide the intelligence. So I think someone put it very succinctly a few weeks ago where Jeff's team finds the badness, Karen's team finds the goodness, and I kind of bring it all together and give it meaning.Natalia Godyla:That's awesome. I love that definition. Nailed it. And stepping back for a moment, I'd love to hear about what brought you to Microsoft and what brought you to security research. As you mentioned, you had a journey that included machine learning and security research. So how did both of those come into your career path?Bhavna Soman:So I was always excited by security. And even from a very young age when we had our first laptop, which was like way, way back. I think it either had Windows 95 or 98. So it was really old. And those days, you get infected by stuff all the time. So for my family, it used to be my job to kind of figure out exactly where was the registry key in which this thing had saved its autorun tactic or persistence tactic. And at that time, I didn't know what any of these were called or anything. But that's how I first got into it. And then I decided that I really loved this sort of adversarial aspect of security. It really brings an excitement to the whole thing for me. Bhavna Soman:My path did not take me directly to security still. My undergraduate studies were in mechanical engineering. So thankfully, I got a fair bit of math and also programming classes in, but I was chasing different things at that time. But after a while of working in that space, I was actually doing pipeline design for this company that constructs oil refineries, which was a very soul-sucking job for me. Yeah. I didn't like it at all. I did that for two years after college, and it just was not for me. So I was like, "Okay, I really love computers. I have to go in that direction." So I started to build software tools for that company. And then that gave me sort of this way to dip my toes in. And then I realized that, okay, this is definitely something I love doing. So I decided to go for masters. Bhavna Soman:And then when I was choosing my area of focus for my masters, I was like, "Yes, security has to be it." So I went to Georgia Tech to do my masters and I specialized in security. So that gave me a great sort of grounding and all of the basic skills, a great background at the industry. And Atlanta has a very good infosec community too. So I had the chance to get plugged into that. Yeah. I really loved going there. And after my education there, I worked for this startup out of Georgia Tech, which incidentally specialized in using machine learning for network security. So that's where I think I got introduced to, hey, machine learning and artificial intelligence can have something to say about this. Bhavna Soman:The more I stayed in the security industry, this problem of how it's all a whack-a-mole where a few people are chasing thousands and millions of different variants of the same attack. It really impressed on me that this is not something I can do manually. I can reverse 10, 15 samples. I can't do a thousand. So that's where the power of AI and machine learning really struck me. So I think that's where I started going deeper and deeper into that.Nic Fillingham:I wanted to come back to something that you touched on about being the family... What did you say? When a virus came on the computer, you would be the one that would be in charge of getting it off? Is that correct?Bhavna Soman:Yeah. Yeah. So at that time, I think they weren't super severe viruses. They weren't doing human operated ransomware stuff. For instance, they'd show you annoying pop-ups or they would change your search engine all the time. And they were doing very annoying things like that. I took on the task of investigating, how exactly is this thing coming back even though I deleted it? And then I started to discover the hidden mode in Windows and I started to discover all of these registry keys and rededit. It kind of went deeper and deeper and deeper from there.Nic Fillingham:Got it. Were these in the days where you could just install as many toolbars as you wanted inside your browser to the point where you could no longer see a web page? Are we going back that far?Bhavna Soman:Yeah, yeah. It was one of those days where... And also, Google was not really a thing. I remember Yahoo chat rooms used to be the big thing.Nic Fillingham:AltaVista, baby. AltaVista.Bhavna Soman:So fun times. There was a simpler world for sure.Nic Fillingham:Bhavna, how long have you been at Microsoft now?Bhavna Soman:It's been three and a half years now.Nic Fillingham:Got it. And and the first role that you came into at Microsoft, was that in the team that you're in or was that in a different group?Bhavna Soman:It was still with Microsoft Defender, but I was doing slightly different stuff. I was focused more on just pure security research and not as much on the machine learning and AI aspect.Nic Fillingham:Three and a half years ago, what were you focused on? And how has that sort of potentially evolved? How has that changed today? Were you still focused on the same types of attacks? They've just sort of evolved in sophistication. Or was it a completely different world three and a half years ago?Bhavna Soman:So when I first came to Microsoft, I was coming fresh off of Intel. At Intel, my focus had been on threat intelligence. Again, this was back when threat intelligence was just starting to become a thing. So I joined Intel before that. And at that time, they needed a threat intelligence platform where you can gather all of the TI information from all these feeds: internal, external, et cetera. So I built that first platform, plugging it into all the internal/external data feeds, organizing the data, and then having that pumped into the various prevention and detection systems. So that's what I was doing primarily at Intel. So when I came here at first, I was still in that mindset, and I was still trying to apply intelligence to improve protection. So I was doing a lot of hunting on VirusTotal, kind of try to find out where our biggest gaps were, and trying to plug those. Bhavna Soman:But very quickly, that pivoted to using machine learning for security was focused on non-PE files. So very heavily focused on the document files that we very often see come in as email attachments, and then they will lead the user to download something actually bad like, again, an Emotet or Dridex or something. So it was very focused on those macro files and other non-PE files. JavaScript was a big one at that time. So writing classifiers to differentiate between malicious JavaScript and the benign kind. Those were some of my first projects here.Natalia Godyla:So you said a couple of times that the draw of machine learning for you is the potential for scale, the potential for helping to fill that skills gap. So as you're shifting into roles where machine learning is playing a bigger and bigger part, what are the achievements that you're focused on? What would you like to try to automate better so that humans can shift to other tasks?Bhavna Soman:So there is one problem, which is very close to my heart. And that is the problem of the core threat intelligence business. So Microsoft Defender has a really big threat intelligence team. And this was something... I was part of the threat intelligence team at Intel as well. And all through my time working with these teams, it's been obvious that threat intelligence is very manually driven right now, right? It has to be a human that is reading files or PDFs or white papers. And then this human is, again, observing traffic data whether by hunting or through the attacks that they are remediating or something like that. So this human is then kind of assimilating all of these insights that they have about these attackers. And then they put it out somewhere. Like maybe they will communicate it to their customers saying, "Hey, this is what you need to be careful about." They may write a white paper or they may do detections as a result of that. So this is a very human thing. Bhavna Soman:And when I look at artificial intelligence and machine learning, to me, using large amounts of data to extract a few critical insights, to me, this is a very good use case for machine learning and AI. So this is a problem that I have been working on for a really long time. My first attempt at this was while I was at Intel, and I did this kind of cross-team project with a team that was in Argentina at that time to work on a method that could use question answering techniques from machine learning to answer questions about attackers. So if I had a question about, "Okay, what is the tool that this attacker uses? Or what is the victim vertical for this attacker?" Can I use question answering techniques and train on the corpus of data available about these attackers and have an AI-based system give an answer? Bhavna Soman:So I've been attacking this problem for many years. My first attempt while I was at Intel was not very successful. But a couple of years ago, I gave it another shot. And this research ended up being... I presented this at Black Hat last year where I was talking about how we can use some new techniques that had come out since then around word embeddings, natural language processing, and domain specific named entity extraction to do similar stuff. So I think I've been making progress on that problem. And now I'm working on a project with University of California, Berkeley on this security AI RFP where now they're expanding some of this work into the security knowledge graph where their aspiration is even bigger. Yes, we grab all of this data from a variety of different data sources. Yes, we do named entity extraction. But what else can we do on top of that? Can we automatically build, for example, YARA signatures based on this? Can we use multiple data sources to achieve consistency internally within this graph? Bhavna Soman:So that's where we're seeing AI and machine learning will take threat intelligence and help it become a little bit less manual, and again, less dependent on manual expertise?Natalia Godyla:What challenges are you facing with achieving some of the goals you've outlined? I'm assuming compute is always something that's in the back of your mind. What else would be a barrier to potentially achieving some of these successes? Or what are you tackling right now to reach your goals?Bhavna Soman:That's a great question. Compute is a big one because on one hand, we have large amounts of data. But on the other hand, A, to process all of that in a deep learning style would take huge amounts of compute that would make our product run very inefficiently on our clients and in organizations' machines. So usually, that's not feasible, which is why one of our big focuses is to find efficiency in whatever techniques we're using so that the model can be lightweight and yet perform with similar degrees of precision and recall. Bhavna Soman:Another big challenge we face is good labels or ground truth. Just because the spectrum of badness is so huge, on one end, you have these just adware things are grayware things that their whole goal might be to show advertisements or cause pop ups. And on the other end, you have APT threats. So in this wide spectrum, we have to find good labels for a large enough set for each particular category so that we can accurately classify threats and inform users about that. That's been a very interesting problem too. Going back to the threat intelligence space, one really huge challenge is that the field is continuously evolving. A particular thing might be used for human operated ransomware on day one, but on day 30, it's hosting some random adware or some software bundle or something. So within that span, even in shorter spans, the situation really changes. The intelligence you have really changes. So all of your machine learning systems have to be able to constantly getting the latest information adapting to that. So those are some of the big challenges we face in this field that we're trying to work around.Nic Fillingham:Bhavna, one of the questions we like to ask on the podcast is, what from your personal life, whether it's a hobby, whether it's something growing up as a kid, whether it's education or previous job, do you bring forward into your current job that could be considered maybe unorthodox? You teased very early on that maybe you play D&D. Is that true? Bhavna Soman:Yeah. I play video games or board games. I'm into all of that.Nic Fillingham:Is that a passion for you? Do you find yourself bringing any game theory or the way that you would approach a D&D encounter into your day job?Bhavna Soman:I think my biggest influence is books and language. I have been into books as far as I can remember. That was my favorite birthday gift when I was a kid. I just dragged my parents to the bookshop and buy a bunch of stuff. And a peculiar way in which humans use language and give meaning to it, to me, that is a source of endless fascination. Which is why one of the favorite authors for me is Patrick Rothfuss and his book, Name of the Wind. I think that book really talks about... It's a fantasy book. So it kind of goes into like if you know the name of a thing, then you have some control over it. It's a philosophical point, but also it says something about language. And in my mind somehow, all of that comes together and that really leads me into, how do machines interpret language? What does it mean for a machine to understand language? And when we're building all these natural language processing models, what exactly are we doing? And then what exactly are we missing from what human communication actually entails?Bhavna Soman:Which is why I'm kind of always drawn into this threat intelligence field because I'm like, "This is really where the importance of language and communication becomes connected to security." So that's kind of this one thing for me that I really, really love. In fact, one of the really cute examples that's always stuck with me is when you do a beginner course on natural language processing, you always kind of get this example. It's called crash blossoms. There was apparently a headline in the newspaper a long time ago where the headline said, "Violinist in Japan Airlines Crash Blossoms." And obviously, the headline meant to say that this violinist who was involved in this air crash a while back is now doing well. But when an NLP based system is trying to process it, it is like, "What is crash blossoms?" And I love that problem because it kind of emphasizes very clearly how machines are different from human beings, and yet how we're trying to bring the two closer for our own benefit.Natalia Godyla:I feel like one of the other unique points about language is just the evolution of slang. So I'll be curious to see how NLP processes and consumes slang because that is such a cultural moment. It depends on the cohorts of people that you surround yourself with, the social context. Bhavna Soman:Yeah, that's a great point. You talked about slang specifically where a meaning of a particular word or phrase can be different based on even the environment or the forum in which it is used. Certain terms, if you use it in an industry specific way, will mean very different than in the general sense. And we come across that in security so much, right? We have all these actor names like Scary Panda or Crawling Spider. And if you think of using like a traditional NLP model and all of this data, you're like, "This is not going to make sense because you're talking about a specific entity, an actor, not an animal." So we do have those kind of challenges in our domain. And I love diving deep into that.Nic Fillingham:So I have another sort of random question. I was possibly laying the ground for this with my previous question about, what from your hobbies do you sort of bring forward into your work? Your avatar, your photo in the Microsoft GAL in our sort of identity system is Megamind. Is that right?Bhavna Soman:That is absolutely right. I think that really ties into my sort of chaotic neutral rogue character because Megamind is a really good example of that, right? Supposed to be a villain but is a hero, but also is a villain in some ways still. This was actually a prank. We had Microsoft Month of Give last month. So your teammates could donate some money and force you to change your profile picture. So that's what I got.Nic Fillingham:Did you choose Megamind or Megamind was thrust upon you?Bhavna Soman:I chose Megamind. I was like, "Okay, this is the most appropriate for me."Nic Fillingham:Oh, so you do resonate with the Megamind character on some level?Bhavna Soman:I do. Yeah. I think so. And also, it's a really good movie that kind of has not had its time in the limelight for a while.Nic Fillingham:I don't know if I've seen it. I think my kids have seen it. That's sort of why I know it because I think I've sort of had to approve them watching the movie, but I don't think I've seen it. It's good, is it?Bhavna Soman:It is amazing. You should definitely watch it. It's a very cute movie.Natalia Godyla:I think we have our homework, Nic. I haven't seen it either.Nic Fillingham:Bhavna, before we let you go, is there anything you would like to plug? Any sort of organizations you're a part of? Any communities, groups? Anything you'd like to say out there to aspiring students of machine learning who either want to get into the field or just want to get better at machine learning?Bhavna Soman:I would love to. So the organization that I want to talk about is not associated with machine learning only. It's associated with security all up. So I am part of a group of women called BlackHoodies. And we are committed to increasing the participation of women in hard technical areas, which sometimes don't see as much participation from minorities. We are across the globe across many companies group. The only I think criteria is you are a woman, whatever your definition of that is, and it's always free. We hold classes at multiple conferences across the world which we'll do things like reverse engineering, Windows, ARM, web hacking tools like Ghidra, all of that. We have all these trainings that are completely free. And now that we are in the pandemic, we're doing some of these remotely. So please follow us on Twitter. And if you're interested in joining one of these trainings, it's super easy. And we really, really welcome anyone who wants to learn about this stuff.Nic Fillingham:As you were talking, I searched Black Hoodie on Bing and just got a thousand results for buying a black hoodie. What is the URL for the community group? I think I may have just accidentally purchased a black hoodie. I've got Amazon, what is it, one click buy. I went a little too quick. I was trying to pay attention to the recording window for the podcast and then searching for what this was. Anyway.Bhavna Soman:I hope it fits. So the website is blackhoodie.re. And we talk about all of the latest events or workshops that are happening there. Usually, when Microsoft holds Blue Hat, we'll do a bunch of trainings at Blue Hat as well. I do the beginners reverse engineering for x86 as part of that. But right now, we don't have in-person conferences, but we're doing virtual stuff.Natalia Godyla:That's great, Bhavna. I think one of our previous guests has also shared BlackHoodies. So thank you for highlighting it. It sounds like a great organization. And to our audience, please check it out. Thank you, Bhavna, for being on the show with us today.Bhavna Soman:Thanks for having me. It was super fun.Natalia Godyla:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us @msftsecurity or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe,Natalia Godyla:Stay secure.
1/20/2021

Under the Hood: Ensuring Firmware Integrity

Ep. 11
How do weensure firmware integrityandsecurity?JoinhostsNic Fillingham andNatalia Godylaand guestNazmus Sakib,aPrincipalLeadProgram ManageratMicrosoft, to dive deeper and assess the complexities and challenges that come along with securingfirmware - bootstraps and all! MegamindBhavna Soman,a SeniorSecurity Research Lead, joins us later in the showand we learn about her journey inoptimizing AI and MLto improve efficiency in securityand give the humans a break. In This Episode, You Will Learn:   How Microsoftensuresfirmware integrity and securityHow firmware is making it harder for attackersWhere AI and ML will take threat intelligencein the near future Some Questions We Ask:  What isfirmware?Do we know where firmware attacks begin?Whatdoes the threat landscape look like for firmware?What part of ML should be automated better so that humans can shift to other tasks? Resources Microsoft Digital Defense Report:  https://www.microsoft.com/en-us/security/business/security-intelligence-report  Nazmus’sLinkedIn https://www.linkedin.com/in/nazmus-sakib-5aa8a6123/ Bhavna’s LinkedIn https://www.linkedin.com/in/bhavna-soman-3004b613/Nic’s LinkedIn https://www.linkedin.com/in/nicfill/  Natalia’s LinkedIn https://www.linkedin.com/in/nataliagodyla/  Microsoft Security Blog:  https://www.microsoft.com/security/blog/ Transcript(Full transcript can be found athttp://aka.ms/SecurityUnlockedEp11)Nic Fillingham:Hello, and welcome to Security Unlocked, a new podcast from Microsoft where we unlock insights from the latest in news and research from across Microsoft security, engineering, and operations teams. I'm Nic Fillingham.Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft Security, deep dive into the newest threat intel, research, and data science.Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft Security. If you enjoy the podcast, have a request for a topic you'd like covered, or have some feedback on how we can make the podcast better-Natalia Godyla:Please contact us at securityunlocked@microsoft.com or via Microsoft Security on Twitter. We'd love to hear from you. Natalia Godyla:Welcome to the latest episode of Security Unlocked. Welcome back to civilization, Nic. I'm hearing that Seattle had a pretty bad windstorm. Glad you're okay.Nic Fillingham:Thank you. Yes we did. We were out of power and internet for best part of two days. That was fun. But yes, we're back online. We have power. We have internet. We're back in the 21st century. How about you, Natalia? Any insane weather events up in the northeast? You guys get ice storms and cats and dogs and locusts falling from the sky, don't you?Natalia Godyla:None this weekend though. I did almost freeze going camping and I had a close call with an attack over the weekend.Nic Fillingham:Oh my gosh, that sounds crazy. What happened?Natalia Godyla:I mean, it happened in Outward. I feel like I probably should have started with that. But Outward, the game.Nic Fillingham:Oh, okay. Phew. I feel like you would have mentioned that to me in advance of recording this podcast had you actually been attacked in real life. What's this game? What's the game you're playing?Natalia Godyla:It's an RPG game where you try to quest through this... Gosh, I don't remember a single name of any of the locations, the cities, or the mountains. I'm not paying attention. I'm really focused on the battles that you have to fight.Nic Fillingham:What are you battling? Can you give something away or is it a spoiler? Is it humans? Is it animals? Is it zombies? Is it aliens?Natalia Godyla:It's a mix. There are bandit camps and then there are troglodyte caves. I think I've taken on a whole lot of the troglodytes at this point though. So I don't know if they're still in existence.Nic Fillingham:Let's take 30 seconds to look up Outward. You said troglodyte, and I really feel like troglodyte is an established word that means something. Oh, okay. So troglodyte is from the Greek, troglodytae, which literally means cave goers. Is that right? Do they live in caves?Natalia Godyla:They do live in caves.Nic Fillingham:Oh, there you go. Okay.Natalia Godyla:This game must have done its research.Nic Fillingham:They're cave goers, but they're also your enemies. Is that right?Natalia Godyla:Yes, but I guess in theory, I brought it upon myself. I mean, I kind of wanted to loot the cave.Nic Fillingham:So you actually went into their territory and were like, "I'm going to smash this jar and get this green jewel out of it." And they were like, "Hey."Natalia Godyla:Yeah. I mean, that's a moral gray area because they saw me and immediately attacked but it was their cave.Nic Fillingham:So you're the bad guy. Nice. All right. We're going to play this. We're going to play Outward. Wonder if we can get all of the Security Unlocked peeps into a single game. That'd be fun.Natalia Godyla:Oh, yes. And I think with that, we can intro our guests. Yeah, there's no connection point here.Nic Fillingham:Speaking of cave-Natalia Godyla:Looting.Nic Fillingham:Looting? No.Natalia Godyla:How do you stop looting from happening?Nic Fillingham:Oh, got it. I got it. If only those troglodytes had better security, people like Natalia Godyla wouldn't just come wandering in to ransack the place looking for leather and iron ore to craft rudimentary weapons. Speaking of better security, today on Security Unlocked, we talk with Nazmus Sakib who is going to spend a bit of time talking to us about firmware and the challenges associated with ensuring firmware integrity and the integrity of device security all up starting with firmware. This is going to be the first of three conversations that we'll have over a number of episodes where we better understand the security of devices from the firmware up. And then after that segment, Natalia, who do we speak with?Natalia Godyla:After that, we speak with Bhavna Soman who is a senior security research lead at Microsoft. And she shares how she got into security, which was really a role that she played in her family. She was the de facto support for network and security issues like antivirus. And as she continued in that role, she got more and more curious and tried to understand what technicians were changing or why something might be affecting her computer. And that role and responsibility just made her that much more interested in the security space and eventually led her here to Microsoft where she works on understanding and getting insights from the data that we have in order to better inform our defender products. Onto the podcast.Nic Fillingham:Onto the pod.Nic Fillingham:Welcome to the Security Unlocked Podcast, Nazmus Sakib or Sakib as we'll call you in this podcast. Thank you so much for joining us.Nazmus Sakib:Thanks, Nic. Thanks Natalia for having me. It's a pleasure to be on here.Nic Fillingham:So two things. We love to start up these deep dives with an introduction. Would you mind explaining? I introduced you as Nazmus Sakib, which is your name. We're going to call you Sakib. Just anything you want to sort of say about that, but also, what do you do at Microsoft? Tell us about your role, the team you're in, the team's mission? What is your day-to-day like?Nazmus Sakib:Yeah. I'm Nazmus Sakib. I go by Sakib. It's usually a sign on the team that you've met me where I get to clarify that growing up, everyone just called me by my last name. I'm originally from Bangladesh and Sakib is just more common as a first name in Bangladesh, which is what most people... My family ended up calling me. There's a famous cricketer by the name of Shakib Al Hasan who some listeners may be familiar with, but this is my first foray into fame.Nic Fillingham:I am familiar with famous Bangladeshi cricketers. Thank you very much.Nazmus Sakib:He's finally back after an unfortunate ban, but I think it's great to have him back on the team. Super excited for the prospects of the Tigers.Nic Fillingham:Do you play cricket? We're going to do this. We're going to take the little party.Nazmus Sakib:Yeah. Let's go down fully on that rabbit hole. So I played a lot when I was younger. I've been in America mostly since 2008, is when I first came for college. But prior to that, like most I think kids in Bangladesh, we play cricket. And usually, I grew up in Dhaka, which is the capital. So it was all improvised for the longest time. We had a little space on our roof. So it was like this flat essentially. And so it was probably about maybe 10 feet by 10 feet or not even. And so me and my cousins be a team of like two or three kids and we'd split it up. Someone would bat, someone would ball. You'd make up the rules in terms of how the runs would work. And same thing with if you find a little space in a back alley, or in any small sort of field or space that you'd get, you'd find a way to make it a cricket field. So good memories from back there. So it was kind of informal, but a lot of fun, especially now that the years have sort of gone on and I'm in a much different place where you just don't do that. It's pretty cool memories.Nic Fillingham:Bring us back to your role here at Microsoft and sort of what you do. Can we think of a good cricketing segue? Is there any famous cricketers that have moved into the cybersecurity field? What's a hard left turn?Nazmus Sakib:I think Satya is obviously-Nic Fillingham:Oh, yes, Satya loves cricket. He's a big cricket fan.Nazmus Sakib:Satya loves cricket, yeah. So I guess he's the most famous former cricketer turned tech luminary that I can think of.Natalia Godyla:10 points for the connection there.Nazmus Sakib:So yes. It is a well worn path, cricket to Microsoft. And I'm just one more traveler on that road. But my day-to-day, I've been at Microsoft for a little over eight years now, actually right out of college. I work as a PM in one of the many security teams at Microsoft. My team currently is in the Azure Edge and platform team. Our team is responsible for the operating systems that we ship as part of Microsoft, and also that operating systems that our customers use on platforms like Azure. So our team has been responsible for building the security that goes into Windows for a long time. Been a part of that team since I started at Microsoft.Nazmus Sakib:And then with the way to serve our customers on Azure, we want to meet them where they're at. And we have a lot of Linux customers on Azure as well. And so increasingly, our team is not just doing Windows work. We're also investing in Linux security technologies to help ensure that if you're a customer coming into Microsoft, if you're using Azure, whether it's on Windows or Linux, really bringing that platform, that operating systems' expertise to help secure whatever it is that you're you're trying to do. Nic Fillingham:Awesome, thank you. I'm really excited for this conversation we're about to have. It's going to be one of sort of three. I won't call them introductory, but it's certainly a little trinity of conversations over the next few months where we're going to talk about firmware. We're going to talk about firmware integrity, the challenges of that, and how you go about ensuring and securing firmware integrity. We're going to follow that up in a future episode talking about the Microsoft Pluton announcement. I'm sure that'll come up at some point in our conversation today. You're joining us today, Sakib, to help us sort of come back to basics a little bit. Can you help orient us in this world of BIOS, UEFI firmware, all the various sort of synonyms for this stuff? We're going to talk about firmware. Let's talk about what is firmware. Let's talk about these acronyms. If you would, just sort of re-educate us so we can start the conversation.Nazmus Sakib:Right. So the easy way to think about firmware is it's the first piece of code that runs on your hardware, right? So it's easy to sort of visualize that when you have a device, it's a desktop, or a PC, or a phone, any kind of computing device, you have the actual hardware, right? You've got the CPU, the motherboard, the power button that you use to turn the whole thing on, you have the hardware. The firmware is really essentially software that's typically baked in to the hardware. So it ships typically as part of the hardware. There's usually some read-only memory chip that's dedicated to storing that firmware just so that when a customer hits the power on button, the hardware knows how to turn everything on essentially. It's the firmware, that piece of software that actually goes and coordinates how devices are being made available to all the other things that run after the firmware, which is the operating system, and then the applications that you use on top of the OS.Nazmus Sakib:So if you were to think about from the point that you turn on a device to the point where you're using an application, whether it's your browser, whether it's Teams or Zoom because it's COVID, usually a very simple workflow for that is you're turning on the hardware. The firmware is the first piece of software that runs on the hardware platform. It bootstraps the operating system. So it could be Windows, it could be Linux. And then after that, once you have the operating system running, you can run applications like your browser, Teams, Zoom on top of that operating system platform. Nazmus Sakib:So the second part of your question, what is BIOS or UEFI? They're essentially flavors of firmware. BIOS has been around for the longest time, I think, in many ways with the history of the IBM PC. The BIOS was what you'd call essentially the firmware that ran on an IBM PC platform. A few years ago now, I think, essentially, the industry got together to revamp the firmware standards. So it's both a specification and an implementation of that specification. So EUFI, you can think about it as the modern BIOS, but because historically, people called firmware BIOS for the longest time, they're almost essentially synonyms. But typically, BIOS and UEFI both refer to the firmware that runs on any particular platform. And in general, they're perhaps used synonymously if we're speaking loosely. But most modern systems today use some implementation of the UEFI specification as the platform firmware.Natalia Godyla:Can you provide some security context around firmware? What does the threat landscape look like for BIOS or the broader term firmware? What's been the history of attacks? What's more or less prevalent for firmwares compared to applications that are at risk?Nazmus Sakib:Right, right. So much work has gone in to so many different parts of the technology stack, right? You think about the work that we've done at Microsoft and across the industry around things like antivirus solutions. You look at modern platforms like Microsoft ATP, Advanced Threat Protection, where you have just a view of the health of your operating system across many devices that's customized for your enterprise. All of those things, in many ways, have already made it harder and are increasingly making it harder for attackers to do things that they would have maybe gotten away with in the past for attacks in the operating system.Nazmus Sakib:And so naturally, when you make one thing harder, you incentivize attackers to go elsewhere, right? And so what we saw as a trend and one of the places where this was really sort of evident to us in a way that felt it wasn't just us looking at it, it was also externally reported is if you look at the NIST which is the American standards body, essentially, the National Institutes of Standards and Technologies, I think, I'll have to go verify that, but they actually maintain the National Vulnerability Database. So if you think about vulnerabilities that get reported, you see in the news and they often have some numbers associated with it. That's actually all the numbers in the National Vulnerability Database. Nazmus Sakib:And so one of the things that you saw in the research that's being done in the industry, this is where all the security researchers report issues. It's like the aggregate. This is how the industry keeps track of all the vulnerabilities that are happening across all technologies. There was a large spike in firmware. If you just go to the NIST website and you go type into firmware, it went from a handful of firmware vulnerabilities being reported in, I think, 2016/2017 to hundreds being reported in the last year or two. And so a huge spike beyond exponential. And that really is because we're making it harder to do the things that perhaps attackers would be able to do in the past and the operating system. And so people are naturally moving elsewhere. And so they're gravitating towards firmware as an avenue. So that's one reason.Nazmus Sakib:The other reason is coming back to what I was talking about in terms of how a platform boots. Firmware, because it's the first thing that runs on your hardware, because it needs to, just by its very nature, set up your hardware in the right configurations, it actually bootstraps a lot of the security on your system. Right? And so it's almost like a double whammy. Attackers are moving to a place where a lot of the problems that have been solved in the operating system from a security perspective, they're trying to work around those protections. And then in firmware, they actually see that you have this highly privileged environment firmware typically has almost usually when it starts up, almost unrestricted access to all the hardware and the data that's on your hardware. And so that's really where we're seeing this trend where attackers are... the security researchers suggesting that attackers are going to be moving there. Nazmus Sakib:And one very recent practical example of a threat where these trends are bearing out is just, I think, last week, there was a report that TrickBot which is almost like a modular malware that's being used in a lot of other ransomware attacks, it's actually added firmware capabilities. So it's using other longstanding well-known vulnerabilities in the operating system, but because of the trends I've just described, we're seeing TrickBot add new firmware attack capabilities as well.Nic Fillingham:Sakib, do we know when firmware attacks begin? Is there a defining moment in time when firmware became an actual viable target? Or has it sort of always been there and it's just recently evolved?Nazmus Sakib:It's always been there. I mean, firmware is always run with high privileges in a way that it may be difficult for operating system software, including security tools, to tell what's going on in firmware. It's easy for firmware malware to hide what it's doing. But if I were to think of a tipping point, if you will, a couple years ago, we saw that at least one example of what's typically associated with a particular nation state threat actor. There were targeted attacks a couple years ago that were using a firmware vulnerability. So in some ways, that was a very clear signal that not only is the security research headed that way, but there's at least that first example. It's almost like the canary in the coal mine, if you will, where we saw an example of an attack that tried to do exactly what I described, is use for a very targeted attack, use firmware to circumvent a lot of the security tools, and find a way to persist. Nazmus Sakib:And with developments like what I talked about for TrickBot, which is generally often used by many different actors trying to orchestrate different ransomware attacks like Ryuk and Conti, we expect to see that trend sort of increase. And so if I were to think about that first tipping point where attacks start to become real, the LoJax attack is, I think, what it's typically referred to as maybe the one I can think of where it really sort of became not just a trend we're seeing in the research, but a really practical attack. Nazmus Sakib:By its very nature, firmware is complex. There's tens of thousands or millions of lines of code running if you think about all the firmware that runs on your system. So if you just think about the basic security principle of trying to reduce your attack surface, trying to have lease privileges, what you really want to be able to get to is that your trust is not necessarily fully dependent on all the firmware being written totally correctly and totally secure and not vulnerable to an attack. Ideally, you want to not trust that huge infrastructure. You want to be able to go do that trust of fewer set of things. And that's sort of the journey that we've been on recently with our OEM partners as well with secured-core PCs is to do that evolution. A UEFI secure boot doesn't go away. It's still an important technology. But we want to be able to start layering on additional capabilities that can start to protect important security properties or security capabilities even from firmware compromise as that's really where the trends are going from an attacker perspective.Natalia Godyla:So your team has done a lot of great work around secured-core PCs. What would it take for an attacker to actually break into one? Is it possible? What do they have to overcome?Nic Fillingham:Without obviously giving away some operational security here, but just like in Bizarro fictional land with infinite compute power and physical access to the device, what are the monumental challenges that would need to be overcome?Nazmus Sakib:There are a couple places that I think are interesting that we're definitely thinking about. Security is not a static thing. It's always dynamic. We do something and then so do attackers. And so if you think about... It comes back to maybe the foundation analogy. We are building a lot of our security promises on things like the TPM. We want to be able to securely record the firmware that's running so that we can actually tell that it's the firmware that we expected. Right? So that's an area that we're thinking hard about and it's part of the motivation for Pluton. I'll leave it up to you all to interrogate Peter around what the effects are, but I think that's one place where a lot of our security promise is built around that.Nazmus Sakib:We spend a lot of time thinking about TPM attacks. And it's a big part of the motivation for why we're adding another choice to the Windows ecosystem around using Pluton, is just being able to continue to raise that bar against attackers. So I'll leave it to you, Nic and Natalia, to interrogate Peter as to how Pluton will help with the security of future Windows systems.Nic Fillingham:We'll absolutely do that. So Sakib, thank you so much for your time. As always, we will have some notes. We'll have some links in the follow-up show notes. And I'm not sure we've actually offered this to listeners before, but if you do have questions about securing firmware, anything that Sakib talked about, contact us on the Twitters. You can send us an email, securityunlocked@microsoft.com, and we'll do our best to point you in the right direction. Thank you much, Sakib.Nazmus Sakib:Yeah, no. Definitely thank you for having me on here. It's just a great competition. I enjoyed it. And I second what you just said. We'd love to hear from listeners around things that we can do a better job of communicating or feedback folks have on how well we're doing in terms of meeting their needs. Nic Fillingham:Sakib, thanks so much for your time, mate.Natalia Godyla:And now, let's meet an expert from the Microsoft security team to learn more about the diverse backgrounds and experiences of the humans creating AI and tech at Microsoft. Today, we have Bhavna Soman on the episode. Thank you for joining us.Bhavna Soman:Thanks for having me, Natalia and Nic. I'm very excited to be here right now.Natalia Godyla:We're excited to have you. So love for our audience to get to know you a little bit more. What is your role at Microsoft? What does your day-to-day look like?Bhavna Soman:Yeah, absolutely. So my official title is senior security research lead. But like it often happens in big organizations, it kind of doesn't accurately reflect what I do. I lead a team of security researchers and data scientists who use machine learning and AI to fight threats on the Microsoft Defender platform. And that kind of reflects my own background as well, which has been checkered with experience in security research and machine learning. So to me, that's a very good fit even though I can't get them to include all of it in my title.Nic Fillingham:Bhavna, we've spoken to a few of your colleagues on the podcast already: Holly Stewart, Jeff McDonald recently, Karen Lavi. How would you describe what you do? What is different about your role and your team compared to maybe Jeff's team or Karen's team, et cetera, et cetera?Bhavna Soman:Yeah, absolutely. So the focus for my team is on using AI and ML on building intelligence and context for our enterprise customers. So when you look at how you want to apply machine learning in data science, I think it all really boils down to how can you reduce the dependency on human beings who have the security expertise? How can you bring in AI to help enterprise customers better defend themselves in this field that has a scarcity of talent, to be honest? And so what they do is look for clean or malware files. Whereas my team is focused on providing, for example, information about emerging campaigns or information about, what are the attacks that are linked to each other and form one incident so that an organization can address them together as a whole and therefore get efficiencies from that analyst as well?Bhavna Soman:So these are just a couple of examples of what I mean when I say like we provide the intelligence. So I think someone put it very succinctly a few weeks ago where Jeff's team finds the badness, Karen's team finds the goodness, and I kind of bring it all together and give it meaning.Natalia Godyla:That's awesome. I love that definition. Nailed it. And stepping back for a moment, I'd love to hear about what brought you to Microsoft and what brought you to security research. As you mentioned, you had a journey that included machine learning and security research. So how did both of those come into your career path?Bhavna Soman:So I was always excited by security. And even from a very young age when we had our first laptop, which was like way, way back. I think it either had Windows 95 or 98. So it was really old. And those days, you get infected by stuff all the time. So for my family, it used to be my job to kind of figure out exactly where was the registry key in which this thing had saved its autorun tactic or persistence tactic. And at that time, I didn't know what any of these were called or anything. But that's how I first got into it. And then I decided that I really loved this sort of adversarial aspect of security. It really brings an excitement to the whole thing for me. Bhavna Soman:My path did not take me directly to security still. My undergraduate studies were in mechanical engineering. So thankfully, I got a fair bit of math and also programming classes in, but I was chasing different things at that time. But after a while of working in that space, I was actually doing pipeline design for this company that constructs oil refineries, which was a very soul-sucking job for me. Yeah. I didn't like it at all. I did that for two years after college, and it just was not for me. So I was like, "Okay, I really love computers. I have to go in that direction." So I started to build software tools for that company. And then that gave me sort of this way to dip my toes in. And then I realized that, okay, this is definitely something I love doing. So I decided to go for masters. Bhavna Soman:And then when I was choosing my area of focus for my masters, I was like, "Yes, security has to be it." So I went to Georgia Tech to do my masters and I specialized in security. So that gave me a great sort of grounding and all of the basic skills, a great background at the industry. And Atlanta has a very good infosec community too. So I had the chance to get plugged into that. Yeah. I really loved going there. And after my education there, I worked for this startup out of Georgia Tech, which incidentally specialized in using machine learning for network security. So that's where I think I got introduced to, hey, machine learning and artificial intelligence can have something to say about this. Bhavna Soman:The more I stayed in the security industry, this problem of how it's all a whack-a-mole where a few people are chasing thousands and millions of different variants of the same attack. It really impressed on me that this is not something I can do manually. I can reverse 10, 15 samples. I can't do a thousand. So that's where the power of AI and machine learning really struck me. So I think that's where I started going deeper and deeper into that.Nic Fillingham:I wanted to come back to something that you touched on about being the family... What did you say? When a virus came on the computer, you would be the one that would be in charge of getting it off? Is that correct?Bhavna Soman:Yeah. Yeah. So at that time, I think they weren't super severe viruses. They weren't doing human operated ransomware stuff. For instance, they'd show you annoying pop-ups or they would change your search engine all the time. And they were doing very annoying things like that. I took on the task of investigating, how exactly is this thing coming back even though I deleted it? And then I started to discover the hidden mode in Windows and I started to discover all of these registry keys and rededit. It kind of went deeper and deeper and deeper from there.Nic Fillingham:Got it. Were these in the days where you could just install as many toolbars as you wanted inside your browser to the point where you could no longer see a web page? Are we going back that far?Bhavna Soman:Yeah, yeah. It was one of those days where... And also, Google was not really a thing. I remember Yahoo chat rooms used to be the big thing.Nic Fillingham:AltaVista, baby. AltaVista.Bhavna Soman:So fun times. There was a simpler world for sure.Nic Fillingham:Bhavna, how long have you been at Microsoft now?Bhavna Soman:It's been three and a half years now.Nic Fillingham:Got it. And and the first role that you came into at Microsoft, was that in the team that you're in or was that in a different group?Bhavna Soman:It was still with Microsoft Defender, but I was doing slightly different stuff. I was focused more on just pure security research and not as much on the machine learning and AI aspect.Nic Fillingham:Three and a half years ago, what were you focused on? And how has that sort of potentially evolved? How has that changed today? Were you still focused on the same types of attacks? They've just sort of evolved in sophistication. Or was it a completely different world three and a half years ago?Bhavna Soman:So when I first came to Microsoft, I was coming fresh off of Intel. At Intel, my focus had been on threat intelligence. Again, this was back when threat intelligence was just starting to become a thing. So I joined Intel before that. And at that time, they needed a threat intelligence platform where you can gather all of the TI information from all these feeds: internal, external, et cetera. So I built that first platform, plugging it into all the internal/external data feeds, organizing the data, and then having that pumped into the various prevention and detection systems. So that's what I was doing primarily at Intel. So when I came here at first, I was still in that mindset, and I was still trying to apply intelligence to improve protection. So I was doing a lot of hunting on VirusTotal, kind of try to find out where our biggest gaps were, and trying to plug those. Bhavna Soman:But very quickly, that pivoted to using machine learning for security was focused on non-PE files. So very heavily focused on the document files that we very often see come in as email attachments, and then they will lead the user to download something actually bad like, again, an Emotet or Dridex or something. So it was very focused on those macro files and other non-PE files. JavaScript was a big one at that time. So writing classifiers to differentiate between malicious JavaScript and the benign kind. Those were some of my first projects here.Natalia Godyla:So you said a couple of times that the draw of machine learning for you is the potential for scale, the potential for helping to fill that skills gap. So as you're shifting into roles where machine learning is playing a bigger and bigger part, what are the achievements that you're focused on? What would you like to try to automate better so that humans can shift to other tasks?Bhavna Soman:So there is one problem, which is very close to my heart. And that is the problem of the core threat intelligence business. So Microsoft Defender has a really big threat intelligence team. And this was something... I was part of the threat intelligence team at Intel as well. And all through my time working with these teams, it's been obvious that threat intelligence is very manually driven right now, right? It has to be a human that is reading files or PDFs or white papers. And then this human is, again, observing traffic data whether by hunting or through the attacks that they are remediating or something like that. So this human is then kind of assimilating all of these insights that they have about these attackers. And then they put it out somewhere. Like maybe they will communicate it to their customers saying, "Hey, this is what you need to be careful about." They may write a white paper or they may do detections as a result of that. So this is a very human thing. Bhavna Soman:And when I look at artificial intelligence and machine learning, to me, using large amounts of data to extract a few critical insights, to me, this is a very good use case for machine learning and AI. So this is a problem that I have been working on for a really long time. My first attempt at this was while I was at Intel, and I did this kind of cross-team project with a team that was in Argentina at that time to work on a method that could use question answering techniques from machine learning to answer questions about attackers. So if I had a question about, "Okay, what is the tool that this attacker uses? Or what is the victim vertical for this attacker?" Can I use question answering techniques and train on the corpus of data available about these attackers and have an AI-based system give an answer? Bhavna Soman:So I've been attacking this problem for many years. My first attempt while I was at Intel was not very successful. But a couple of years ago, I gave it another shot. And this research ended up being... I presented this at Black Hat last year where I was talking about how we can use some new techniques that had come out since then around word embeddings, natural language processing, and domain specific named entity extraction to do similar stuff. So I think I've been making progress on that problem. And now I'm working on a project with University of California, Berkeley on this security AI RFP where now they're expanding some of this work into the security knowledge graph where their aspiration is even bigger. Yes, we grab all of this data from a variety of different data sources. Yes, we do named entity extraction. But what else can we do on top of that? Can we automatically build, for example, YARA signatures based on this? Can we use multiple data sources to achieve consistency internally within this graph? Bhavna Soman:So that's where we're seeing AI and machine learning will take threat intelligence and help it become a little bit less manual, and again, less dependent on manual expertise?Natalia Godyla:What challenges are you facing with achieving some of the goals you've outlined? I'm assuming compute is always something that's in the back of your mind. What else would be a barrier to potentially achieving some of these successes? Or what are you tackling right now to reach your goals?Bhavna Soman:That's a great question. Compute is a big one because on one hand, we have large amounts of data. But on the other hand, A, to process all of that in a deep learning style would take huge amounts of compute that would make our product run very inefficiently on our clients and in organizations' machines. So usually, that's not feasible, which is why one of our big focuses is to find efficiency in whatever techniques we're using so that the model can be lightweight and yet perform with similar degrees of precision and recall. Bhavna Soman:Another big challenge we face is good labels or ground truth. Just because the spectrum of badness is so huge, on one end, you have these just adware things are grayware things that their whole goal might be to show advertisements or cause pop ups. And on the other end, you have APT threats. So in this wide spectrum, we have to find good labels for a large enough set for each particular category so that we can accurately classify threats and inform users about that. That's been a very interesting problem too. Going back to the threat intelligence space, one really huge challenge is that the field is continuously evolving. A particular thing might be used for human operated ransomware on day one, but on day 30, it's hosting some random adware or some software bundle or something. So within that span, even in shorter spans, the situation really changes. The intelligence you have really changes. So all of your machine learning systems have to be able to constantly getting the latest information adapting to that. So those are some of the big challenges we face in this field that we're trying to work around.Nic Fillingham:Bhavna, one of the questions we like to ask on the podcast is, what from your personal life, whether it's a hobby, whether it's something growing up as a kid, whether it's education or previous job, do you bring forward into your current job that could be considered maybe unorthodox? You teased very early on that maybe you play D&D. Is that true? Bhavna Soman:Yeah. I play video games or board games. I'm into all of that.Nic Fillingham:Is that a passion for you? Do you find yourself bringing any game theory or the way that you would approach a D&D encounter into your day job?Bhavna Soman:I think my biggest influence is books and language. I have been into books as far as I can remember. That was my favorite birthday gift when I was a kid. I just dragged my parents to the bookshop and buy a bunch of stuff. And a peculiar way in which humans use language and give meaning to it, to me, that is a source of endless fascination. Which is why one of the favorite authors for me is Patrick Rothfuss and his book, Name of the Wind. I think that book really talks about... It's a fantasy book. So it kind of goes into like if you know the name of a thing, then you have some control over it. It's a philosophical point, but also it says something about language. And in my mind somehow, all of that comes together and that really leads me into, how do machines interpret language? What does it mean for a machine to understand language? And when we're building all these natural language processing models, what exactly are we doing? And then what exactly are we missing from what human communication actually entails?Bhavna Soman:Which is why I'm kind of always drawn into this threat intelligence field because I'm like, "This is really where the importance of language and communication becomes connected to security." So that's kind of this one thing for me that I really, really love. In fact, one of the really cute examples that's always stuck with me is when you do a beginner course on natural language processing, you always kind of get this example. It's called crash blossoms. There was apparently a headline in the newspaper a long time ago where the headline said, "Violinist in Japan Airlines Crash Blossoms." And obviously, the headline meant to say that this violinist who was involved in this air crash a while back is now doing well. But when an NLP based system is trying to process it, it is like, "What is crash blossoms?" And I love that problem because it kind of emphasizes very clearly how machines are different from human beings, and yet how we're trying to bring the two closer for our own benefit.Natalia Godyla:I feel like one of the other unique points about language is just the evolution of slang. So I'll be curious to see how NLP processes and consumes slang because that is such a cultural moment. It depends on the cohorts of people that you surround yourself with, the social context. Bhavna Soman:Yeah, that's a great point. You talked about slang specifically where a meaning of a particular word or phrase can be different based on even the environment or the forum in which it is used. Certain terms, if you use it in an industry specific way, will mean very different than in the general sense. And we come across that in security so much, right? We have all these actor names like Scary Panda or Crawling Spider. And if you think of using like a traditional NLP model and all of this data, you're like, "This is not going to make sense because you're talking about a specific entity, an actor, not an animal." So we do have those kind of challenges in our domain. And I love diving deep into that.Nic Fillingham:So I have another sort of random question. I was possibly laying the ground for this with my previous question about, what from your hobbies do you sort of bring forward into your work? Your avatar, your photo in the Microsoft GAL in our sort of identity system is Megamind. Is that right?Bhavna Soman:That is absolutely right. I think that really ties into my sort of chaotic neutral rogue character because Megamind is a really good example of that, right? Supposed to be a villain but is a hero, but also is a villain in some ways still. This was actually a prank. We had Microsoft Month of Give last month. So your teammates could donate some money and force you to change your profile picture. So that's what I got.Nic Fillingham:Did you choose Megamind or Megamind was thrust upon you?Bhavna Soman:I chose Megamind. I was like, "Okay, this is the most appropriate for me."Nic Fillingham:Oh, so you do resonate with the Megamind character on some level?Bhavna Soman:I do. Yeah. I think so. And also, it's a really good movie that kind of has not had its time in the limelight for a while.Nic Fillingham:I don't know if I've seen it. I think my kids have seen it. That's sort of why I know it because I think I've sort of had to approve them watching the movie, but I don't think I've seen it. It's good, is it?Bhavna Soman:It is amazing. You should definitely watch it. It's a very cute movie.Natalia Godyla:I think we have our homework, Nic. I haven't seen it either.Nic Fillingham:Bhavna, before we let you go, is there anything you would like to plug? Any sort of organizations you're a part of? Any communities, groups? Anything you'd like to say out there to aspiring students of machine learning who either want to get into the field or just want to get better at machine learning?Bhavna Soman:I would love to. So the organization that I want to talk about is not associated with machine learning only. It's associated with security all up. So I am part of a group of women called BlackHoodies. And we are committed to increasing the participation of women in hard technical areas, which sometimes don't see as much participation from minorities. We are across the globe across many companies group. The only I think criteria is you are a woman, whatever your definition of that is, and it's always free. We hold classes at multiple conferences across the world which we'll do things like reverse engineering, Windows, ARM, web hacking tools like Ghidra, all of that. We have all these trainings that are completely free. And now that we are in the pandemic, we're doing some of these remotely. So please follow us on Twitter. And if you're interested in joining one of these trainings, it's super easy. And we really, really welcome anyone who wants to learn about this stuff.Nic Fillingham:As you were talking, I searched Black Hoodie on Bing and just got a thousand results for buying a black hoodie. What is the URL for the community group? I think I may have just accidentally purchased a black hoodie. I've got Amazon, what is it, one click buy. I went a little too quick. I was trying to pay attention to the recording window for the podcast and then searching for what this was. Anyway.Bhavna Soman:I hope it fits. So the website is blackhoodie.re. And we talk about all of the latest events or workshops that are happening there. Usually, when Microsoft holds Blue Hat, we'll do a bunch of trainings at Blue Hat as well. I do the beginners reverse engineering for x86 as part of that. But right now, we don't have in-person conferences, but we're doing virtual stuff.Natalia Godyla:That's great, Bhavna. I think one of our previous guests has also shared BlackHoodies. So thank you for highlighting it. It sounds like a great organization. And to our audience, please check it out. Thank you, Bhavna, for being on the show with us today.Bhavna Soman:Thanks for having me. It was super fun.Natalia Godyla:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us @msftsecurity or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe,Natalia Godyla:Stay secure.
1/13/2021

Tracking Nation State Actors

Ep. 10
Watchdogs in tow,hostsNic Fillingham andNatalia Godylaarejoined by guest RandyTreit, Principal Security Leader at Microsoft,toexaminethe process ofidentifyingthe source of athreatand stopping the spreadbyprotecting“patient zero.”Randyhas a fewkeytricks up his sleeve as a defender, butyoucan decideifthey’remoreimpressivethan theantics he and his identical twinhave pulled while working at Microsoft.In the second segment,Jeremy Dallman,Principal Program Manager at Microsoft,discusses why some bad actors are known in the security world under some of the most seemingly harmless codenames, such as “Fancy Bear” and “Charming Kitten”, and highlights the techniques his team is using to protect Microsoft’s customers from Nation State actors.In This Episode, You Will Learn:  How Microsoft is defending and protecting patient zeroThe history of Defender andantimalware The processoffinding gaps inprotections The importance of protecting customers from Nation State actors How and why security vendors use codenames to refer to threat activitygroupsSome Questions We Ask:What is different about focusing on patient zero than other aspects ofsecurity?How does Microsoft measure the false positive rate in protecting patient zero?What tools are being used on a day-to-day basis in defender security?Why does Microsoft partner with the industry to identify Nation State actors?How many groups are utilizing AI and MLto enhance their ability to become a threat?ResourcesMicrosoft Digital Defense Report:https://www.microsoft.com/en-us/security/business/security-intelligence-reportRandy’s LinkedInhttps://www.linkedin.com/in/rtreit/Jeremy’s LinkedInhttps://www.linkedin.com/in/jeremydallman/Nic’s LinkedInhttps://www.linkedin.com/in/nicfill/Natalia’s LinkedInhttps://www.linkedin.com/in/nataliagodyla/Microsoft Security Blog:https://www.microsoft.com/security/blog/Transcript(Full transcript can be found at http://aka.ms/SecurityUnlockedEp10)Nic Fillingham: Hello, and welcome to Security Unlocked, a new podcast from Microsoft, where we unlock insights from the latest in news and research from across Microsoft security engineering and operations teams. I'm Nic Fillingham. Natalia Godyla: And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft security, deep dive into the newest threat intel, research and data science. Nic Fillingham: And profile some of the fascinating people working on artificial intelligence in Microsoft security. If you enjoy the podcast, have a request for a topic you'd like covered or have some feedback on how we can make the podcast better. Natalia Godyla: Please contact us at securityunlockedatmicrosoft.com or via Microsoft Security on Twitter. We'd love to hear from you. Hey Nic, how's it going? Nic Fillingham: Hellow, Natalia. It's going well, thank you. Welcome to episode 10 double digits. It feels like a milestone. That's a milestone, right? Natalia Godyla: Heck, yes. I think we were proud of ourselves after episode two. So I feel like this feels a little bit more legitimate, a good start to 2021. Nic Fillingham: Great start to 2021. But we were talking, just before we started recording and this, some sad news. Natalia Godyla: Okay. So to listeners that had heard and loved our story about the Somerville Turkey, of course. The Somerville Turkey is no longer, so the Somerville residents had fed the turkey and the turkey became aggressive as a result. And it is no longer a hallmark of our city. Nic Fillingham: The problem was they fed the turkey pure creatin, that was the issue and Red Bull. Natalia Godyla: They didn't publish that in the news story, they're trying to keep that hash, hash. Nic Fillingham: That's why it got aggressive. But no, if you have no idea what we're talking about on our Thanksgiving episode, Natalia told us about a famous turkey in Boston that has a name and it's got an Instagram page or something like that, but unfortunately it's no more, it's pretty sad. Natalia Godyla: Now that the turkey is no longer, maybe we should memorialize it. Nic Fillingham: Ooh, so you're thinking we could potentially adopt the Somerville Turkey as our Security Unlocked mascot. Maybe we could create some kind of small statues, some kind of plush toy, is that where you're going? Natalia Godyla: For some reason, my immediate image was a butter sculpting contest, in which we sculpted butter sculptures of the turkey. Nic Fillingham: Hang on, what? So, I had said as a mascot and something, I think I said the word swag, at least it was in my brain. So something we could send to listeners, and so I just immediately jumped to the logistics of how do you send butter through the US Postal Service in an intricate shape, like that of a turkey? Natalia Godyla: Yeah. I don't think you should be taking my suggestions quite so realistically, I mean- Nic Fillingham: If we had to choose though, between memorializing the Somerville Turkey and our previous plan which was the mighty alpaca as our animal mascot, where are you leaning? Natalia Godyla: Alpaca. Nic Fillingham: Can we justify that from a security perspective? Is there any security link whatsoever from either a turkey, Somerville Turkey or an alpaca? What are you looking up? You're looking up something right now. Natalia Godyla: I'm looking up facts about alpacas because I have to be honest, this is purely on level of cuteness for me. Nic Fillingham: Okay. So our Executive Producer, Bruce Bracken has just [inaudible] in saying that, god llamas and god alpacas are a thing. So it says here that a god llama, alpaca or hybrid can be used in farming to protect sheep, goats, hens, or other livestock from coyotes, dogs, foxes, and other predators. Ladies and gentlemen, we have a winner. We now have a solid link from the alpaca to security. Well done everybody, congratulations, mission accomplished, we can go home now. All right, beautiful. Natalia Godyla: On a minimum, we can talk about our next episode. Nic Fillingham: Absolutely. All right, so let's table that. We've decided it is going to be the alpaca because the alpaca can be employed as a rudimentary guardian of livestock. But speaking of the podcast, on today's episode, first up we have Jeremy Dallman joining us from the MSTIC Group. I'm not going to explain what MSTIC stands for because Jeremy will talk about it. And it's a great start to the conversation. Jeremy is coming on to talk to us about the nation-state section or chapter in the Microsoft Digital Defense Report, the MDDR, this is the third of five conversations that we're going to be having on Security Unlocked, where we deep dive into some of the topics covered in that report. Nic Fillingham: This is also I think, the first time that the MSTIC team have compiled a lot of their nation-state tracking activity over a sort of 12 month period into a single report. So first of all, it's a great read, make sure you download the report, aka.ms/digitaldefense. And then, it's a great conversation with Jeremy who really helps us sort of understand some of the core principles and ideas around sort of why is Microsoft in this space, and then sort of what does Microsoft do with tracking nation-state actors. And then after Jeremy, we talk to- Natalia Godyla: Randy Treit, a Principal Security Researcher at Microsoft, a long time employee at Microsoft who has seen a lot of different groups and brings that expertise to his security team today. So we're talking to him about his path to security and he is another security professional who doesn't have a formal or standard path to security. So he doesn't have a formal education. And I think it's a good testament to the fact that so many security folks are autodidactic and just have a love of technology and find themselves continuously passionate and interested in it, and eventually get to do their passion for a job. Nic Fillingham: On with the pod? Natalia Godyla: On with the pod. Nic Fillingham: Jeremy Dallman, welcome to the Security Unlocked podcast. How are you doing?Jeremy Dallman: I'm doing great guys. Thanks for having me. Nic Fillingham: Thank you so much for coming on the podcast. This is one of several conversations we're going to have with folks that have contributed to the Microsoft Digital Defense Report that was released in September of 2020. Jeremy, thanks for coming on. You're going to talk to us about chapter two, which is the chapter that talks about nation-state threats. This is going to be a fascinating conversation. I'm really, really interested and excited to hear what you've got to tell us. But can we just start a little bit with, who are you? What's your job? What do you do at Microsoft? What does your day-to-day look like?Jeremy Dallman: Sure. So let's see, in Microsoft terms, I'm a Principal Program Manager, in the Microsoft Threat Intelligence Center. We call ourselves MSTIC. So I'll probably use that term off and on throughout the conversation, it's much easier to say it than Microsoft Threat Intelligence Center. As a Program Manager in MSTIC, I am responsible for, let's see, directing a large number of projects that kind of span incubation and driving threat intelligence [inaudible], both in MSTIC and across Microsoft.Jeremy Dallman: I do things around building and creating strong collaboration partnerships across the security industry, because most as actors, like nation-state actors don't just target Microsoft. I also work on sourcing the best possible tooling for our analyst and managing all of our public facing messaging around MSTIC and the threats that we track. So I guess in general, my role is always looking for ways to improve how MSTIC protects our customers, making sure that the analysts are successful and effective at hunting. And making sure that MSTIC knowledge outside the company is communicated effectively to protect our customers and enable better protections across the ecosystem. Nic Fillingham:I have ask, is MSTIC a backronym? Did you guys get in a room and say, "How can we come up with the coolest acronym in the company, and then make it work for what we do?"Jeremy Dallman: There's actually a couple of others I think, that are cooler, as well though. Nonetheless, no, our GM is notorious for let's just say, obscure acronyms that translate into words. So it took a little bit of effort, it took a little bit of time, but we came up with Microsoft Threat Intelligence Center and M-S-T-I-C pronounced as MSTIC. So we worked through a few other variations, but I think this was the best one that came out and it seems to have stuck. Nic Fillingham: I think there needs to be an offshoot team for analytics and learning at the end. Does anyone get that-Jeremy Dallman: Yes, Nic. Yes, yeah. Nic Fillingham: Okay, good.Jeremy Dallman: [crosstalk] I know a couple of people on the analytical side that might actually run with that, I might have to jot a note down. Nic Fillingham: There you go, you can have that one for free, no royalties from me, that's fine. Natalia Godyla: The next one's charged, though. Nic Fillingham: The next one's not free, this first one's free. So Jeremy, you're going to walk us through chapter two, the nation-state threats, it's a pretty lengthy section of the MDDR. It's also, I think, correct me here or [inaudible 00:09:14] me here, this is the first time that we've done sort of an annual wrap-up of what Microsoft has seen on the nation-state space. I think obviously, we've had lots of blog posts and activity over the many years on the activity, that we've seen and sort of how we've contributed to it. But previous sort of security intelligence reports didn't really include a lot of nation-state activity. I mean, correct me if I'm wrong here, but is this sort of the first time that we've done an annual look back at what happened in the nation-state space?Jeremy Dallman: Historically, our team hasn't been very publicly outspoken and we haven't really, historically didn't spend a lot of time talking about what we've done externally. So this is definitely unprecedented and something that's brand new for our team. It's kind of along the lines of what we've been doing over the last couple of years, talking a little bit more publicly about threat actors and such. So I think this is a fantastic roll up in view of what we do. I think it goes along with our expansion of MSTIC as an organization and kind of what we've been trying to do, informing our products and customers more broadly. Natalia Godyla: So Jeremy, why does Microsoft do this work? Why do we partner with the industry to identify nation-state actors?Jeremy Dallman: Sure. I think the short version is that Microsoft customers using our products are often the target of nation-state actors. And those customers expect Microsoft security products and Microsoft to help protect them from those threats. So MSTIC tracks nation-state activities to protect our platforms, to protect our services and protect our customers from those more sophisticated threats. Nic Fillingham: So, Jeremy, I've got the report open here in front of me and for those playing along at home, you can download the report. It's the Microsoft Digital Defense Report @ aka.ms/digitaldefense. And if you scroll down to page 44, there is a really interesting sort of graphic here. It says, "The sample of nation-state actors and their activities." And there's a bunch of what look like sort of chemistry symbols from sort of the periodic table of elements with a lot of chemistry names and symbols. And then there's some sort of other things as well. Can you sort of walk us through, what are we looking at here? Is this actual sort of nation-state actors and sort of how they're referred to? And the names that are being used to refer to them?Jeremy Dallman: Across the security industry, a number of different security vendors use different code names to refer to sets of activity that are tied to certain actors or sets of activity groups. So we use code names because we can't always necessarily tie that to a specific country, or we may want to do attribution. Other security vendors will use kittens and tigers and bears, some use numbers and a variety of different code names. And at Microsoft and in MSTIC, when we were trying to figure out how we were going to do code names, we tried a bunch of different things. I think initially, there was some use of dinosaur names, that got fairly complicated and hard to pronounce fairly quickly. I think we played around with a bunch of other things. At one point, I recall we were looking at flavors on the beer flavor wheel, I'm not sure there was enough of them.Jeremy Dallman:So we played around with this a little bit and we ended up basically at periodic table of elements because there's not really a licensing violation there, so we didn't need to worry about that. And there was plenty of them and they were fairly unique. So we code name our actors by elements in the periodic table. And we will name an actor, an element, once we understand that actor has a unique set of activity. But on that page 44 in the report is a summary of a few of our key activity groups via their element names. And largely focusing on the four regional sets of actors that we, and most threat intelligence teams will focus on, Iran, China, North Korea, Russia. Nic Fillingham: And is there any sort of logic to the particular element that's chosen? I mean, I noticed that there's no hydrogen, there's no oxygen. Well, they seem to be up towards the top end of the periodic table. I've never even heard of-Jeremy Dallman: Yttrium? Nic Fillingham: Yttrium? [crosstalk] Did Kanye West come up with that one? What's that?Jeremy Dallman: No, it's kind of funny because we actually have an individual on our team over in our UK office. She's responsible, she's our librarian, is kind of the role that she plays and she is responsible for naming. So I don't think there's any specific logic or pattern to who gets what name. I don't even know if our analysts have a say in picking any of the names, but our librarian is the person who basically gives these names out. And I don't think she has any set structure or method for picking the names. Nic Fillingham: I was really hoping you were going to say there was a periodic table of elements stapled to the wall, and then you had to start with dots. Natalia Godyla: Somehow, I knew I was going to be dots.Jeremy Dallman: You know what? [crosstalk]. I honestly would not be surprised if that was actually the case, but I can't verify that. Nic Fillingham: All right. Well, that's for another episode of the podcast for us to follow up on. Natalia Godyla: So can you provide a little bit more context on the players? What do we know about them? Their motivations? Infrastructure?Jeremy Dallman: Sure. So a number of these actors are pretty well known. When you talk about kind of the more popular, more widely discussed actors, it's kind of hard to not fairly rapidly, get to Strontium, which others refer to as APT28 or Fancy Bear. And this is an actor set that we believe originates in Russia. And Jeremy Dallman: This is someone that we've... an activity set that we've talked about fairly extensively over the years of public discussion around these actors. Whether targeting individuals or campaigns or entities involved with politics. So they're probably the more well known out of Russia. I'll just kind of hit a couple in each one of these here.Jeremy Dallman: Phosphorus, which is an actor set that we believe is originating from Iran, also known as APT 35 and Charming Kitten. They're well known for targeting government defense industrial, especially in the region, in the Middle Eastern region. Especially fond of targeting personal email accounts and going after personal email accounts as a way to gain access to systems that they're targeting or individuals and surveil individuals. A lot of activity there tied to sanctions and research around policy, that sort of thing.Jeremy Dallman: In China, we have actors that more broadly, I would say are more likely to use more sophisticated technical solutions. Trying to bypass or using more sophisticated malware, but technology, supply chain targeting, targeting education and medical research. Actors like Barium known as APT 41. Manganese, which will often target communication infrastructure. They'll even go after things like satellite or defense industry or GPS navigation.Jeremy Dallman: And then North Korea actors like Thallium and Zinc. We'll see them targeting human rights organizations and surveilling human rights organizations that might be involved in their region geographically. But we'll also see them often targeting think tanks and governments that are involved in sanctions or policy decision-making that might be tied to the Korean peninsula. Nic Fillingham: Why is Strontium a nation state actor and not simply just a sort of independent group of baddies?Jeremy Dallman: No, that's a great question. I think the simple definition of a nation state activity group is we defined it as cyber threat activity that originates in a particular country with an intent to further national interests. So because that activity fits that parameter, there's an assumption that it's more well-funded, potentially more sophisticated. And they'll more likely going to be using what we call advanced persistent threats which is an adversary that possesses a sophisticated level of expertise and significant resources that allow it to achieve its objectives using a lot of different attack vectors. It's a combination of expertise and significant resources, adequate funding to achieve specific objectives in a particular country with intent to further the national interests. Natalia Godyla: And what about attack techniques? So you hinted at that in your definition. So what are some commonalities or patterns that you can identify across nation state actors that differentiate them from other threat actors?Jeremy Dallman: So when you think about nation state actors, and I would say in most of our threats even outside of nation state actors, you're going to see most threats start with email. I think there was a blog post we put out not too long ago that said 95% of threats start with email. Start with an email lure. From a nation state actor perspective, that's largely a technique to achieve reconnaissance. To find out or identify who the people are that they need to target to achieve the objective that they're trying to achieve. So they will do things like password spray techniques to attempt to guess log in passwords for a number of accounts tied to a specific organization that they're trying to target. They will do brute force login attempts, trying to guess the passwords and try to brute force their way into an organization. That early reconnaissance technique allows them to establish an initial foothold into an organization and also then harvest credentials.Jeremy Dallman: So if they can start guessing passwords and they can understand what those passwords might be, they can harvest those credentials, store those credentials and then use those in future operations to come back into that network and execute whatever operation or mission they might be trying to achieve. Once they've actually established in there, and often as a way to get a foothold into a network, they'll use malware. Malware is a very common method by nation state actors. And I would say some actors on the nation state side, because of the excessive funding that they have at their disposal, they will go above and beyond in building up particularly sophisticated malware techniques to bypass common detections by security vendors and some networks. So that's constantly a game that we're playing to understand these malware techniques. We'll also see nation state actor using very sophisticated and personalized lures.Jeremy Dallman: They will spend a significant amount of time. And this is something we just blogged about a couple of weeks ago, an actor named Phosphorus, which originates in Iran. We're actually using building rapport and building relationships with individuals that are tied to international policy. And by building that rapport with those people, they were actually able to send them invitations masquerading as the Munich security conference, which is a prominent international policy conference. Masquerading as the conference and trying to lure that person to their fake invitation so that they could steal their credentials. A little bit of social engineering happening there. But a nation state actor is going to have the resources and funding at their disposal to be able to build out those more sophisticated techniques. And then finally, I would say there's a lot of nation state actors that spend a significant amount of time building out capabilities, relying on common weaknesses.Jeremy Dallman: So when a new patch goes out, patching a security flaw within a Microsoft product, for example. A lot of actors will reverse engineer that flaw. Better understand it then use it to weaponize a new exploit. Which is why it's exceptionally important for customers to patch as quickly as they can to avoid that weakness that Microsoft is attempting to patch. That weakness becoming an entry point for a malicious actor because nation state actors will move rapidly to take advantage of that and then attempt to exploit those weaknesses where they can. So that's a couple of techniques that I would say, like I said, we dive a little bit more into in the report. But there's more in there, especially things like web shell based attacks, which we see increasing, but I'll let you go read that into the report. Natalia Godyla: Yes. Nice teaser for our audience. One interesting point made in the nation state section of the MDDR was the downstream effect. So if I understand it correctly, the nation states will pursue these techniques and then eventually other actors will pick them up. So how does that happen if they are these sophisticated groups that are leveraging, like you said, more complicated malware? Is it that the other attackers use simplified versions of it, or as it's in the wild they get more exposure and are educated on that strain of malware and then are able to use it? So what does the process look like from nation state actor using these attack types to another attacker in the wild?Jeremy Dallman: Yeah, I think you nailed it there with the second example you gave. Because that's typically what happens is once this exploit gets out in the wild it's not just Microsoft watching for these more sophisticated threats. All of the other actors out there, whether they're criminal organizations or individual hackers, whoever it might be. There's a whole ecosystem of people out there that are watching for these threats to evolve and looking for new techniques. So when a nation state actor might have a particularly sophisticated attack that goes out, there's any number of people who will pick up and discover that through various security researchers in the ecosystem. And then they will immediately go do exactly what we do, which is reverse engineer that, understand how it works. And then you'll see variants come out. You look at things like the VPN exploits that came out in mid 2019.Jeremy Dallman: Those VPN exploits were picked up and used by an actor that we call Manganese to steal credentials and gain access to victim networks, using VPN infrastructure and holes in unpatched systems on VPN networks. So when you think about a world, the world we live in right now, where everybody's working remote. And global enterprise IT departments are relying on VPNs to improve connectivity and security for their systems. If that VPN infrastructure is not updated in its patching, actors like Manganese were taking advantage of that patch, reverse engineering it, and then going out to find VPN infrastructure that hadn't been patched and then exploiting it to gain access to those networks. Well, what we've seen subsequently is everybody else saw the technique and realized, hey, VPN, everybody's using those right now. And they started taking that and tweaking the same technique. And now those exploits have become, unfortunately become fairly commonplace. Nic Fillingham: Jeremy, you said that one of the characteristics of a nation state group is the sophistication in their techniques. And so I sort of have to ask, do we know if many of these groups, any of these groups are utilizing AI machine learning? If so, how?Jeremy Dallman: We don't have conclusive evidence I don't think. I mean, short of us walking into their infrastructure and taking pictures of systems, which isn't something we do. But I think there's enough- Nic Fillingham: Why not?Jeremy Dallman: ... indicators. Nic Fillingham: That sounds like a great idea. I'd make that a priority.Jeremy Dallman: That would definitely make our jobs a lot more interesting. I would say that we've seen indication of nation state actors starting to take advantage of whether it's machine learning or AI. It's unclear. They're starting to take advantage of more sophisticated techniques in those directions. When you think about a password spray campaign, where you are trying to attempt to guess the passwords for a number of different accounts across one organization, that takes a certain amount of compute, a certain amount of effort and a certain amount of automation that can be enabled. But if you take that and you expand it into something like we blogged about from Strontium in September, for example. We saw Strontium attempting to password spray a number of organizations, and they were spearfishing hundreds of organizations with thousands of password guesses in very short periods of time.Jeremy Dallman: And then on top of that they were using thousands of IP addresses and anonymization platforms to obfuscate their activity. So when you think about the complexity of that operation and the speed at which they were able to execute it, it would make sense that actors like that are starting to take advantage of machine learning or some automation capabilities on the backend to increase the speed, the effectiveness and the scope of their operations. Natalia Godyla: I think all of this is leading up to what is Microsoft doing? So how are we disrupting nation state threats today?Jeremy Dallman: So we do a number of different things. I would say probably the best and most effective way is using Microsoft's voice to raise awareness of these activities. And that comes in a number of different ways. We have the blog posts that we put out. The Microsoft On The Issues blog puts on a lot of interesting content that's derived from MSTIC research. And what that does is it kind of helps drive that broad discussion around what can be done to combat malicious nation state activity against governments, academia, social organizations, individuals. A lot of nation states like to target your personal email accounts, but we still defend those private email accounts because whether it's Outlook or a personal email account, that's something that we also have to protect our customers who might be getting attacked through that type of a vector. I would say probably one of the more interesting ways has been on the legal side.Jeremy Dallman: So one of our unique ways to target nation state actors has been partnering with our colleagues in the Digital Crimes Unit here at Microsoft. And the Digital Crimes Unit is responsible for pulling together a lot of the evidentiary information and understanding the threats for a legal perspective. And then they take that to courts and use litigation to seize domains and other assets that are being used by these nation state actors. And then actually through legal action shutting down those attack vectors. And then from time to time, we'll also, if we have sufficient information to warrant one time action to delete or shut down infrastructure or assets that are associated with the nation state actor. We'll also take those proactive measures against that infrastructure to basically eliminate visibility or capability on an actor and forcing them to go rebuild that infrastructure. They will typically rollover infrastructure and start rebuilding and come back later.Jeremy Dallman: So that's not necessarily a whack-a-mole game we want to get into in a lot of cases, but if it's for the protection of our customers, or if we feel it's particularly effective, that is something that we'll do as well. So that's a variety of a few ways. Obviously the one that I didn't touch on is probably the most obvious one, is leveraging our own technology and using all the knowledge that mystic collects about these threats, these actors, their tactics, their techniques and translating those into detections. Transforming and putting those into blocks and protections that show up in our security products and protect our customers in their environments. And the whole objective there has always been to make sure that we're implementing relevant, accurate and actionable threat intelligence for our customers. Nic Fillingham: Where can folks go apart from reading the MDDR? Where can they go for more information on how to protect themselves against a nation state attacks if they find themselves in one of these targeted industries?Jeremy Dallman: So we don't have a MSTIC page. I would say in the MDDR, Jeremy Dallman:We definitely have a section at the end of the Nation-States Reference called comprehensive protections required and it walks through to defensive positions that you can take, the strategies that you can enable there. And then at the end of the digital defense report, we have what are called actionable learnings. And I would recommend you go there and dive into that section as well. And every time [Mystic] puts out a blog post, we will always have something at the bottom that are generalized recommendations also. So if we put out a technical blog posts that walks through the techniques of gadolinium or strontium, we will always have at the bottom the specific techniques for that threat that would help you mitigate or protect yourself from that threat. So always watch for those blog posts and then probably for the digital defense report. Go out and look at the actionable learnings. That's probably the best place to start. Nic Fillingham: Hey, Jeremy. Thank you so much for your time. This has been a fascinating conversation. We've really only scratched the surface of that nation-state threat section of the MDDR report. So if you enjoyed this conversation, would like to learn more head to aka.ms/digitaldefense and download the report, and there's lots more detail and lots more articles linked too, that you can read to learn more about this space. Jeremy Dallman, thank you so much.Jeremy Dallman: This was fun. Thanks for having me guys. Natalia Godyla: And now let's meet an expert in the Microsoft security team to learn more about the diverse backgrounds and experiences of the humans creating AI and tech at Microsoft. Today we are joined by Randy Treit. Thank you, Randy, for being here.Randy Treit:I'm happy to be here. Thanks for having me. Natalia Godyla: Great. Well, let's kick things off by chatting a little bit about what you do. So what's your role at Microsoft? What does your day to day look like?Randy Treit: My title is principal security researcher. I'm on the Defender endpoint team. So focused mainly on detecting new threats that we haven't seen before. Protecting patient zero is a big focus of mine. Recently I've started looking into some new kinds of attacks using OAuth phishing. So that's sort of my current main focus area, but I've done a lot in the cloud protection. I've been on the team forever. So I've worn a lot of hats and done a lot of roles. Natalia Godyla: So what were some of the other roles that you've been at at Microsoft? What was the first one that brought you to Microsoft?Randy Treit: I've been at Microsoft 20 years. I started in the exchange team and worked on some mobility stuff. But pretty quickly... So I started in 2000. In 2003, I joined the antivirus team, which was brand new at Microsoft. Really Microsoft's first foray into trying to get serious about the antivirus space. And I joined as a program manager, actually. So security research is a fairly new role for me, but was basically worked on the backend infrastructure for the antivirus platform in the early days. And that was the days of worms running rampant everywhere you had SQL Slammer, MSBlast, Sasser worm, Code Red, Nimda. All the greatest hits of when security was a very dark, dark time at Microsoft. And that's when I started and then have done a ton of stuff since then. So I worked on the antivirus engine as a PM and from the engineering side. Eventually moved on to do a lot of work with our cloud protection system in the last period. And then, about two years ago, I guess I moved from engineering side into security research. Natalia Godyla: So were you sold on security after being part of the AV team? Was that what did it for you?Randy Treit: Our customers, Microsoft's reputation, friends and family, everybody was just getting hammered by security threats at the time. And I really wanted to do something about that. Working on exchange was fascinating from a technical perspective, but getting into the security space where there was a real opportunity to go to battle against the bad guys and try and really protect. I'm sure we all, back in those days, this is mid-2000s, early 2000s, had friends and family who got hit by a worm or a virus or a scam. And so it was very motivating for me to get into a place where I could do something about that. And that's sort of driven me ever since. And I've done a few other forays into some stuff, like I took a break from security for about two years. Around 2012, went and worked on Xbox for the Xbox One when that was getting released and learned a ton about services. And that was a good break, but I couldn't stay away from the security space. Nic Fillingham: Randy, I'd love to come back to that first gig of yours working in the anti-malware space. So for whatever reason, I actually went down a rabbit hole recently trying to better understand the history of Defender. It sounds like you were there at its sort of inception. My understanding is that the first anti-malware, antivirus client, first of all, it wasn't built into the OS. It was a download. And was it something that we built in-house or was it an acquisition? Was it a combination? Do you know the history? Were you there for that?Randy Treit: Yeah. So I was the third PM hired into the antivirus team and it was right after the decision to acquire [Rav] from a Romanian company called [GCAD]. And so I started on a Monday and on Wednesday all of the Romanian developers showed up, many who are still on the team today. Marty [Marinescu], who was the lead developer of the engine, is still the lead architect on the antivirus engine. And I remember, it was an interesting cultural experience, because they all came in and the custom in Romania was that you would, every morning, go to everybody's office and shake their hand and greet them in the morning. And so that was- Nic Fillingham: That's awesome!Randy Treit: Yeah, it was great. Unfortunately they, I think, became acclimatized to the not as polite American way of doing things. That sort of died out after a few weeks. But yeah, it was an acquisition and we didn't actually know what we were going to do with it at the time. So there was always a desire bring the protection capability into the operating system, but that's a big rock to lift and eventually we got there with Defender in the interim. It started out as, like you said, a download. So the initial... For years we've had the malicious software removal tool that comes out every patch Tuesday and runs on everybody's machine to clean up the ecosystem of malware.Randy Treit: But before that it was actually the very first release of the same engine that runs in Defender today, was something called Blast Clean. It was a Blaster removal tool to remove the Blaster worm. And we released that in late 2004. I have some stories about testing it out on my home machine and actually infecting it. And my kids not being able to play Magic School Bus the next day, and getting a call at the office. So those were fun times. Nic Fillingham: Can you elaborate on that? Is that the story? Is there more to it?Randy Treit: So what happened was the Blaster worm, there was a particular patch that if you weren't patched, it would infect your computer within a few seconds of being online. And so we had the early builds... This was December, heading into Christmas season in 2004. And I decided, well, I've got my computer at home. I'll just uninstall the patch and let it get infected. And then I will run our removal tool and make sure that it works. It was not the brightest thing to do. Don't do this at home kind of thing. I was younger and more eager to just do crazy stuff that I would probably be a little more careful these days, but I did it. I uninstalled the patch. The machine got infected. Rebooted, which was part of the infection.Randy Treit: And then it came up and I ran our removal tool and it worked great and then I decided to try it again. So for those who may remember the Blaster worm, there was another worm called Nachi that somebody else had written and released, exploiting the same vulnerability. And Nachi tried to remove Blaster and then patch your computer. And so our tool removed both of those. And what happened, in my case, was the machine got infected with Nachi, but it was a copy of the Nachi worm, that had itself been infected with a file infecting virus, which infected all the executables is on my machine and then basically bricked it and made it so it wouldn't boot. Nic Fillingham: I know that I got infected with Blaster worm. I couldn't remember that because I got in big trouble from my dad.Randy Treit: Oh, yeah. Nic Fillingham:But I sort of can't remember what it did. I know that it stopped... No one could use the computer. It just completely... The computer was unusable, but can you just kind of bring us down memory lane? If you were infected by Blaster worm, what actually happened?Randy Treit: It was not a worm that was exfiltrating data off your machine. Now it's all about money and these crime groups trying to exploit the ecosystem with Ransomware and that kind of thing. It was really just designed to spread. So it was purely, as I recall and if I'm remembering correctly, but it would just try and infect... It would infect your machine and your machine should actually be able to run with the infection. Although like in my case, and maybe in yours, if it got infected with a version that was itself infected with something else, it would just brick the machine. Like if there was a file infector, which is what I experienced with the Nachi worm. But essentially it would just try and spread to other machines that were unpatched, randomly spraying IP addresses trying to find another machine that had the vulnerability. Natalia Godyla: So you mentioned that, right now, part of your role is to focus on protecting patient zero. So how is that different than some of the work you've done in the past? And what's different about focusing on patient zero in specific?Randy Treit: The attackers could guarantee that they could release something into the wild that wasn't detected because it wasn't detected by current signatures. So before we had cloud protection, you just had the heuristics and signatures that were on disc in these virus definition updates that computers would download periodically. Typically, a few times a day. So you couldn't really protect patient zero because the attackers would always be able to tweak their malware until they saw from scanning with, say, the virus signatures that you weren't going to be able to detect it. And then they would release it. And then the clock starts ticking at that point. And you have a certain amount of time before, say, a customer reports that to Microsoft, or we discover that a sample from some sort of honeypot, or whatever.Randy Treit: And then now you have, okay, we need to quickly add a signature and ship that out to the customers. So the cloud has been a real game changer because it gives us an opportunity to run all these machine learning models in real time, in milliseconds to make an evaluation of a file that we've never seen before and decide that it's malicious and then block it. That has been a huge game changer in terms of protection capability and really shrinking that time to protection to milliseconds from where it used to take days and hours to get a signature out. Nic Fillingham:And how do you measure the false positive rate? If there is one, in that sort of protecting patient zero. How do you measure and then how do you find that balance between a couple of false positives, which would be, probably, annoying. But do you allow yourself a few of those to slip through in order to genuinely protect patient zero? Or are the models so good now that the false positives are extremely rare?Randy Treit: Oh, well, we're always going to have some false positives. ML is not perfect and human expert rules and human logic is not perfect. So there always will be false positives. We have certain thresholds that we try and keep our rules under, or that are basically lines in the sand that, hey, in order to release a new, say, detection rule in our cloud protection infrastructure, it has to run in an experimental period for a certain amount of time. Typically, even a few weeks while we gather all the data on what it would have blocked on, and then we can evaluate, is it having a nice, low, false, positive rate? So there are certain thresholds that we need to make sure all those rules are running under.Randy Treit: And then we have guard rails to make sure that if all of a sudden a rule or an ML model starts... Something changes under the hood and it starts having too high of a false positive rate, then we have systems to alert and automatically disable things until somebody goes and investigates and that kind of thing. So we're definitely very cognizant of trying to find that balance between blocking the bad stuff, but not causing too many false positives and causing pain and headache for our customers. Nic Fillingham: And does your team monitor those metrics? Is that what your team, as part of looking after patient zero, is that one of the things that you track day to day, or is that another part of the org?Randy Treit: Yeah, it's definitely our team. There are other kind of data science focused people who will do a lot of the infrastructure work to support running those metrics. But our team looks... That's creating the cloud rules and some of that capability. We'll work on writing watchdogs and guardrails and alerts and things like that. Just as part of the end to end pipeline of creating that protection. Nic Fillingham:What are some of those tools that you use day in, day out, Randy? When you start your day, where are you going to? Do have some sort of team dashboard, or are you going into some kind of Azure ML service? Yeah, what's in your toolbox?Randy Treit: So we definitely have our dashboards and tools that are the sort of go-to place for, oh, you want to see the trend of detections over time, and these kinds of things and monitor your rules and whatnot. I tend to go a lot deeper into the actual data. So I'm a big fan of Jupyter notebooks and pandas on Python. I've done a bunch of stuff in [AR], in the last couple of years. Lately I've been using Databricks notebooks, which are fantastic because it basically lets you do big data. Sorry. I don't know if you're familiar with the notebook type environment, but it's essentially a combination of marked down notes and graphs and visualizations. Nick, I know you've seen some of my heat maps that I like to generate, showing where we're seeing particular attacks happening globally.Randy Treit: That's all done in this notebook environment where you have this data under the hood. You can write Python code or AR or Scala, and then, to process the data, and then not the other, it'll spit out a beautiful global heat map or graphs or data. And you can just sort of have instant [inaudible] at your fingertips. So typically, my day starts with usually firing up some kind of a notebook, pulling in some data. I'm often looking for gaps, so where are we not doing Randy Treit: Well, so what did we see over the last... Let me find files that we're now blocking in the cloud, because our cloud learned that these are malicious, but maybe we miss patient zero and maybe we missed the first 25 encounters. Now, then we started blocking. Oh, let me figure out what happened there. Why didn't we block? How do we close that gap? Randy Treit: My day job, I would say, is really trying to find protection gaps where we're not doing a good job and figure out how we close them. They go actually implement something to close those gaps. I tend to work with Python mostly day-to-day in a Jupyter Notebook or more recently, these Databricks Notebook type environment. I love it. Compared to the old days of you're running just SQL queries against a small set of SQL data, the things you can do with these, I would say, data scientist type tools like Jupyter Notebooks is very freeing. I guess that's how I would put it. Nic Fillingham: Randy, what's flagging those gaps? So, you said you look for gaps. Is that just your experience, your expertise, you know what you're looking at when you see data, when you see dashboards, when you see reports; or are there a combination of processes that are specifically looking for a detection that picks something up and then went backwards in time and realized that "Oh, here are some historical detections that we actually miss"? How do you find gaps? I think that's the question. Randy Treit: It's a combination of manual speed locking on into the data and going off intuition or things I've done before, but we do have automation that will flag certain events. We have watchdogs and other rules that researchers write. In my mail inbox in the morning, often, I will have a list of these potential misses where maybe we missed detection on the first patient 0 through 10, and then we started blocking. So, I might go and look at, "Oh, let me dig into that a little bit and find out what happened there." So, in some cases, it might be that we have a malware probability threshold that we were looking forward to say from an ML model that says, "Oh, block if the probability is 0.95. So, 95% probability that this file is malware." Randy Treit: Going into the data in telemetry, I might see that we didn't block because the probability was 0.93. So, one of the things I would look into then, oh, can we reduce that probability that we're looking forward to block from that 0.95 threshold to 0.93? Maybe code up something to model that or to run for a few days in experimental audit mode and see, "Does that lower threshold still meet our false positive targets?" If that's looking good, we can turn that on live, something like that. Natalia Godyla: This is a bit of a deviation, but it would be great to understand, "What kind of context do you bring to this role from previous jobs? What were you studying in school? What did you intend to do? What were your jobs prior to Microsoft, and how do you use them in your day-to-day?" Randy Treit: Yeah, that's a great question. So, I was actually studying Philosophy in Pacific Lutheran University down in Tacoma. I'm a native Washingtonian. So, Microsoft was right in my backyard. It was basically the height of the dotcom boom and the end of the '90s. I had finished up the Philosophy Program at PLU and was planning to become a philosophy professor but needed to get a job. In the interim, I was married. We had a young child, another one on the way. So, I decided to take a break from school, get a job. I started as a technical writer actually at Microsoft on the Exchange Team. I think you talked to Emily Hacker. I listened to the interview and learned that she was also started as a technical writer. So, that was pretty cool. Randy Treit: And then worked in exchange for a few years before I got asked about joining this newly formed antivirus team. I made the jump there. I actually never finished my four-year degree. So, I made a plan with my advisor. I finished the philosophy program, but still had some general university stuff to finish up. But once I started at Microsoft, I was just off and running and never looked back. So, it's been an interesting journey. Sometimes I definitely suffer from, I would say, imposter syndrome here and there, where I spent a lot of time writing code day-to-day, but I've never been formally trained in computer science. It's all been self-taught or picked up on the job thing. Randy Treit: When I moved from a program management and the engineering side into research, I came without the deep reverse engineering background that a lot of my colleagues had. So, that was something that I felt like, "Oh, this is going to be hard for me to pick up." Sometimes that lack of a formal academic background, I feel like it was a bit of a chip on my shoulder, but I just try and do the best I can and go from there. Nic Fillingham: Talk a bit about philosophy, and then I'd love for you to talk about how and if you use it in your job today. Randy Treit:Yeah. So, I was not a good student in high school. So, I barely graduated high school with a very low GPA. So, when I decided to finally get my act together and go back to school, I started at a community college. I needed to take English 101 just as part of every college requirement. So, the English 101 class I took was a combined English 101 and Philosophy taught by two professors who were husband and wife. Debbie Kuder, the wife taught the English portion, and then her husband, John taught the Philosophy portion. It was basically an amazing class. My identical twin brother, who also works at Microsoft by the way, was in the same class with me. We both just fell in love with philosophy. Randy Treit: I think, I just love the idea of open-ended questions that had no answers. So, philosophy, I think differentiated from the sciences, it's dealing with questions that will never actually be answered, like what is beauty and what is a good argument? There's always going to be different opinions. Just the idea of these big open-ended unsolvable questions, but the people will keep getting closer and closer to the truth hopefully over time, I just fell in love with that. In terms of applying philosophy at work, I think the biggest thing that I got out of studying philosophy in undergraduate school at PLU was the rigorous approach to problem solving. So, even though you have these big open-ended problems, like I said, there probably are never going to get answered. Randy Treit: The approach of philosophical approach is very rigorous and requires incredibly good communication skills to be able to communicate your ideas effectively and, in your arguments, cogently. That, I think, has stood me in extremely good stead in my career. I think that's one of the things that I bring to the table. I think someone like Emily, you mentioned with the journalism background, it's just that ability to communicate. There's so many brilliant people who work in the technical field, but who are unfortunately not great communicators. Often, they need someone to help translate what their brilliant ideas into something that other people can actually understand what they're aiming at. Randy Treit: That's something that I think I've been able to do fairly successfully. Just that ability to really rigorously attack a problem and break it down into small components, which I think comes from some of that training I think has also done a great job or has stood being a good stead with malware analysis and threat analysis and that kind of thing. Natalia Godyla: So, I know Nic is dying for me to ask this, but you said you had an identical twin, you just dropped it in there casually that works at Microsoft. Do you guys pull pranks together. Have you done it as kids? Do you do it at Microsoft? Randy Treit: You have no idea. So, Mike actually worked on the antivirus team at the same time as I did. So, he joined Microsoft before me and has worked on NT 5, which became Windows 2000 and is a brilliant dev, but he was actually one of my devs and I was his PM working on the antivirus. This is probably mid-2000s. For a number of years, we were on the same team. And then he went off to Intune. But I mean, the amount of confusion we caused when people would walk into meetings or even just down the hall, it was quite fun. I'm sure we played some pranks. It's been great. Randy Treit: There was one time very early on, we weren't on the same team at that point, where he was in my office over an exchange. He had come over to grab a coffee. He was across the street. I had gone down to get a refill or use the restroom or something. This guy, David came in and started talking to Mike, like he was me, "Hey, Randy, I've got some questions about this thing." Mike was like, "Oh, I'm not Randy." David looked at him and just shook his head and said, "So, anyway, I've got questions. Do you know about this?" Mike's like, "No, I'm not Randy." He looked at him and he said, "wait, are you serious?" So, we've had those kinds of incidents. Randy Treit: Mike is my go-to person whenever I get stuck on a programming problem, because he's a brilliant programmer. So, I'm constantly sending him my code and saying, "Hey, I'm struggling with this." He usually responds with something like, "What is this monstrosity?", and things like that since I'm not nearly the coder that he is. Natalia Godyla: Subtle. Nic Fillingham: Who's the older twin by a fraction of a second or a minute?Randy Treit: Mike's four minutes older than I am. Nic Fillingham: I love it that your prank was actually a wholesome misunderstanding, an unintentional wholesome misunderstanding. I was typing frantically with Natalia, trying to see if there was some example, where you each went to the other's annual review and just tried to just say ludicrous things to the manager to see when they caught on, but no.Randy Treit: No, I haven't done too much of that at work. Although, I mean, in high school, he would skip class and I would go to his art class, because I had a girlfriend who was in the same class. One day, I got called up to make a presentation, the person they thought I was Mike. I was completely unprepared and I just fumbled my way through it. I learned that, "Oh, that didn't work out the way I was hoping it would." I'll throw this out there. My younger brother also works at Microsoft. He is a producer on Xbox video stuff. So, there's a bunch of us running around. Nic Fillingham: How many other Treits are there?Randy Treit: My sister, Tammy worked on Exchange at the same time I did back in the day. There are six of us Treit siblings. I guess four of us have worked at Microsoft. My younger sister is a doctor in Seattle, and my older sister is a teacher in Germany. Natalia Godyla: Thank you, Randy. We're happy to have you at Microsoft. Happy to have two-thirds of your family at Microsoft here, and we'll definitely love to have you back. Randy Treit: That was a lot of fun. I really enjoyed the conversation. Natalia Godyla:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode. Nic Fillingham: Don't forget to tweet us, @msftsecurity, or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe. Natalia Godyla: Stay secure.
1/6/2021

Unpacking the New ML Threat Matrix

Ep. 9
Yeehaw! “Data Cowboy” is in the building. Join us as Nic Fillingham and Natalia Godyla sit down with Ram Shankar Siva Kumar, aka “Data Cowboy” at Microsoft, for an exciting conversation about the release of a new adversarial ML threat matrix created for security analysts. Have no fear, we made sure to find out how Ram acquired the name, “Data Cowboy”, so saddle up and get ready for the ride!Stick around to hear Nic and Natalia explore the urgency of surfacing threats at a faster rate with Justin Carroll, a Threat Analyst at Microsoft, and why it is more important now than ever before.In This Episode, You Will Learn:  How Microsoft is using the new ML threat matrix against cyber attacksThe approach and philosophy for putting the threat matrix on GitHubML applications in regard to healthcareand why it is worrisomeWhat needs to happen in order to be successful in combatingcertainthreats  Some Questions We Ask: What is an adversarial ML threat matrix?How will the community on GitHub contribute to the evolution of the ML threat matrix?What resources are available to learn about all things VM?What techniques are being used to find threats at a faster speed?How do AI and ML factorintothe role of managing data and collaborating with other teams?ResourcesRam’s Blog:https://www.microsoft.com/security/blog/2020/10/22/cyberattacks-against-machine-learning-systems-are-more-common-than-you-think/Microsoft Security Blog:https://www.microsoft.com/security/blog/Nic’s LinkedInhttps://www.linkedin.com/in/nicfill/Natalia’s LinkedInhttps://www.linkedin.com/in/nataliagodyla/Ram’s LinkedInhttps://www.linkedin.com/in/ram-shankar-siva-kumar-7b04a73a/Justin’s LinkedInhttps://www.linkedin.com/in/justin-carroll-20616574/Transcript(Full transcript can be found athttp://aka.ms/SecurityUnlockedEp09)Nic Fillingham:Hello, and welcome to Security Unlocked. A new podcast from Microsoft, where we unlock insights from the latest in news and research from across Microsoft security engineering and operations teams. I'm Nic Fillingham.Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft security, deep dive into the newest threat intel, research and data science.Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft security. If you enjoy the podcast, have a request for a topic you'd like covered, or have some feedback on how we can make the podcast better.Natalia Godyla:Please contact us at securityunlocked@microsoft.com or via Microsoft security on Twitter. We'd love to hear from you. Hi Nic. Welcome back. How were your holidays?Nic Fillingham:Yes. Thank you, Natalia. Welcome back to you as well. Mine were great. You know, normally you drive somewhere or you fly somewhere, you go visit people, but this was all the FaceTimes and the Zooms and the Skypes, staycation, but it was still nice to eat too much and drink too much over the holiday period. How about you?Natalia Godyla:Yes, it was... to quote my boss. "It was vegetative." It was definitely just... well actually you know what? I did have a big moment over the holidays. I got engaged. Nic Fillingham:Oh, what! Natalia Godyla:I know.Nic Fillingham:Congratulations. Natalia Godyla:Thanks.Nic Fillingham:That's amazing. Natalia Godyla:I feel like it was absolute relaxation, really high point during the five minute proposal. And then we went back to our natural state and just absolute relaxation, lots of video games.Nic Fillingham:Hang on. So were you both sitting on the couch, playing some switch, eating your 95th packet of Doritos, and then all of a sudden your partner pauses and says, "You want to get hitched?"Natalia Godyla:There was a little bit more pomp and circumstance to it. Though I think that would have been very fitting for us. Nic Fillingham:Wow! Good on you guys. That's awesome. Natalia Godyla:I'm sure that like us, everyone has forgotten what they were doing at work, and I'm sure also what this podcast is doing. So why don't we give everyone a after the holiday refresher?Nic Fillingham:So just before the holidays, we partnered with Petrie who run the patrie.com site throughout.com. First Ring Daily, a bunch of other great blogs, podcasts, email newsletters, and so welcome to all our new listeners who've come to us from Patrie, from Throughout from First Ring Daily. Yeah. So what is security unlocked? Well, first and foremost, Natalia, and all your co-hosts, we are Microsoft employees and we will be interviewing, and we do interview on this podcast, other Microsoft employees, but we talk about security topics that hopefully are relevant to all security professionals and those who are interested in the state of cybersecurity. Nic Fillingham:And what we'll do in each episode is the first half is we'll pick a sort of a recent ish topic and we'll speak to a subject matter expert or an author of a recent blog post and ask them about the thing that they're working on, or that they've announced in the AI and ML space, hopefully try and demystify some new terms or concepts that may be either nascent or sort of difficult to wrap one's head around. And then in the second half...Natalia Godyla:We talk to again, another Microsoft security expert, this time more focused on the individual and their path to cybersecurity. So we'll ask them about what interested them about cyber security, what compelled them to join the industry, what jobs they've had, how they've come to Microsoft or their current role. In addition, we also have a new announcement about the podcast, which is we'll be switching to a weekly cadence. So prior to this, we were bi-weekly, now more goodness coming your way.Nic Fillingham:More pod in your pod app. What is the collective receptacle for pod? What is it? More pods in your cost, more cost in your pod?Natalia Godyla:More beans in your pod.Nic Fillingham:I like that. More beans in your pod. And I think the other thing that's worth reiterating Natalia is if you have a cyber-security topic you would love to learn more about, or a perspective you'd like to hear from, please let us know, we'll go after it for you and try and bring that to a future episode.Natalia Godyla:Yes, absolutely. We're really thankful to everyone who has reached out thus far and just keep it coming.Nic Fillingham:On today's episode in the first segment, which we call our deep dive, we speak with Ram Shankar Siva Kumar, whose title I will not give away in the intro because we talk about it in the conversation. And it's an awesome one. Ram works in the Azure Trustworthy ML team. And he's here to talk to us about a blog post that Ram co-authored with Ann Johnson that announces a new adversarial ML threat matrix that has been built and published up on GitHub as a collaboration between Microsoft, MITRE, IBM, Nvidia, Bosch, a bunch of other organizations as a sort of open source approach to this upcoming sort of nascent threat category in adversarial machine learning. And it was a great conversation. And then after that, we speak with...Natalia Godyla:Justin Carroll of the Microsoft Threat Intelligence Global Engagement and Response team. He started in networking very on the ground and only got his education in cybersecurity later in his career, which I think to anybody out there, who's looking to transition to security, who has a different background in security and is wondering whether they can make it, you can. He also chats a little bit about what inspired him to join cybersecurity. Some of it came from video games, which is a theme we're seeing again and again.Natalia Godyla:So he had a unique spin on vigilantism within video games and ensuring that those who had an unfair advantage by using mods were checked and tried to level the playing field for all the rest of the players of that game. And of course we touch on Ninja Turtles, which is really the highlight of the episode. I think, with that on with the pod.Nic Fillingham:Ram Shankar Siva Kumar, thank you for joining us on Security Unlocked.Ram Shankar Siva Kumar:Hey, thanks for having me, Nick and Natalia. Really appreciate it.Nic Fillingham:So we're going to talk about a blog post that you co-authored with the wonderful Ann Johnson. The title is, it's a great title. I'll get straight to the point. Cyber attacks against machine learning systems are more common than you think. Before we get into that, though, I just have to ask, you list your title as data cowboy, which is fantastic. I would love data cowboy, anything cowboy. I would love that for my title. Could you explain to people, what does a data cowboy do and what is the Azure Trustworthy ML group?Ram Shankar Siva Kumar:Oh, totally. First of all, this is like every kid's dream is to be Woody from Toy Story. It's just like, I realize it in my own way. So when I joined Microsoft in 2013, there really wasn't an ML engineer position. So my boss was like, "You can be whatever you want. You can pick your own title." I was like, "Yes, Toy Story comes to life." So it was like, this is a brown version of this Woody that you kind of get. So basically what the Trustworthy Machine Learning group does is our promise to Microsoft is to essentially ensure we can enable engineers and customers to develop and deploy ML systems securely. So it's kind of a broad promise that we make to Microsoft and our customers.Nic Fillingham:Got it. I would love to come back to just the data cowboy one more time. Tell me what you do. I mean, I have visions of you riding around the office on a hobby horse. La suing errands databases. Tell us about your day to day. What does it look like?Ram Shankar Siva Kumar:Yeah. So what really happens is that, like I said, I really wish I can write it on my office, kind of like my home and my 500 skirt apartment definitely not recommended, but most of the time we end up doing is this wonderful Hiram Anderson who's part of our team. He's militarily looking at how we can detect attacks on machine learning systems. So really working with him and the rest of the Microsoft community to kind of keep our eyes and ears on the ground, see like what sort of attacks on machine learning systems we are seeing, our various different channels and trying to see how we can detect and respond and remediate those sort of attacks. So that's the first one big one. The second thing is like I get to work with a wonderful Will Pears. So I get to work with him to think about actively attacking red teaming Microsoft's machine learning system. So even before our attackers can look at, exploit the vulnerabilities Will and Hiram go and actively attack Microsoft ML systems.Natalia Godyla:So how does the work you do connect to the different product groups. So as you're identifying these cyber attacks, are you then partnering with our products to build those into the detections?Ram Shankar Siva Kumar:Yeah, that's a great question. So one of the things I really like about Microsoft is that super low slake to meet with somebody from another product team. So the amazing Mira Lane who heads the Azure Cognitive Services, really worked very closely with her. And I believe you ever had a Holly Stewart in your podcast as well, so worked very closely with her team. So it's really a big partnership with working with leaders from across Microsoft and kind of shopping around what we're doing and seeing how we can kind of help them and also learn from them because they also have sensors that necessarily might not have.Nic Fillingham:Let's talk about this blog post. So you and Ann both announced this really interesting sort of consortium of 11 organizations, and you're releasing an adversarial ML threat matrix. It's open source, it's on GitHub. Very exciting. Tell us about it.Ram Shankar Siva Kumar:So the goal of the adversarial ML threat matrix is essentially to empower the security analyst community so that they can start thinking about building detections and updating their response playbooks in the context of protecting ML systems. And one of the things that's kind of like we want to be mindfully different is the attacks that we see to this framework with, all these techniques, we kind of only put the ones that Microsoft and MITRE jointly vetted that were effective to be against production machine learning systems. Ram Shankar Siva Kumar:So first of all, the whole area of attacking machine learning systems goes all the way back to 2004. In fact, you can find Daniel Loud, whose Twitter handle is Dloud on Twitter today. He continues to work on this super cool fields and there's a wonderful timeline by this other researcher called [Battista Bisho 00:11:05] that he also linked to the blog, but he can basically see that this work has gotten immense academic interests for the last 16 years. And especially in the last four years after a very Seminole paper was released in 2014.Ram Shankar Siva Kumar:So when a lot of people think about spiel, they think of as, oh, this is something that is really theoretical. This is something that... Oh, Greg, you're working in academic setting, but no, that's not true. There are marquee companies, who've all had their ML systems subverted for fun and profit. So the whole point of this blog post with MITRE and this whole corpus of industry organizations was, this is real. Attacks on machine learning systems is real, you need to start thinking about this.Ram Shankar Siva Kumar:Gartner released a report on 2019 saying, 30% of all cyber attacks in 2022 is going to involve a tax on machine learning systems. So this is not a pie in the sky. Oh, I'll get to it when I get to it. 2022 was a year and a half, it's a year away from now. So we got together in this blog post to really empower our security analysts community and help them orient for this new threats.Natalia Godyla:Can you talk a little bit more about what exactly is the adversarial ML threat matrix and how you envision security analysts using this tool?Ram Shankar Siva Kumar:Yeah, totally. So one of the things that before we even put this matrix together, we kind of conducted a survey of 28 organizations. We spoke to everybody from SMBs to governments to large organizations and we spoke to the security analyst Persona, as well as the MLG person. I asked them, "Hey, how do you think about securing ML systems? This is a big deal. What are you doing about it?" And they were like, "Well, we don't have the tools and processes in place to actually go and fix these problems." So the first thing we realized is that we wanted the security analysts community to be introduced to adversarial ML as a field, try to condense the work that's happening in a framework that they already know. Because the last thing we want to do is to put another framework another toolkit on their head.Ram Shankar Siva Kumar:And they're just going to be like, "Nope, this is not going to work out. This is one more thing for them to learn." So we took the MITRE's attack framework. So this is something that was again, bread and butter for any security analyst today. So we took the attack framework and we kind of said, "Hey, we've been really cool." If you took all the ML attacks and put it in this framework, and that's exactly what we did. So if you look at our track matrix, it's modeled after the MITRE attack framework. Ram Shankar Siva Kumar:So the wonderful folks from MITRE's ML research team and us, we got together and we basically aligned the attacks on machine learning systems, along reconnaissance persistence, model evasion, Exactly. filtration. So if you look at the top of our matrix, the column headers are essentially tactics and the individual ones are techniques.Ram Shankar Siva Kumar:So let's say that an attacker wants to gain initial access to a machine learning subsystem, let's say that's her goal. So she has a couple of options to kind of execute her goal. She has a couple of techniques in her kitty. The first thing is that she can just send a phishing email to an ML engineer. That's very valid. Phishing is not going to go away. The second thing that she can do is she can take a pre-trained ML model available that people generally download and she can backdoor it. So the whole point of this attack matrix is to A, build a common purpose of attack techniques and attack tactics in a framework that a security analyst already has knowledge off.Natalia Godyla:Are you seeing any trends? What's most common to combine.Ram Shankar Siva Kumar:Oh, that's a great question. So before I just step into this, I first want to tell you about this attack called model replication. So the easy way to think about this and Natalia, I will get to this, I promise. Natalia Godyla:I love the excitement. I'm so ready for it.Ram Shankar Siva Kumar:We're going to take a little detour like Virgil and Homer. So essentially the best way to think about model replication is that open AI is a very famous and I'll start up. And they last year released a model called GPT-2, and they said, "Hey, you know what? We're not going to release the entire model immediately. We're going to release it in a stage process." We're going to just... because we want to do our own verification and before they could release the entire model, these spunky researchers, so I love that. They're still cool. Vania Cohen. And I know his like little other persons [inaudible 00:16:01] work name is Skylion with a O, they replicated GPT-2 it was like 1.5 billion parameter model, and they've leased it on the internet on Twitter. And they call it open GPT-2. And I love their tagline, which is GPT-2 of equal or lower value.Ram Shankar Siva Kumar:So [inaudible 00:16:22] even before the company could release, they replicated the ML model based on the data sets that were available based on the architecture. And they basically at the end of the day, and we also references our case study is that they basically tweaked an existing model to match GPT-2 and they publish that for everybody to use. No, it does not have the same accuracy or the same metrics as the original GPT-2 model. But the fact that an attacker can even replicate a ML model using publicly available data sets and having some insights about the architecture is something for people to think about.Ram Shankar Siva Kumar:So now to come back to your excellent question. So what exactly is a common pattern? So what essentially we see attackers doing is that they go interact with the machine learning system, attackers might send some data. They might get some responses back and they keep doing that enough amount of time. And they now have sufficient data to replicate the ML model. So the first step is that they go and replicate the ML model and from the ML model that they have replicated, they go do an offline attack. Because now they their own ML model, they try to evade this ML model and then they find a way to evade the ML model. And they take the examples of the test points that evade the ML model and now evade the online, the real ML that's out there taking that and then boom, fooling the real online ML model. So that's a common data point, but three case studies in our adversarial ML GitHub page that actually kind of shows this.Nic Fillingham:So the sort of takeaway from that. If your data set is public, don't make your ML architecture public and or vice versa.Ram Shankar Siva Kumar:That's a great question. And I've been thinking about this a lot, first of all, we definitely want to be transparent about the baby builder ML models, right? [Marcus Sanovich 00:18:25]. Oh gosh, he's such an amazing guy. But for the last so many years in RSA has been like medicinally, been talking about how we build our ML models for security purposes, because we want to give insights into our customers about how we actually built ML models. And the data sets are machine learning as a field, it has as norms of opening up our data sets. In fact, one can attribute the entire deep learning revolution to [Dr. Fayfee Lee's 00:18:55] image in a dataset which really sparked this whole revolution. So, I really don't want anybody to think that being open with our data sets or being open with our ML platforms is a good idea.Ram Shankar Siva Kumar:Because even if you think of traditional cyber security, right? Security by obscurity is never a good strategy. So the way we want to push people to think about is how are you thinking about detection? How are you thinking about response? How are we thinking about remediation? So really trying to take the assumed breach mindset and feeding it into your ML systems is how we want to push the field towards. So if you take away anything from this is continue to be opening your systems for scrutiny, because that's the right thing to do, that's the norms, don't be upset. And that's important to advance research in this field and think about detection strategies and think about, and assume breach strategies for building ML systems. Ram Shankar Siva Kumar:We wanted to distinguish between traditional attacks and attacks on ML systems. So the one thing that I want to think about is the threat matrix contains both traditional attacks and attacks on ML systems. Whereas the taxonomy only contains attacks on ML systems. The second difference is that, like I said, the matrix is meant for security analysts. This one is meant for policymakers and engineers. The third best that's more important difference is that in the context of the threat matrix, essentially we are only putting a tax that we have validated against commercial ML systems. It's not a [inaudible 00:20:34] list of attacks. We're not trying to taxonomize. Nic Fillingham:I wonder if you could talk about the approach and the philosophy here for putting this on GitHub and making it open to the community. How do you hope folks will contribute? How would you like them to contribute? Ram Shankar Siva Kumar:Yeah, absolutely. So Miguel Rodriguez, who runs the MITRE, who we collaborated with, wonderful team over there before putting this out on GitHub, there was a little [inaudible 00:20:57], right? Because this is not fully baked product. This is something that 13 organizations found useful, but doesn't mean everybody in the community might find useful. And I think he said something to the effect of-Nic Fillingham:It's almost as if you're a cowboy.Ram Shankar Siva Kumar:Yeah. Then you go hurting people. It was like, we're putting this out, acknowledging this is a first cut attempt. This is a living document. This is something that we have found useful as 13 organizations, but we really are hoping to get feedback from the community. So if you're listening to this podcast and you're excited about this, please come and contribute to this matrix. If you think there are attacks that are missing, if you would like to spotlight a case study on a commercial ML system, we are super looking to get feedback on this. Ram Shankar Siva Kumar:And we also kind of realized that we wanted a safe space almost to talk about attacks on ML systems. So we were like, you know what? We're just going to have a little Google groups. And the membership of the Google groups is extremely diverse. You've got philosophers that are interested in adversarial machine learning. We've got people who are looking from various perspectives, joining our Google groups and kind of like giving us feedback and how we can make it better.Natalia Godyla:Yeah. As you mentioned, there are tons of different perspectives coming into play here. So how do you envision the different roles within the community interacting? What do you think needs to happen for us to be successful in combating these threats?Ram Shankar Siva Kumar:Yeah. This is a great question. The one thing that I've learned is that this topic is immensely complex. It's mind boggling to wrap the different personas here. So I'll just give you a rundown, right? So, so far we know that policymakers are interested in securing ML systems because every national AI strategy out there is like, securing ML systems is top priority for them. ML engineers are thinking about this, academic researchers. There were like 2000 papers published in the last, I want to say five or six years on this topic. So they are like a hotbed of research we want to rope into. We've got security analysts from these companies that we're talking to are interested. Csos are also thinking about this because this is a new threat for them. So as a business decision maker, how should they think about this?Ram Shankar Siva Kumar:One thing that I got an opportunity with Frank Nagle, who's a professor at HBS. We wrote up piece at Harvard Business Review talking about, is it time to insure ML systems. ML systems are failing so if you're ML powered like vacuum cleaner burns a home down, what do you do about it? We try and rope in the insurers to come in participate in this. So, Natalia this is such a green field and the only way we're going to like get ahead to really get people excited and try for clarity together as a community.Nic Fillingham:How would an ML powered vacuum cleaner work?Natalia Godyla:I was going to say that sounds like a 2020 headline, ML powered vacuum cleaner burns down house [inaudible 00:23:58] threat.Ram Shankar Siva Kumar:Oh my gosh. So, okay-Nic Fillingham:Man bites dog. Ram Shankar Siva Kumar:There you go. It's funny because this was not an example that I made up. I wish I did. I know. Yes, Nic. I see [crosstalk 00:24:10].Nic Fillingham:What's my joy [crosstalk 00:24:12]. Ram Shankar Siva Kumar:Yes. Nic Fillingham:All right.Ram Shankar Siva Kumar:This is a well-documented paper called a concrete problems in AI safety. And they talked to the most it's like Final Fantasy. Everything that needs to go wrong is going wrong. So, they're like robots that are burning down homes, breaking things that they can clean up. So if your machine learning system is not trustworthy, there are going to be problems. And you really need to think about that. Nic Fillingham:I can't even get my kettle to boil.Ram Shankar Siva Kumar:But the thing that really worries me is ML applications used in health care. You keep seeing headlines like machine learning systems being used by radiologists, amidst radiologists when it comes to identifying Mulligan tumors and things like that. There's a fantastic work by [Samuel Firelson 00:25:03] from Harvard. He show that if you take an x-ray image, just take it and slightly rotate it and you give it to the ML system. It goes from very confidently thinking that it's malignant to very confidently judging it's benign. And that is really scary.Ram Shankar Siva Kumar:In the beginning of the podcast, we spoke a lot about how an adversary can subvert machine learning systems for fun and profit. Oh boy, there is an entirely separate world of how machine learning systems can fail by themselves. What we call unintentional failure modes. And trust me, you will want to go live in the middle of the North cascades in a cabin after you read that work. It'd be like, I am not getting anything ML powered until they figure this out. But the good news is there're extremely smart people, including Hiram and Will from my team who are looking into this problem. So you can feel a little bit like a shore that they're the true Avengers out there.Natalia Godyla:And with all the head nodes [inaudible 00:26:11]. I feel like it underscores the fact that we only know a percentage of the knowledge on ML. So we just need a community behind this. No one company person can know all of it. Ram Shankar Siva Kumar:Absolutely. Oh my gosh. Yeah. When we open the adversarial ML threat matrix Google group, we now went from zero. We felt like nobody's going to join this Google group. It's going to be like a pity party where I'm going to email Michel from MITRE and he's going to respond back to me. But no, we went from zero to 150 right now over just the last four days.Natalia Godyla:Ram, thank you for giving us all of this context on the adversarial ML threat matrix. So what's Microsoft's continued role. What's next for you in ML?Ram Shankar Siva Kumar:First of all, we are hiring. So, if you'd like to come and join us, we are looking for developers to come and join us in this quest. So please email anybody, even Nic, and he can forward his resume. Nic Fillingham:Do you need to have a cowboy hat? Is a cowboy hat a necessity?Ram Shankar Siva Kumar:Not at all. We will accept you for who you are. Natalia Godyla:Do you provide the cowboy hats?Ram Shankar Siva Kumar:We will provide everything. Anything to make you feel comfortable. So we are growing and we'd love to work with the folks. With the adversarial ML threat matrix, like I said, we really are looking for feedback from the community. We really think that like Natalia very correctly pointed out this is a problem so big that we can only solve it if we all come together. So please go to our GitHub link. I'm sure Nic and Natalia [inaudible 00:27:44] the link to it. We'd love to get their feedback.Ram Shankar Siva Kumar:The second thing is if you kind of are... We are especially looking for people to come in at case studies, if you think we're missing a tactic, or if you think that you've seen an attack on a ML system on a commercial Ml system, please reach out to us and we'd be happy to include that in the repository. Nic Fillingham:If your autonomous vacuum cleaner has attempted to undermine democracy, let us know. Ram Shankar Siva Kumar:And the one thing that I want everybody to take away is that when we did our survey, 25 out of 28 organizations did not have tools and processes to kind of secure the ML systems. So if you're listening to this podcast and you're like, "Oh my gosh, I don't have a guidance." Do not feel alarmed. You're tracking with the majority of the industry. In fact, three organizations, all of whom were large in our survey even thought about this problem. So there are tools for you and processes that we put out. So in our docs@microsoft.com, there's a chat modeling guidance, there's taxonomy, there's a bunk bar that you can give to your incident responders so that they can [inaudible 00:28:52] bugs. And for the security analysts community, there is the adversarial ML chat matrix. So please go read them and please give us feedback because we really want to grow.Natalia Godyla:I love it. Thank you for that. That's a great message to end on.Ram Shankar Siva Kumar:Awesome. Thank you, Nic and Natalia for having me. Really appreciate it. This was really fun.Natalia Godyla:And now let's meet an expert in the Microsoft security team to learn more about the diverse backgrounds and experiences of the humans, creating AI and tech at Microsoft. Today, we're joined by Justin Carroll, threat analyst on the Microsoft threat intelligence, global engagement and response team. Well thank you for joining us, Justin.Justin Carroll:Thanks for having me. Natalia Godyla:Well can we kick things off by you just sharing your role at Microsoft. What does your day to day look like?Justin Carroll:So my role is related to threat hunting across large data sets to find advanced adversaries and understand what they're doing. Look for detection opportunities and communicate out the behaviors of the specific threats that we're finding to partner teams or to our customers to help them understand the threat landscape and kind of staying on top of what attackers are doing.Natalia Godyla:That's super interesting. And can you talk a little bit about any recent patterns that you've identified or interesting findings in your last six, eight months?Justin Carroll:Well, it's been a busy six or eight months, I would say, because everybody's been very busy with COVID. We've been seeing quite a large increase in human-operated ransomware and stuff like that. So I've been working really hard to try and figure out different ways to try and surface their behaviors as early as we can to customers to help them take action before the ransom happens. And we've been seeing quite a few other different really advanced adversaries compromising networks. Justin Carroll:A lot of it's kind of the same old, same old, just more of it, but it's always interesting and there's never a shortage of new findings each day and kind of moments of, "Oh, that looks like this, or they're doing this now." Awesome. Great.Natalia Godyla:You mentioned you're constantly trying to find new ways to identify these faster. What are the techniques that you're trying to use to find the threats quicker?Justin Carroll:There's a whole bunch of different ways that you kind of try and surface the threats quicker. Some of it's research and reading other people's work and blogs and stuff like that. I tend to live in the data most of all, where I'm constantly looking at existing attacks and then trying to find similar related behaviors or payloads or infrastructure and pivoting on those to try and attempt to find the attack, to be ready to find it as early as possible. And what's called the kill chain.Justin Carroll:So from the time that the attacker gets in the network, how quick can we find them before they've had a chance to conduct their next set of actions? So whether if they're stealing credentials or something like that, can we surface them before they've had a chance to do the credential theft and then kind of always trying to move earlier and earlier in the kill chain to understand how they got there. And then what are some of the first things that they did when they did get there and how do we surface those next?Justin Carroll:Because a lot of those are a little bit more difficult to surface because it can kind of tend to blend in with a lot of other legitimate activities. Nic Fillingham:What kind of tools do you use Justin? Are you in network logs and sort of writing queries, is there a big giant futuristic dashboard that you sit in front of and you have virtual reality gloves moving big jumps of numbers left and right. Well, what are the tools of your trade? Justin Carroll:So one of the tools that we use a lot, there is a bunch of data that's stored... Customer facing, it's usually called Azure data Lake. It's these huge databases with large amounts of information where you can construct queries with what's called KQL, I believe it's Kusto query language. So there's a specific tool for kind of deep diving into all of that data across our many different sources. And then using that to basically structure and create different queries or methods of finding interesting data and then kind of pivoting on that data. Justin Carroll:Then in addition, I've built some of my own tools to kind of help improve my efficiency or automate some of the stuff that I have to do all the time and then just to make me faster at hunting for the things that I'm looking for.Nic Fillingham:Is it an AI version of yourself? Is it a virtual Justin?Justin Carroll:No. We work with the ML team to try and share as much knowledge with them as possible. There is no tool for an AI Justin, as of yet.Nic Fillingham:Well, let's back it up a bit. So one of the things we would like to do in these interviews with the security SMEs, I'm not even sure if we've explained what an SME yet. We call it a Subject Matter Expert. That's an acronym. We use a lot here at Microsoft. I think it's pretty broadly known, but if you've heard of SME or SME, that's what it means.Nic Fillingham:Now, you and I, we crossed paths about a year ago for the first time when Jessica Payne, who actually hasn't been on the podcast yet, Jessica introduced me to you and she said, "You have to talk to Justin." And she gave me three sort of very disparate, but intriguing bits of data about you. She said, "Justin used to climb telegraph poles. He is a big Star Wars fan and is in a metal band." And I'm sure I've gotten those three things slightly wrong. Could you kind of talk about your journey into the security space and then sort of how you found yourself working for Microsoft. But first of all, these three things that Jessica told me are any of them true?Justin Carroll:Mostly they are. So some of these will kind of combine for the telephone climbing aspect. I used to work for a wireless internet provider that had leases or specific towers, cell phone towers or other towers on top of mountains, essentially, where we would have wireless radio dishes that would communicate to each other. So I was occasionally tasked with installing and or fixing said towers, which is okay if you are fine with heights, I wasn't at first, but you just kind of get used to it. And you kind of realize once you're above 20 feet, it really doesn't make any difference. If you fall, it's going to hurt, but climbing a tower in the winter and in the wind and where you can barely feel your hands and all that wasn't great. Justin Carroll:I was a pretty big Star Wars fan growing up as a kid, even more of a Ninja Turtle fan. And as for metal, I used to be in a band with some friends and have been playing guitar for 25 or 26 years. And music has been a very huge part of my life and remains to be.Nic Fillingham:I think we'll circle back to Ninja Turtles. I'm not going to let that one go, but so let's talk about your path into security. So was this you're working for the wireless internet provider was this your first job. Was this mid career. Where does that fit in your sort of LinkedIn chronology? And at what point did you use formerly into insecurity?Justin Carroll:So it's been a long and winding road to get here I would say. So the internet provider was what I would guess I'd call my first career job of sorts. I had started there in my early 20s and worked for them for about... sorry my cat is right in front of the microphone. One second. Nic Fillingham:There's a cat there. Justin Carroll:She wanted to say her piece. So I worked for the internet company for just under a decade. I used to do some networking type fun stuff in Halo 2, to kind of maybe garner a little bit of an advantage, I guess I would say, and use those learned skills to land that first job. And I did that for quite a while, but realized I was kind of stuck in this job. It was in a city that I didn't want to live in. And I had kind of maxed out my capabilities there. I had attempted to move to Portland because I wanted to have a bigger city. Experienced, applied to 254 jobs, got one interview for basically a office tech support role was the only position I got hired, but it wasn't feasible to live in Portland.Justin Carroll:So after quite a bit of soul searching and realizing that basically nobody cared that I had eight years of on the job experience because I didn't have a college degree. There were not any doors open for me for the most part. I then decided to take a pay cut and go get a job at a university that was just a city over and work full-time and go to school for a degree in cybersecurity while working full-time for the university doing kind of technical work for them, helping them understand their... Sorry, my cat is a whole thing right now.Nic Fillingham:Your cat's just trying to interject with like don't. Hey, you glossed over that Halo 2 thing, you better to come back to that.Justin Carroll:[inaudible 00:37:59] come here. Nic Fillingham:We're leaving all this in, by the way. Natalia Godyla:Yeah. We're very much enjoying it.Justin Carroll:So kind of advising the university on different technologies that they could use for their students. So I did that for about three and a half years while going to school and then graduated top of my class and applied for another 150 some odd jobs and mostly the Seattle area this time and was about to give up because even though I now had a degree and almost 10 years of experience, it still wasn't enough. And everybody that I kept losing to had between 10 and 20 years experience. And it just wasn't an option for folks with less specific cybersecurity experience to kind of enter the field. Justin Carroll:There were a lot of walls that were put up. I had a friend of a friend who worked for cybersecurity at a company somewhere in Arizona, who I'd never met. And he decided to go out of his way, even though I'd never met him and looked for some cybersecurity type jobs in my area that he thought maybe I'd be good for and helped me look at my resume and stuff like this. And that helped me land a vendor role for Microsoft, where I kind of started my path and career towards cybersecurity specific stuff.Justin Carroll:I had basically given up at that point on ever working in cybersecurity and had kind of thought that it just wasn't meant for me. So that was kind of a big break and a guy almost closed the application to apply for the job and then figured what's the worst they can say is no, that is kind of how I finally got to Microsoft and cybersecurity, where I was able to work as a vendor for the team evaluating kind of telemetry. And I was kind of given an opportunity to learn a lot and that eventually transitioned into when a position became available, where I started working full-time as a Microsoft employee and went from there.Natalia Godyla:So what in your soul search brought you to cyber security? Was it your background, the fact that you already had those foundations as a network admin, or was there something in particular in the cybersecurity world that just attracted you?Justin Carroll:I'd always found it fascinating. When I started university, they just launched the cybersecurity program. The quarter that I started there, and one of my friends who was a computer science major, basically called me up immediately and was like, "Hey, they just launched this. You need to do this." And there's the very popular culture aspect of it where everybody thinks it's fascinating and you sure there was a little bit of a grab with that. But I like learning how computers work and I like kind of the constant problem solving nature of everything. And the first class I took on it I was hooked and still remains that day where it's just, it's fascinating and it's really fun to just kind of continually work to see what attackers are doing. But I also, there's a huge aspect of it like I like helping people. I think it's important and having a role where I'm able to help millions or even potentially billions of people through better detections or stopping malware. It feels pretty great.Nic Fillingham:What other aspects Justin, of your path to security, your path to Microsoft, do you feel you're sort of bringing forward? I want to ask about you very briefly mentioned something about Halo 2 and I want to know what that was. And then I wonder if there were other sort of dare I say, sort of maybe unorthodox or non-traditional things that you worked on where you learned a bunch of bunch of tools or tricks of the trade that you're bringing forward to your work right now. Justin Carroll:So Halo 2 was a fun one. Back in those days, there were lots of what were called modders, who would mod their Xbox's to gain an unfair advantage. So I would use my networking know-how basically, and learned a lot of it too, when encountering a modder to kick them out of the game. I think it was possibly a little frowned upon, but I was tired of having cheaters constantly win, so I did a lot of research and I didn't know a whole lot about networking at that point, but I tried to not use it as a competitive advantage, but more to just level the playing field, but it was a great way to learn how firewalls worked and network traffic and building more on my understanding of computers. Justin Carroll:And then kind of that side of foundation for me, of understanding, there's always going to be stuff that I don't know and what I have done, but I did it all through college and continued all the way till basically getting full-time employment at Microsoft was I set up a lab environment and I would set up servers and clients and I would attack them and monitor the logs on my own little private lab on my machine and see what worked, what didn't, try and figure out why it worked, what didn't and try and build different tools to see how I could make it more effective or deal with different issues.Justin Carroll:Just kind of both playing attacker and defender at the same time on my network, all by myself, essentially and kind of learning from all of that data was massively important and anybody who's looking to get into security, I highly recommend both learning how to attack, on a safe, your own little lab environment where you're not hurting anybody. And what's it like to try and defend and find those attacks because both sides are-Nic Fillingham:Red Justin versus blue Justin. Justin Carroll:Exactly. Yes.Natalia Godyla:You noted earlier that just the sheer amount of data can be overwhelming, especially as you moved through your career and then came to Microsoft where we have billions of signals. So the same transition happens from Halo to now just the sheer scale and scope of your role and the amount of good that you can do. So, how did you handle that overwhelming amount of information, amount of impact that you can have?Justin Carroll:So when I was first brought on one of the things that made a significant difference was I had somebody that kind of instructed me in a lot of the ways of kind of how to work with the data, but I was also given quite a bit of an area for trial and error. So there was lots of opportunity to fail and to learn from what didn't work and to kind of keep building on that. And then any time that I got stuck or I would kind of just do everything I could to attempt to solve the problem or work with the data. If I kind of hit a wall that I couldn't climb on my own, I could go to him and then we would solve it together. So it was kind of both a mentoring and a guidance thing, but also kind of given that ability to experiment and try and learn. So that was kind of one of the biggest ways of learning to pivot on that data and understand it and consume it.Justin Carroll:And then honestly, collaboration with other folks on my team and other team was massively instrumental to be able to kind of learn what they had already learned or pass on my knowledge to them. And just that constant sharing and understanding because there is so much data, it's quite impossible almost to be an expert at all of it. So having those folks that you can reach out to you that are experts in each basically set of their data. So you can understand what the data is trying to tell you, because that's one of the things that is particularly difficult is to take the data and actually glean understanding from it. The data is trying to tell you something, you just need to make sure you're interpreting the message correctly.Natalia Godyla:How do AI and ML factor into your role into helping you manage this data and collaborating with other teams.Justin Carroll:So I work quite a bit with a lot of different data science folks on a few different teams to either use a lot of the models that they're creating to kind of a source, a lot of the malicious information or a particular attackers or stuff like that. And then also collaborating back in sharing my knowledge and intelligence to them to say, this is what an attack looks like. This is what it should look like in the data and kind of giving them the ideas and signals for what they should be looking in their data to kind of train those models. Justin Carroll:It's really important to have that partnership between security and data science for AI and ML to kind of help them understand the security sphere of it. And then they can kind of take the real math and data prowess that they've got and turn our knowledge into ML or AI to detect and surface a lot of these things. Nic Fillingham:If it's possible, Justin, how would you sort of summarize your guidance to other Justin Carroll's that are out there that are... They want to get into security, they're fascinated by cybersecurity in sort of a macro sense, but they feel either don't have a degree or they're not even sure what they should go study or they're trying to work at, how can they translate their current sort of career experience and sort of skills? Can you summarize that into some guidance of what folks should do to try and break in?Justin Carroll:Sure. One, if you're in school, remember that school is not going to teach you a lot of the stuff that you need to know. It's lots of taking what you're learning and building upon it outside. So if it's cybersecurity, that's an interest, try and experiment and fail. Cyber security is huge. There are so different facets of it. Find out the thing that kind of scratches the itch and peaks your interest. For me, that was setting up a lab, right? Where I could play both the attacker, the defender, the person monitoring logs, the person setting up all the configurations to try and stop the attacks and was able to kind of see all different aspects of the industry. Nic Fillingham:So just jumping in, was that literally just a bunch of VMs on your machine or did you have multiple PCs sort of networked together? Just very quickly, what did that look like? How accessible is setting up a lab? I guess I'm what I'm asking. Justin Carroll:It is pretty accessible. So while I was in college, it was actually multiple machines and I had four different machines and I set up a router that you can pick up for 50 bucks and a smart switch that I could mirror the traffic on to understand everything for 100 bucks. So there's a little bit of cost. That was kind of my college setup. And as I was kind of learning where I at that point, it made a little more sense to do it with actual machines and for extra clarity. My college was only a couple of years ago. I did not go to college. So the next route that I did once I headlined did my vendor role and was kind of like securities for me and I want to keep building on it.Justin Carroll:I did it all with VMs. So I just had a desktop computer that was okay. Specifications and I configured two clients, the domain controller, server on the device and then a mail server. And then basically you just connect to each client and then network them all together. So at that point you can use VirtualBox, you can use lots of different stuff. So the availability of doing that, it's actually pretty good. There isn't a lot of overhead costs or anything like that. You just have to have a okay computer.Natalia Godyla:What about resources to learn how to do all of that? Are there organizations or sites that someone could turn to, if they're interested in starting to do some of this starting to experiment with what they're interested in?Justin Carroll:Honestly, I would say one of the best resources that I had throughout was YouTube. It was a great place to get walkthroughs for every different thing. So like I wanted to learn how to set up a VM and configure it with networking to another VM. I turned to YouTube. I wanted to learn how to attack the VM using Kali Linux, YouTube. And there's a whole bunch of different channels out there that specifically focus on that. And then the other thing is because it's so much more open for creators to share content. You can find people who are at a similar level or maybe just a few steps ahead of you. So you can really kind of join along with other people. Justin Carroll:There are a few websites for coding, I think one's called hacking the box as far as attacking different things. And that was also kind of fun where a lot of the devices that need to be attacked we're already pre-configured for you. But for me, honestly, a lot of the fun was setting up those devices and then learning what I did that worked and didn't and what allowed it to be attacked and what I could do to stop that.Natalia Godyla:Quick plug Microsoft security also has a YouTube channel in case somebody would like to get any, how to content on our products.Nic Fillingham:Natalia [inaudible 00:51:08] may have been involved in that channel, just full disclosure there.Natalia Godyla:Yeah. I couldn't help myself. But it is also great to hear that you found people to work with in the community as well. That's something that's been noted by a few of our guests, like Michelle Lamb, that as she was entering the space, she found mentors. She found conversations, people readily available to either work on a problem alongside her, or just answer questions. So I'm glad that you've also been able to turn to the community for that. So what's next for you? Is there a new challenge that you'd like to solve?Justin Carroll:Definitely want to work on the toolkit that I'm building and kind of continue that growth. It's been interesting to kind of see the hurdles I run into. And even last week I ran into one that felt insurmountable and was able to chat with one of the devs and solve in a few minutes and learned a whole lot and going forward, now I have that in my pocket. And then both-Nic Fillingham:Hang on. Did you say you went from found a new challenge, thought all this is insurmountable and then a few minutes later you solved it?Justin Carroll:With a little support from people that knew how to solve the problems. So collaborating with like one of the other devs on the team and basically having him kind of explain the part it felt like a giant wall, but really once you kind of have somebody to break it down a little bit for you, it was just like, "Oh, okay. I see what I'm missing here." And then it was just like, "Got it. Okay. Moving forward."Nic Fillingham:Oh, I see. So that that's more an endorsement. Yeah, I got it. Justin Carroll:Yeah. Yeah. It's more an endorsement of others teaching abilities and just kind of those times of being able to reach out to others for when you really get stuck and how much of a difference it can make. I had spent an hour on something and was just like, this is ridiculous. This should work. Why isn't it working? What's wrong with me. I'm not smart. And then just chatting with them a little bit and then figuring it out and then like, "Oh, okay. Oh, okay. That's actually pretty simple." I wasn't thinking about it in the right way and kind of getting that other perspective. Justin Carroll:And then what's next kind of going forward is a kind of continued partnership with a lot of the data science folks to, I think we've only scratched the surface in many ways as an industry on how data science and cybersecurity can work together. So I am very excited to kind of see what kind of stuff we can accomplish, whether it's, you know, surfacing attacks shortly after they happen, very early in the kill chain or understanding related behaviors and trying to understand who the might be, or I think most of all, the intent of the attack or adversary.Justin Carroll:Intent can sometimes be a very difficult to suss out, even for socks and their entire center. They have all these folks that are trying to figure out what happened. Why did it happen? What does it actually mean? So if we can have data science that can provide a lot of context on that, through understanding existing attacks and modeling what future ones might look like, I think there's some pretty exciting opportunities there.Nic Fillingham:All right, I'm doing it. We're coming to Teenage Mutant Ninja Turtles. You're a fan. How much of a fan are you, Justin?Justin Carroll:I'd say quite a fan. I do have a couple of figurines and a mint package on open from 87 I think something like that. And then have a Ninja Turtles tattoo on my back of Raphael. So that was kind of one of those moments where I was trying to think about what steps I wanted to take forward in life and things like that. And I had kind of thought about what are the things that actually make me happy? Justin Carroll:This was probably my mid 20s quarter life crisis kind of thing. And I was like, "I always liked the Ninja Turtles as a kid." They always brought me great joy. I still get excited about watching them. The movies are definitely a guilty pleasure. I realized they're not great. But now I'm talking about the original movies, not the new ones. We won't talk about the new movies. And it was just one of those like, "Yeah, I identify with this. This is a huge part of my life. It's been around since I was... it was started the year I was born." So I was just like, "All right, let's do it." And haven't regretted it at all.Nic Fillingham:I was going to ask who your favorite turtle was, but you've obviously... If you've inked Rafaelle on your back so that question is moot. I'm a Donatello guy. I've always been a Donatello guy.Justin Carroll:I would think of myself as Raf, but really I'm more of a Donatello. Ralph was kind of the cool guy with a little bit of an attitude, but really I was Donatello. When I was 10 dressed up for Halloween, I was Donatello. I'm definitely Donatello with a little bits Raf thrown in for good measure.Nic Fillingham:Well, this has been a blast. Thank you, Justin, for walking us down, Teenage Mutant Ninja Turtle memory lane, and Halo 2 memory lane and sharing your story with us. It was great. Wonderful to get your perspective. Great to have you as a part of the threat hunter team here at Microsoft and contributing in all the ways that you do. Thanks for joining us. I'm sure we'll talk to you again at some point on the Security Unlocked podcast, but keep doing you Cowabunga, dude.Justin Carroll:Thanks very much for having me. I appreciate it. It was great to talk to you all.Natalia Godyla:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us @msftsecurity or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then stay safe.Natalia Godyla:Stay secure.
12/23/2020

Tackling Identity Threats With AI

Ep. 8
The last thing we all need this year is an identity crisis.Fear not, hostsNic FillinghamandNatalia Godylaare here withMaria Puertas Calvo, Data Science Lead of Microsoft’s Identity Security and ProtectionTeam, to learnhow AIisbeing used to protect ourpersonalidentities.Maria also reveals previously undisclosed information – her favorite food and her famous top-secret recipe, so get ready to take notes!Later,thehostsbringback a previous guest,Geoff McDonald,ML Research Lead at Microsofttounpackhis career in cybersecurityand how game hacking led him to where he is now.In This Episode, You Will Learn:How offline detections are used for account compromise preventionThe importance of multi-factor authenticationHow Microsoft is taking a new approach with AI to identify threats with real-time preventionThe problem with adversaries and malware attackersSome Questions We Ask:How is Microsoft applying AI to solve problems for account compromise prevention?How do humans play a role inlabeling data sets?How is Microsoft measuringsuccessof their new enhanced AI?What is the future for neural networks?ResourcesMaria’s Bloghttps://techcommunity.microsoft.com/t5/azure-active-directory-identity/enhanced-ai-for-account-compromise-prevention/ba-p/1994653Microsoft Security Bloghttps://www.microsoft.com/security/blog/Nic’s LinkedInhttps://www.linkedin.com/in/nicfill/Natalia’s LinkedInhttps://www.linkedin.com/in/nataliagodyla/Maria’s LinkedInhttps://www.linkedin.com/in/mariapuertas/Geoff’s LinkedInhttps://www.linkedin.com/in/geoff-mcdonald-76655029/Transcript(Full transcript can be found athttp://aka.ms/SecurityUnlockedEp08)Nic:Hello and welcome to Security Unlocked. A new podcast from Microsoft where we unlock insights from the latest in news and research from across Microsoft Security engineering and operations teams. I'm Nick Fillingham.Natalia:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft Security, deep dive into the newest threat Intel research and data science.Nic:And profile some of the fascinating people working on artificial intelligence in Microsoft Security. If you enjoy the podcast, have a request for a topic you'd like covered or have some feedback on how we can make the podcast better-Natalia:Please contact us at securityunlocked@microsoft.com or via Microsoft Security on Twitter. We'd love to hear from you.Nic:Hello, Natalia. Welcome to episode eight of security unlocked. How are you?Natalia:I'm doing great. We're right about at Christmas. I am feeling it in my onesy right now.Nic:You're feeling Christmas in your onesy? Is it a Christmas onesy?Natalia:No. I feel like onesys just highlight the Christmas spirit. I mean, you're in PJs all weekend.Nic:We've been in work from home for seven years now. We're all in perpetual onesy land.Natalia:Well, I mean, I try to put in effort. I don't know about you.Nic:I don't put any effort. I wonder if we should issue a subscriber challenge. I wonder if we could hit 1000 subscribers. We might make a security unlocked onesy. I wonder what other swag we could do? What would be good for a security unlocked podcast?Natalia:All right. I mean, I guess I'm a little biased but the security blanket is clever. The ones that Microsoft gives away.Nic:I don't think I have one of those.Natalia:It's a blanket with security images on it.Nic:Images of security in it? Just images of very strong passwords. Images of two factor authentication. What about a horse blanket? Like a blanket you put over your horse?Natalia:What does that have to do with security?Nic:Under the saddle. I'm just following the blanket thread, that's all. I'm just thinking different types of blankets. In two episodes have already talked about the bratty pigs. I wonder if we could turn the bratty pigs into our mascot and on the security blanket there could be like an animated picture of the bratty pigs running away with a padlock and key or something.Natalia:Have I not, and excuse the pun, unlocked the new technology in blankets and animated pictures? Is that possible and blankets now?Nic:Did I say animated? I meant illustrated, I'm sorry. Oh wow, I bet you there's some brand new piece of printing technology that's over in like Japan or South Korea that we haven't got over here yet where they've got animation on their blankets, that would be good. What about one of those automatic cat feeders for when you go away on holiday and it dumps a little bit of dry food into their bowl every 12 hours? And then we just put security unlocked on the side of it.Natalia:As long as it has our logo on, it fits.Nic:You know what? Also, this is our last episode for 2020.Natalia:How'd you feel about it?Nic:About this episode or about the year of 2020?Natalia:Well, the year 2020 is probably too much to unpack. What about our podcast adventure in 2020?Nic:Yeah, I've enjoyed it greatly. I listened to the first couple of episodes just the other day. And while they were great, I certainly heard an evolution in just eight episodes from that humble first back in October. So yeah, I've definitely enjoyed the trip. I'm very much looking forward to 2021. What about you?Natalia:I feel like our guests are making me smarter. With each new episode. I've got a few more terms under the belt. Terms I'd heard before but never got that clarity from experts and what the definition is especially as they're moving around. We see that with a lot of the machine learning and AI terms. Like neural networks when we're talking to experts, they have different lenses on what that should mean.Nic:The other thing that I found fascinating is everyone that you and I have reached out to internally, Natalia, and said, "Hey, do you want to be a part of this podcast?" Everyone said, Yes. Everyone has said, "Yeah, I'd love to share my story of how I got into security. I'd love to share my story of how I got to Microsoft." I love that we've spoken to such a incredible variety of people that have come to security and to Microsoft from just... I mean, everyone has a completely different story and everyone's been so willing to tell it. So I'm just very, very happy that we've been able to meet these great people and have these conversations.Natalia:Yes. And even in their diversity, I've been happy to see that there are really positive themes across the folks that wants to be in security that are in the security space. They're all so passionate about what they do and really believe in the mission, which is just great to see. And like you said, there's just awesome community. The fact that they want to go out and have these conversations and are always open to receiving questions from you guys. So please keep them coming. Our experts are equally as hungry as we are to hear not just feedback but questions on the topics that we discuss.Nic:So on today's episode, we chat with Maria Puertas Calvo. Fantastic conversation, very excited to have Maria on the podcast. I'm not sure if many folks picked up but a lot of the experts we've spoken to so far have been more on the endpoint detection side of the house. We've talked to folks over in the defender team and those who sort of look at the email pipeline. Maria and her team focused on identities, so protecting identities and protecting our identity platforms. And so she's going to talk about how AI and ML are used to protect identity. And then after Maria, we talked to...Natalia:Jeff McDonald. So he is a member of the Microsoft defender for endpoint research team. And he's joined us on a previous episode to talk about unmasking malicious threats with MC and ML. And today, he's chatting with us about his career in cybersecurity, which started with game hacking. So making changes in the game to get more skills, get new characters and he's got some amusing stories as to how far he took that. But it's also a theme we're seeing across a few of our guests that game hacking seems to be a gateway to cyber security.Nic:Yeah, hopefully the statute of limitations on game hacking has well and truly expired on the various games that Jeff mentions in his interviews. I hope we're not getting him in trouble. Enjoy the pod, and we'll see you all in 2021.Nic:Maria Puertas Calvo, thank you so much for joining us. Welcome to the security unlocked podcast.Maria Puertas Calvo:Hi, thank you for having me.Nic:If you could tell us about your role at Microsoft and what your day to day looks like in the team you're in. The mission and sort of scope of that work, that'd be great.Maria Puertas Calvo:Yeah, absolutely. So I am a principal data science manager in identity security and protection. So I lead a team of five data scientists that work within a big engineering team. And our big mission is to protect all of Microsoft's users from account compromise and other things like the [Bs 00:07:10] and fraud. As a data science team, we just analyze and look through all the huge amount of data that we get from all our customer logs and everything. And then we use that to build automated statistical based models or machine learning models or heuristic made models that are trying to detect those bad options in our ecosystem. So compromised attacks or malicious bots that are trying to do bad things in our identity systems.Natalia:And Maria, we understand that your team also recently authored a blog on enhanced AI for account compromised prevention. So can you talk a little bit about what that blog entails, how we're applying AI to start solving some of these problems?Maria Puertas Calvo:Yeah, we're actually really excited about this work. But it just went into production recently and it has really enhanced what we call the bread and butter of really what we do. Which is trying to prevent compromise from happening in the ecosystem. Basically, we have been using artificial intelligence and AI to build detections for a pretty long time. And everything that we do, we try to start with whatever the long hanging fruit. We do offline detections, which are basically using the data after authentications or attacks already occurred and then detect those bad attacks and then we will inform the customer or make the customer reset their password or do some type of remediation.Maria Puertas Calvo:But being able to put AI at the time of authentication and so meeting that end goal that we're trying to not just detect when a user has been compromised and remediate it but we're actually able to prevent the compromise from happening in the first place. So this new blog talks about this new system that we've built. We already had real time compromised detection but it wasn't using the same level of artificial intelligence.Natalia:So is it correct to say then that in the past we had been doing is identifying a known attack, a known threat, and then producing detections based on that information and now we're trying to preempt it? So with this even more intelligent AI, we're trying to identify the threat as it's happening, is that correct?Maria Puertas Calvo:Yeah, that's correct. So we did already have real time prevention but most of our artificial intelligence focus used to be in the, after the fact. Now we have been able to move this artificial intelligence focus also to the real time prevention. And what we have achieved with this has really improved the accuracy and the precision of this detection itself. Which means now we're able to say that the signings that we say are risky, they're way more likely to actually be bad than before. Before we would have more noise and more false positives and then we would also have some other bad activities that would go undetected.Maria Puertas Calvo:With this new artificial intelligence system, we have really increased the precision. Which means, now if a customer says, "Oh, I want to block every single medium risk log in that comes my way that is trying to access my [tenant 00:10:17]." Now, fewer of their real users are going to get blocked and more actual attackers are going to get blocked. So we've really improved the system by using this new AI.Natalia:What's changed that's increasing the precision?Maria Puertas Calvo:Yeah, so we actually published another blog with the previous system which was mostly using a set of rules based on user behavior analytics. So the main detection before was just using a few features of the signing itself and comparing them to the user history. So if you're coming from a new IP address, if you coming from a new location, if you're coming from a new device, there was like a deterministic formula. We were just using a formula to calculate a score which was the probability of how unfamiliar that finding was. Now we're taking way more inputs into account. So we're using... It depends on which protocol you're using.Maria Puertas Calvo:It has more intelligence about the network, it has some intelligence about what's going on. for example, if you're coming from an IP address that has a lot of other traffic that AD is seeing, it has also information about what AD is saying from that IP address. Does it have a lot of failed logins or is it doing something weird? And then instead of us manually setting a mathematical formula or rules in order to build that detection, what we do is we train an algorithm with what is called label data. So label data is just a set of authentications, some are good and some are bad and they're labeled as such. So we use that label data to tell the algorithm, "Hey, use this to learn," Right? That's how machine learning works.Maria Puertas Calvo:So the algorithm trains and then it's able to use that data to decide in real time if the authentication is good or bad.Nic:Yeah, thank you. And then where, if any, do human analysts or humans in specialty roles, if it's data science or analytics, when do they come in to either verify the results or help with labeling new sets of data? So you've got your known goods, you've got your known bads and I assume you end up with a bunch of unknowns or difficult to classify one way or the other. Is that a role for a human analyst or human data scientists to come in and create those new labels?Maria Puertas Calvo:Yeah, even though getting all this labels is extremely important. That is not really what... The data scientist is not there just classifying things as this is good, this is bad, just to get labels to feed it to the algorithm, right? What the data scientist does that is very crucial is to build the features and then train this machine learning model. So that is the part that is actually really important. And I always really try to have everybody in my team to really understand and become a great domain expert on two things, One is the data that they have to work with. It is not enough to just get the logs as they come from the system, attach the label to it and then feed it to some out of the box classifier to get your results.Maria Puertas Calvo:That is not going to work really well because those logs by themselves don't really have a lot of meaning. If the data scientist is able to really understand what each of the data points that are in our laws, sometimes those values, they're not coded in there to be features for machine learning. They're just added there by engineers to do things like debugging or showing locks to the user. So the role of the data scientist is really to convey those data points into features that are meaningful for the algorithm to learn to distinguish between the attack or the good. And that is the second thing that the data scientist needs to be really good at. The data scientist needs to have a very good intuition of what is good and how that looks in the logs versus what is bad and how the looks in the logs.Maria Puertas Calvo:With that [battle 00:14:04] knowledge basically knowledge of what the data in the logs mean and the knowledge of what attack versus good look in that data, then that is the feature engineering role. You transform those logs into all their data points that are calculations from those logs that are just going to have a meaning for the algorithm to learn if something is good or an attack. So I can give an example of this, it's very abstract. For example, when I see an authentication in Azure AD logs maybe one of the columns that I'd want him to know is like IP address, right? Every single communication over the internet comes from some client IP address which will be the IP address that's assigned to the device that you are on at the time that you're doing an authentication.Maria Puertas Calvo:There are billions, if not trillions of IP addresses out there. And each one is just some kind of number that is assigned to you or to your device and it doesn't really have any meaning on its own. It's just like if you have a phone number, is that a good or a bad phone number? I don't know, that's just not going to help me. But if I can actually go and say, "Okay, this is an IP address but is this an IP address that Nick use yesterday or two days ago? How often have I seen Nick in this IP address? What was the last time I saw Nick in this IP address?" If you can just play with those logs to transform it into this more meaningful data, it's really going to help the model understand and make those decisions, right?Maria Puertas Calvo:And then you also end up with fewer things to make decisions on, right? Because if I just had that one IP address to train the model, maybe my model will become really good at understanding which IP addresses are good and bad but only among the ones that we have used to train that model. But then when a new one comes in, the model doesn't know anything about that IP address, right? But if we instead change that into saying, "Okay, this is a known IP address versus an unknown IP address," And then now, instead of having trillions of IP addresses, we just have a value that says, Is it known or unknown. Then for every single new log in that comes in, we're going to be able to know if it's known or unknown.Maria Puertas Calvo:We don't really need to have seen that IP address before, we just need to compare it to the user history and then make that determination of it is this known or unknown and that ends up being much more valuable for the model.Natalia:So just mapping out the journey you've talked about. So we've gone from heuristics signature based detections to user analytics and now we're in a space where we're actively using AI but continuously optimizing what we're delivering to our customers. So what's next after this new release of enhanced AI? What is your team working on?Maria Puertas Calvo:So lots of things but one thing that I am really interested in that we're working on is making sure that we're leveraging all the intelligence that Microsoft has. So for example, we built a system to evaluate in real time, the likelihood that a finding is coming from an attacker. But all of that is just using the data that identity processes like Azure Active Directory sign ins and what's happening the Azure Active Directory infrastructure. But there's so much more that we can leverage from what is happening across the ecosystem, right? Like the user who signs into Azure Active Directory is probably also coming in from a Windows machine that probably has Microsoft dependent Defender ATP installed on it. That it's also collecting signal and it's understanding what it's happening to the endpoint.Maria Puertas Calvo:And at the same time, when the sign in happens then the sign in doesn't happen just to go to Azure AD, right? Azure AD is just the door of entry to everything, Usher, Office, you name it. Third party applications that are protected by things like Microsoft Cloud App Security. And all of the security features that exist across Microsoft are building detections and collecting data and really understanding in that realm, what are the security threats and what's happening to that user? So there is a journey, right? Of that sign in. It's not just what's happening in Azure AD but it's everything that's happening in the device. What's happening in the cloud and in the applications that are being accessed after.Maria Puertas Calvo:So we're really trying to make sure that we are leveraging all that intelligence to enhance everything that we detect, right? And that way, the Microsoft customer will really benefit from being a part of the big ecosystem and having that increased intelligence should really improve the quality of our risk assessment and our compromise detections.Nic:Maria, how much of this work that you talked about in the blog and the work that your team does is trying to mitigate the fact that some folks still don't have multi factor authentication? Is any of this a substitute for that?Maria Puertas Calvo:We know from our own data studies that accounts that are protected by multi factor authentication, which means every time they log in, they need to have a second factor, those accounts are 99.9% less likely to end up compromised because even if their password falls in the hands of a bad actor or get gassed or they get phished, that second factor is going to protect them and it's way more likely to stop the attack right there. So definitely, this is not supposed to be a substitute of multi factor authentication. Also, because of that, our alerts do not... They still will flag a user if the sign in was protected by multi factor authentication but the password was correct. Because even if there's multi factor authentication, we want to make sure that the user or the admin know that the password was compromised so they're able to reset it.Maria Puertas Calvo:But the multi factor authentication is the tool that is going to prevent that attack. And you asked earlier about what's next in other feature things and one thing that we're also really working on is, how do we move past just detecting these compromises with the password of using multi factor authentication as a mitigation of this risk, right? Like the way a lot of the systems are implemented today is if you log in and we think your log in is bad but then you do MFA. That is kind of like a reassuring things that we committed a mistake, that was a false positive and that's a remediation event. But the more people move to more MFA and more password less, our team is starting to think more and more of what's the next step?Maria Puertas Calvo:How are attackers are going to move to attacking that multi factor authentication. It is true that multi factor authentication protects users 99.9% of the time today but as more people adopt it, attackers are going to try to now move to get to bypass our multi factor authentication. So there's many ways but the most popular multi factor or second factor that people have in their accounts is [tough 00:20:46] any base. So there's SMS or there's a phone call in which you just approve the Sign In. There are phishing pages out there that are now doing what is called real time men in the middle attack in which you put your username and password, the attacker grabs it, puts it in the actual Azure AD site and then now you're being asked to put your SMS code in the screen. So the attacker has that same experience in their phishing site, you put in your code and the attacker grabs the code and puts it in Azure AD sign in page and now the attacker has sign in with your second factor, right?Maria Puertas Calvo:So two challenges that we're trying to tackle is, one, how do we detect that this is happening? How do we understand that when a user uses their second factor, that is not a mitigation of the risk? It's more and more possible with time that attackers are actually also stealing this second credential and using it, right? So we need to make more efforts in building those detections. And the second really big thing is, what then, right? Because if we actually that the attacker is doing that, then what is the third thing that we asked you? Now you've given us a password, you've given us a second factor, if we actually think that this is bad, but it is not. What is the way for the user to prove that it's them, right?Maria Puertas Calvo:So we need to move and I think this is extremely interesting, we need to move to from a world in which the password is the weak crab and everything else is just considered good. which today, it's very true. If you have a second factor, that is most likely going to be just fine but in the future, we we need to adapt to future attacks in which this won't be the case. So we need to understand what is the order of security of the different credentials and what is the remediation story for attacks that are happening with these second factors.Nic:I'd like to propose that third challenge, that third factor, should be a photograph of you holding today's newspaper doing the floss or some other sort of dance craze that's currently sweeping the nation.Maria Puertas Calvo:Sure, we'll add it to the bar code.Nic:I think that would just stamp out all identity theft and fraud. I think I've solved it.Maria Puertas Calvo:You did. I think so.Natalia:I think you'll be bringing back newspapers along with it.Nic:Yes. Step one is to reinvigorate the print newspaper industry. That's the first step of my plan but we'll get there.Natalia:So Maria, in your endeavors? How are you measuring success, for instance, of the new enhanced AI that your team has developed?Maria Puertas Calvo:Yeah, so our team is extremely data driven and metric driven and everything we do, we're trying to improve on one metric, right? The overall team mission really is to reduce the amount of users who fall victims of compromised account or what we call unauthorized access. So we have a metric that we all review every single day, we have a huge dashboard that is everybody's homepage in which we see in the last three months, what percentage of our monthly active users fell victim to compromised account and our main goal is to drive that metric down. But that is really the goal of the whole team including the people who are trying to make users adopt MFA and conditional access and other types of security measures.Maria Puertas Calvo:When we look into detection metrics and the ones like the AI detection metrics, we mostly play with those precision and recall metrics that are also explained in the blog. So precision is the percentage of all of the detected users or detected signings that you detected as bad that are actually bad, right? Out of everything that, let's say, you would block, how many of those were actually bad? So it really also tells you how much damage you're doing to your good customers. And the other one is recall and recall is out of all the bad activities that are out there, so let's say all the bad sign ins that happen in a day, how many of those that your system catch?Maria Puertas Calvo:So it's a measure of how good you are at detecting those bad guys. And the goal is to always drive those two numbers up. You want to be really high precision and you want to be really high recall. So every time we'll have a new system and a new detection or whatever it is or we perform improvements in one of our detection, those are the two metrics that we use to compare the old and the new and see how much we've improve.Natalia:And how are we getting feedback on some of those measures? And what I mean by that is the first one you mentioned. So precision, when you're saying how many were actually bad and we need to figure out how many were the true positive? How do we know that? Are we getting customer feedback on that or is there a mechanism within the product that lets you know that it was truly a bad thing that was caught?Maria Puertas Calvo:Yeah, so the same label and mechanisms that I was talking about earlier that we need both labels to be able to train or supervise machine learning models, we also need those labels in order to be able to evaluate the performance of those machine learning models. So knowing at least for a set of our data, how much is good and how much is bad and understanding what our systems are doing to detect the good and the bad. So one of the mechanisms is, as I was talking, the manual labeling that we have in place but the other one you mentioned is customer feedback, absolutely. Actually, one of the first thing we did when we launched editor protection is to include feedback buttons in the product.Maria Puertas Calvo:All of our detections actually go to an Azure Portal [inaudible 00:26:12] in the identity protection product and admins there can see all of the risky sign ins and all of the risky users and why they were detected as risky. Everything that my team is building gets to the customer through that product. And that's where the admin can click buttons like confirm safe or confirm compromised. Those are labels that are coming back to us. And users now also, there's a new feature in entity protection called My Finance. End users can go to my sign ins and look at all their recent signings that they did and they can flag the ones that they think it wasn't them. So if they were compromised, they can tell us themselves, this was not me.Maria Puertas Calvo:So that is another avenue for us to understand the quality of our detections. And then we're extremely customer obsessed as well. So even, it's not just the PMs in our team who have customer calls. The data scientists, many, many times get on calls with customers because the customers really want to understand what's the science behind all of these detections and they want to understand how it works. And the data science teams also wants the feedback and really understand what the customer thinks about the detection. If we're having false positives, why is that? It's really challenging too in the enterprise world because every tenant may have a different type of user base or different type of architecture, right?Maria Puertas Calvo:We had a time that we were tracking... We always track what are the top 10 [inaudible 00:27:32] that get flagged by the technicians. For example, airlines used to be a big problem for us because they had so much travel that we had a lot of false positives, right? We were flagging a lot of these people who because they're flying all over the world and signing in from all over the world. So it would trigger a lot of detections but there are other customers in which this is not the case at all. All of their users stay put and they're just only logging in from the corporate network because it's a very protected environment. So this quality of detections and this precision and recall can really vary customer by customer.Maria Puertas Calvo:So that is another challenge that I think we need to focus more in the future. How do we tune our detections in order to make more granular depending on what the industry is or what type of setup the customer or the tenant has.Nic:Changing subjects just a little bit and maybe this is the last question, Maria. I noticed on your Twitter profile, you refer to yourself as a guacamole eater. I wondered if you could expand upon that. There are very few words in your bio but there's a lot of thought gone into those last two words. Tell us about eating guacamole.Maria Puertas Calvo:Well, what can I say? I just really love guacamole. I think I may have added that about a year ago, I was pregnant with my twins who were born five months ago and when you're pregnant with twins they make you eat a lot of calories, about 3000 calories a day. So one of the foods that I was eating the most was guacamole because it's highly nutritious and it has a lot of calories. I went on a quest to finding the best recipe for guacamole and-Nic:Okay, walk us through your best guacamole recipe. What's in it?Maria Puertas Calvo:Absolutely. So the best guacamole recipe has obviously avocado and then it has a little bit of very finely chopped white onion, half jalapeno, cilantro and lime and salt. That's it.Nic:No tomatoes?Maria Puertas Calvo:No tomatoes. The tomatoes only add water to the guacamole, they don't add any flavor.Nic:What about then a sun dried tomato? No liquid, just the flavor? Is that an acceptable compromise?Maria Puertas Calvo:Absolutely not. No tomatoes in guacamole. The best way to make it is, you first mash the jalapeno chili with the cilantro and the onion almost to make a paste and then you mix in the avocado and then you finally drizzle it with some lime and salt.Nic:Hang on. Did you say garlic or no garlic?Maria Puertas Calvo:No garlic, onion.Nic:No garlic, I see. So the onion is the substitute for I guess that's a savoriness? I don't know how you classify... What's garlic? Is it Umami? I don't know the flavor profile but no garlic? Wow, I'm making guacamole when I'm at my house.Natalia:Well, you heard it here first guys. Maria's famous guacamole recipe.Nic:I think we'll have to publish this on Twitter as a little Easter eggs for this episode. It'll be Maria's definitive guacamole recipe.Maria Puertas Calvo:Now the secret is out.Nic:Well, Maria, thank you so much for your time. This has been a fantastic chat I think. I have a feeling we're going to want to talk to you again on the podcast. I think we'd love to hear a bit more about your personal story and I think we'd also love to learn more about some of the AI techniques that you talked to us about but thank you so much for your time.Maria Puertas Calvo:Yeah, of course, this was a flasher. I had a great time and I'll come back anytime you want me. Thank you.Natalia:And now let's meet an expert from the Microsoft Security Team to learn more about the diverse backgrounds and experiences of humans creating AI and tech at Microsoft. Today, we're joined by Jeff McDonald, who joined us on a previous episode, unmasking malicious scripts with machine learning to talk to us about anti-malware scan interface or AMC. Thank you for joining us again on the show, Jeff.Geoff McDonald:Yeah. Thank you very much. I really enjoyed being here last time and excited to be here again.Natalia:Great. Well, why don't we start by just giving a quick refresher to our audience? Can you share what your role and day to day function is at Microsoft?Geoff McDonald:I lead a team of machine learning researchers and we build our machine learning defenses for Microsoft defender antivirus product. So we built lightweight machine learning models which go into the antivirus product itself which run on your device with low memory and lower CPU costs for inference. We also deploy a lot of machine learning models into our cloud protection platform where we have clusters of servers in each region around the world. So that when you're scanning a file or behavior on your device, it sends metadata about the encounter up to our cloud protection in real time to the closest cluster to you. And then we do real time running of all of our machine learning models in the cloud to come back with a decision about whether we should stop the behavior or attack on your device.Geoff McDonald:So we're a small team of probably about five of us. We're a mix of threat researchers and machine learning and data science experts. And we work together to design new protection scenarios in order to protect our customers using machine learning.Nic:Jeff, when you go to a security conference, some kind of industry get together, do you describe yourself as a machine learning engineer? What do you use when you're talking to other security professionals in your field? Is machine learning... Is it sort of an established subcategory or is it still sort of too nascent?Geoff McDonald:Yeah. I used to call myself maybe a threat researcher or a security researcher when I would present at conferences and when I would introduce myself. But I'd say nowadays, I'd be more comfortable introducing myself as a data scientist because that's my primary role now. Although I come from a very strong background in the security and security research aspect, I've really migrated to an area of work where really machine learning and data science is my primary tool.Natalia:What's driven that change? What prompted you to go deeper into data science as a security professional?Geoff McDonald:So when I first started at Microsoft, I was a security researcher. So I would do a reverse engineering of the malware itself. I would do [inaudible 00:33:31] deep analysis of the attacks and threat families and prepare defenses for them. So I think learning pretty early on while doing all the research in response to these attacks, it was very clear that the human analysis and defense against all these attacks was really not scalable to the scale that we needed. So it really had to be driven by automation and machine learning, in order to be able to provide a very significant protection level to our customers. So I think that really drove the natural solution where all these human resources, these manual analysis doesn't scale to where we need it to be and where we want our protection level to be.Geoff McDonald:So it really encouraged finding the automation and machine learning solution has. And I have previously had some experience with machine learning. At the time, it was kind of a natural fit where I began a lot of exploration of the machine learning application to protect it against these threats and then pivoted into that as my primary role eventually, as it was quite successful.Natalia:So your unique set of skills, data science and security, is one that's definitely sought after in the security space. But considering the fact that we're still trying to fill just security jobs, it's definitely a challenge. So do you have any recommendations for companies that are looking for your set of skills and can't find a unicorn like yourself that has both? And if were looking for multiple people, how should these teams interact so that they're leveraging both skills to protect companies?Geoff McDonald:When we look to fill new positions on our team, we try to be really careful to try to be as inclusive as possible to a lot of different candidates. So when we're pushing our new data science positions where we're looking for the data science experience, like in the machine learning and data science application, you'll see in our job applications, we don't actually require cybersecurity experience for our job positions. We're really looking for someone who has a really great understanding of the data and good understanding of ML. And being able to have a strong coding background in order to be able to implement these pipelines and machine learning models and try out their experiments and ideas in ways that they can implement and take them end to end to deploying them.Geoff McDonald:So really, for people that were looking to join our team, often, you don't actually necessarily have to have a background in cybersecurity for all of our positions. Sometimes we're looking for really strong data scientists who can pick up the basics of security and apply it in a very effective way. But we would also want our team have different sets of people who are more experienced in the security background to help drive some of the product and feature and industry and security trends for the team as well. Our team currently has quite a mix of backgrounds where there's some threat researchers and there's some pure data scientists who have come from related fields who actually haven't come from a cybersecurity background specifically.Nic:I wonder if we can back it up. If we can go back in time and start with you, your story, how did you first get into security, get interested in security? Did it start in elementary school? Did it start in high school? Did it start in college? Did you go to college? Can we back up and learn about the young Jeff McDonald?Geoff McDonald:I grew up in a small town near Calgary, Alberta, Canada. I guess it started with my family being a software developing family, I would say. Like my dad had his own software company and as a result, we were really lucky to have the opportunity to learn to code from a young age. So, we would see our dad coding, we knew that our dad coded so we're really interested in what he was doing and we wanted to be able to learn and participate.Nic:When was that Jeff? We're talking in 80s, 90s?Geoff McDonald:So that would be when I was probably around 10 years old when I started coding. And that would be I guess, 96 or so.Nic:I'm trying to learn like was that on some cool, old Commodore 64 hardware or were we well and truly in the x86 era at that point?Geoff McDonald:Yeah. I mean, an x86 I do believe. So it's just Visual Basic which is very simple coding language. The classic Visual Basic 6.0, we're really lucky to be able to learn to code at a pretty young age, which is awesome. And although my brother went more into... My older brother was about two years older, a big influence on me coding wise as well. He was really into making, you might say, malware. We both had our own computers, we had often tried to break into each other's computers and do things. My brother created some very creative hacks, you can say. Like, one thing I remember is he burned a floppy disk, which would have an autorun on it and the way that I'd protect my computer is a password protected login.Geoff McDonald:But back in those days, I think it was windows 98 at the time, it really wasn't a secure way of locking your computer where you have to type in your password. You can actually insert a diskette and would run the autorun and you could just terminate the active process. So my brother created this diskette and program, which would automatically be able to bypass my security protocols and my computer, which I thought was pretty funny.Nic:Is he still doing that today? Is he still red teaming you?Geoff McDonald:No. Not red teaming me anywhere, luckily.Natalia:So what point were you like, "Well, all of these things that I've been doing actually apply to something I want to be doing for a career?"Geoff McDonald:Yeah. So although was in a really software development friendly household. My dad was really concerned about the future of software development. He was discouraging us from going into software development as a primary career path at the time. Going into university I was mostly considering between engineering and business. I ended up going into engineering because I really liked the mathematical aspect of my work and it is a mix of coding and math, which is kind of my two strong suites. So I went into electrical engineering program, during my electrical engineering for four years is when I really changed from doing game hacking as my hobby to doing software development for reverse engineering tools. So as my hobby, I would create a reverse engineering tools for others to use in order to reverse engineer applications. So I went to universities in Calgary, Alberta there. And in Alberta, the primary industry of the province is oil and-Nic:Is hockey.Geoff McDonald:Good one. Yeah. So in Alberta, the primary industry in the sector is really oil and gas. There's a lot of oil and gas, pretty much all engineers when they graduate, the vast majority go into the oil and gas industry. So really, that's what I was thinking of that I'd probably be going into after I graduate. But either way, I continued the reverse engineering tool development, I did some security product kind of reverse engineering ideas as well. Approaching graduation, I was trying to figure out what to do with my life. I loved control systems, I loved software development, I loved the mathematical aspects and I want to do grad school. So then I looked at programs in security because my hobby of reverse engineering security, I didn't really take very seriously as a career.Geoff McDonald:I didn't think it could be a career opportunity, especially being in Alberta, Canada where oil and gas is the primary sector, there's not much in the way of security industry work to be seen as far as I could tell at the time in the job postings and job boards. So I ended up going for a master's in control systems continuing electrical engineering work. So basically, it's more like signal processing work where you're doing analyzing signals doing fault detection, basically, mount vibration sensors to rotating machines was my research. And then from the vibration signal, you're trying to figure out if there's a fault inside the motor or the centrifuge or the turbine or whatever it's attached to.Geoff McDonald:And in that field, there was a lot of machine learning in the research area. So that's where I got my first exposure to machine learning and I loved machine learning but that wasn't my primary research focus for my topic. And then approaching graduation, I started looking at jobs and I happen to get really lucky at the time that I graduated because there happened to be a job posting from Symantec in Calgary. And when looking at the requirements for the job postings, it had all of the reverse engineering tools and assembly knowledge and basically everything I was doing as a hobby, had learned through game hacking and developing these reverse engineering tools. It was looking for experience in only debug assembly. I'm like, "Oh, my goodness. I have all those skills. I can't believe there's actually a job out there for me where I could do my hobby as a career." So I got really lucky with the timing of that job posting and so began my career in cybersecurity instead of oil and gas.Nic:So you talked about the adding sensors parts to, I guess, oil and gas related sort of instrumentation. And then there was some machine learning involved in there. Is that accurate? So can you expand upon that a little bit, I'd love to learn what that look like.Geoff McDonald:So basically, the safety of rotating machines is a big problem. There was an oil and gas facility actually in Alberta which has centrifuges which spins the... I'm sure I'm not using the right terminology, but it spins some liquid containing gas to try to separate the compounds from the water, I think. And they had one of these... Actually, the spindle of the centrifuge broke and then it caused an explosion in the building and some serious injuries. So it was really trying to improve the state of the art of the monitoring of the health of a machine from the mounted accelerometers to them.Geoff McDonald:Two of the major approaches were machine learning, where you basically create a whole bunch of handcrafted features based on many different techniques and approaches and then you apply a neural network or [SVN 00:43:25] or something like that to classify how likely it is that the machine is going to have a failure or things like that. Now, I think at the time the machine learning was applied but it wasn't huge in the industry yet because machine learning in application to signals, that was, especially in convolutions, not as mature as it is now. The area I was working on was de-convolutions. A lot of machine learning models involve doing... At least a lot of machine learning models nowadays would approach that problem as a convolutional neural network. The approaches that I was working on next one was called a de-convolution approaches.Geoff McDonald:So I was able to get a lot of very in depth research into convolutions and what the underlying mean. And that has helped a lot with the latest model architectures where a lot of the state of the art machine learning models are based on convolutions.Natalia:So what was that a convolutional neural network? Can you define what that is?Geoff McDonald:So convolution is basically where you're applying a filter across the signal. It could be an image or it could be a one dimensional signal. So in this case, it's a one dimensional signal where you have... Well, at least it's a one dimensional signal if you have a single accelerometer on a single axis for the machine. You think of it like the classic ECG river heartbeat going up and down. It's kind of like that kind of signal you can imagine which is the acceleration signal. And then you basically learn to apply a filter to it in order to maximize something. What filter you apply can be learned in different ways. So in a convolutional neural network, you might be learning the weights of that filter, how that filter gets applied based on back propagation through whatever learning goal you're trying to solve.Geoff McDonald:In a typical CNN model, you might be learning something like 1000 of these filters where you're adjusting the weights of all these filters through back propagation according to... To try to minimize your loss function. I guess in my research area, I was working to maximize, design a filter through de-convolution to maximize the detection of periodic spikes in the vibration signal. Meaning that something like an impact is happening every cycle of the rotor, for example.Nic:Well, so convolution is a synonym for sort of complexity. So de-convolution, is that a oversimplification to say that it's about removing complexity and sort of filtering down into a simpler set, is that accurate?Geoff McDonald:I wouldn't say it's so similar to the English language version of it. It's a specific mathematical operator that we apply to a signal. So it's kind of like you're just filtering a signal. And de-convolution is sort of like de-filtering it. It's my best way to describe it.Nic:Oh, right. Okay, interesting. De-filtering it. Could you take a stab at just giving us your sort of simplest if possible definition of what a neural network is?Geoff McDonald:Okay. A simplest stab of a neural network, okay.Nic:And Jeff, there's very few people have asked that question of but you're one of them.Geoff McDonald:Okay, cool. When you look at the state of the art, you'll actually find that neural networks themselves are not widely used for a lot of the problems. So when it comes to like a neural network itself, the best way I might describe it is that it's basically taking a bunch of different inputs and it's trying to predict something. It could be trying to predict the future stock price of Tesla, for example, if they're trying to predict whether Tesla's going to go up or down or they could be trying to predict it. Especially in our Microsoft defender case, we're trying to predict, "Based on these features, is this malicious or not?" Is our type of application.Geoff McDonald:So it's going to mean taking a whole bunch of inputs like, "Hey, how old is this file in the world? how prevalent is this file in the world? What's its file size? And then what's the file name?" Well, maybe I'll say, "Who's the publisher of this file?" Well, it's going to take a whole bunch of inputs like that and try to create a reasoning... It's going to try to learn a reasoning from those inputs to whether it's malware or not as the final label. We do it through a technique called back propagation because we have imagined a million encounters where we have those input features. So then we use these known outputs and inputs in order to learn a decision logic to best learn how to translate those inputs to whether it's Malware or not.Geoff McDonald:So we do this through a lot of computers or sometimes GPUs as well in order to learn that relationship. And a neural network is able to learn nonlinear relationships and co-occurrences. So for example, it's able to learn a logic like is it more than 10,000 file size? And is the publisher not Microsoft? And the age is less than seven days, then we think it's 70% malicious. So it's able to learn sort of more complex logic like that, where it can create and conditions and create more complex logic depending on how many layers you have in that neural network.Natalia:Do you think there's a future for neural networks? It sounds like right now you see a specific set of use cases like image recognition but for other use cases it's been replaced. Do you think the cases you described right now like image recognition will eventually be replaced by other techniques other than neural networks?Geoff McDonald:I think they'll always play a role or derivatives of them will play a role. And it's not to say that we don't use neural networks at all. Like in our cloud protection platform, you'll find tons of logistic regression single neuron models, you'll find GBM models, you'll find random forest models. And we've got our first deep learning models deployed. Some of our feature sets have a lot of rich information to them and are really applicable to the CNN, the convolutional neural network model architecture and for those, we will have a neural network at the end of the month. So it still definitely plays its specialty role but it's not necessarily what's driving the bulk of protection. And I think you'll probably find the same for most machine learning application scenarios around the industry. That neural network is not key to most problems and that it's not necessarily the right tool for most problems but it does still play a role and it definitely will continue to play a role or derivatives of it.Nic:My brain's melting a bit.Natalia:I want to ask for a definition of almost every other term but I'm trying to hold back a bit.Nic:Yeah, I've been writing down like 50 words that Jeff has mentioned like, "Nope, I haven't heard that one before. Nope, that one's new." I think, Jeff, you've covered such a lot of fascinating stuff. I have a feeling that we may need to come back to you at other points in the future. If we sort of look ahead more in general to your role, your team, the techniques that you're sort of fascinated in? What's coming down the pike? What's in the future for you? Where are you excited? What are you focused on? What are you going to see in the next six, 12 18, 24 months?Geoff McDonald:One of the big problems that we have right now is adversaries. So what malware attackers do is that they build new versions of their malware then they check if it's detected by the biggest antivirus players. And then if it's detected by our AV engines, what they do is they keep building new versions of their malware until it's undetected. And then once it's undetected, they attack or customers with it and then repeat. So this is been the cat and mouse game that we've been in for years, for 10 years at least. Now, what really changed about six years ago is that we put most of our protection into our cloud protection platform. So if they actually want to check again, so like our full protection, and especially our machine learning protection, they have to be internet connected so they can communicate with a real time Cloud Machine Learning protection service.Geoff McDonald:And what this means is if they want to test their malware against our defenses before they attack our customers, it means that they're going to be observable by us. So we can look at our cloud protection logs and we can see, "Hey, it looks like someone is testing out their attack against our cloud before they attack our customers." So it makes them observable by us because they can't do it in a disconnected environment. Originally, when we came out with cloud protection, it seems like the adversaries were still testing in offline environments. Now we've gotten to the point where so many of the advanced adversaries as well as commodity adversaries are actually pre-testing their attacks against our cloud defenses before the attack our customers. And this introduces a whole bunch of adversarial ML and defensive strategies that we're deploying in order to stay ahead of them and learn from their attacks even before the attacker customers.Geoff McDonald:So we have a lot of machine learning and data science where we're really focused on preventing them from being able to effectively test with our cloud as a way to get an advantage when attacking customers. So that's one that we have a lot of work going into right now. A second thing that I really worry about for the future, this is like the really long term future, hopefully it won't be a problem for at least another decade or two or even hopefully longer. But having reinforcement learning, if we have some big breakthroughs, where we're able to use reinforcement learning in order to allow machine learning to learn new attacks by itself and carry out attacks fully automated by itself by rewarding it.Geoff McDonald:Luckily, right now, our machine learning or reinforcement learning state of the art is not anywhere close to the technology that would be needed to be able to teach an AI agent to be able to learn new attacks automatically and carry them out effectively. At least nowhere close to the effectiveness of a human at this point. But if we get to the level of effectiveness where we can teach an AI to come up with and explore new attack techniques and learn brand new attack techniques and carry out the attacks automatically, it could change the computing world forever, I think. We might be almost going back to the point where we have to live on disconnected computers or extremely isolated computers somehow but it would be kind of like a worst case scenario where machine learning has allowed the attackers to get to the point where they can use AI to automate everything and learn new attack techniques, learn new exploits, and et cetera, entirely by itself which would be a humongous problem for defensiveness.Geoff McDonald:And there's a lot of ongoing research in this right now but it's very much on the defensive side where, "Hey, we're going to use reinforcement learning to teach an attacker so that we can learn from defending against it automatically." That hypothesis is great but it's been created with the goal of trying to improve our defenses. But actually, it's also building the underlying methods needed in order to carry out attacks automatically by itself. And I think if we get to that point, it's a really big problem for security. It's going to revolutionize the way computer security works.Nic:Well, hopefully, Jeff, you and your colleagues remain one or two steps ahead in that particular challenge?Geoff McDonald:Yeah, we will.Nic:I hope you share that goal. Jeff, what are you and your team doing to make sure that you stay ahead of your sort of adversarial counterparts that are looking to that future? What gives you hope that the security researchers, the machine learning engineers, the data scientists are, hopefully, multiple steps ahead of adversaries out there?Geoff McDonald:I think our adversary situation is much better than it used to be back in the day. Back in the day, they'd be able to fully test our defenses without us even being able to see it. And now that we've forced them into the game of evading our cloud protection defenses, it allows us to observe them even before they attack our customers. So the defenses we have in place that we've already shipped as well as a lot of what we have planned is really going to be a real game changer into the way that we protect our customers where we can actually protect them even before our customers are attacked. So we're in a much better defensive situation since we're able to observe them before the attack our customers nowadays.Natalia:Thank you, Jeff, for joining us on today's show. As always, it was fantastic chatting with you and like Nick said, definitely need to have you back on the show.Geoff McDonald:Thank you very much. really love being on here.Natalia:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.Nic:And don't forget to tweet us @MSFTsecurity or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe...Natalia:Stay secure.
12/9/2020

Threat Modeling for Adversarial ML

Ep. 7
How ready is your corporate security team to handle AI and ML threats? Many simply don’t have the bandwidth or don’t see it as a priority.That’s where security engineers like Microsoft’sAndrew Marshallstep in. In this episode, hostsNic FillinghamandNatalia Godylaspeak with Andrew about just what his team is doingto teach security professionals and policy makers about the dangers of AI and ML attacks, andwalks through some of the documentation, available for freeonline, that can help guide the response. Plus, why he really, really doesn’t want to talk about Windows Vista.Nic and Natalia thenexplore what it’s like to hunt down threats withSam Schwartz, a program managerwith Microsoft Threat Experts. She came to Microsoft right out of college and didn’t even know what malware was. Now, she’s helping coordinate a team of threat hunters on the cutting edge of attack prevention.In This Episode, You Will Learn:Why datascience and security engineering skills don’t necessarily overlapHow attackers are using ML to change decision makingWhat security teams are doing to protect AI and ML systemsHow threat hunters are tracking down the newest security risksWhy Microsoft Threat Experts are focused on human adversaries, not malwareSome Questions We Ask:What does the ML landscape look like at Microsoft?How are ML attacks evolving?What is ‘data poisoning’?Why do threat hunters need to limit the scope of their work?What skills do you need to be a security program manager?Resources Threat Modeling AI Systems and DependenciesMicrosoft Security BlogTranscript(Fulltranscriptcan be found athttp://aka.ms/SecurityUnlockedEp07)Nic Fillingham:Hello, and welcome to Security Unlocked, a new podcast from Microsoft, where we unlock insights from the latest in news and research from across Microsoft security engineering and operations teams. I'm Nic Fillingham.Natalia Godyla:And I'm Natalia Godylain. In each episode, we'll discuss the latest stories from Microsoft security, deep dive into the newest threat intel research and data science.Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft security. If you enjoy the podcast, have a request for a topic you'd like covered, or have some feedback on how we can make the podcast better...Natalia Godyla:Please contact us at securityunlocked@microsoft.com, or via Microsoft Security on Twitter. We'd love to hear from you.Nic Fillingham:Hello, Natalia, welcome to episode seven of Security Unlocked. How are you?Natalia Godyla:I'm doing well. Refreshed after Thanksgiving break. What about yourself? Did you happen to eat the pork that you were planning? Those bratty pigs?Nic Fillingham:The bratty pigs actually have survived for another day. They live to tell another tale to eat more of my home delivered fresh produce, but we did eat a duck that we farmed on our farm. So that's the second time we've been able to enjoy some meat that we grew on the farm, the little mini farm that we live on, so that was pretty exciting.Natalia Godyla:Yeah. That's been the goal all along, right? To be self-sustaining?Nic Fillingham:To some degree. Yeah. So we achieved that a little bit over Thanksgiving which was cool. How about you, what'd you do over your Thanksgiving break?Natalia Godyla:Well, I made apple bread. Oh, there's a special name for the Apple bread, but I forgot it. Pull-apart Apple bread. And I spent a lot of time on WolframAlpha.Nic Fillingham:You spent a lot of time on WolframAlpha? Did your firewall break and it was the only accessible website? How did you even get to WolframAlpha?Natalia Godyla:WolframAlpha is like Sporcle. It's like if you have time, you get to play with their technology and they've got...Nic Fillingham:Sporcle? Sorry, [inaudible 00:02:00] Sporcle?Natalia Godyla:What? Do you not know Sporcle?Nic Fillingham:I'm really old. You'll have to explain that one to me. Is this a millennial thing?Natalia Godyla:Wow. Okay.Nic Fillingham:Bring me up to speed on Sporcle.Natalia Godyla:Sporcle is like fast, quick trivia games that you play with a group in one person just types in the answers while you're running through it.Nic Fillingham:I thought it was when you take a fork and a spoon and you dip them in glitter. Anyway, so you're on Sporcle, and you're like, "I've completed Sporcle. What's next?"Natalia Godyla:And you go to WolframAlpha. That's the next step?Nic Fillingham:So, what did you pose to WolframAlpha?Natalia Godyla:All right, at what point does a cat's meow become lethal to humans? Good question, right?Nic Fillingham:At what point does a cat's meow become lethal to a human? When it's connected to a flame thrower? When the meow is a series of poison darts? What does that mean?Natalia Godyla:There are a lot of use cases for cats. In this one, it's how high the decibel of their meow is, because that can eventually hurt a human. But it's really about spacing. Where you [crosstalk 00:03:03] put the cat is very critical.Nic Fillingham:The question was how loud can I make a cat's meow, so that it hurts a human?Natalia Godyla:A well-trained army of cats that meow at the exact same time, synchronized cats.Nic Fillingham:Oh, a synchronized army of cats, all directed at a single person. Would their collective uber meow, would that serve as a rudimentary weapon? That was your question?Natalia Godyla:Yes.Nic Fillingham:And? Answer?Natalia Godyla:Theoretically, but it depends on how far away all the cats end up being. I'm now thinking that I should have just like planned to capture the cat's meows in a can or something.Nic Fillingham:Capture them in a can. What does that mean?Natalia Godyla:Your can of whoopass.Nic Fillingham:Who would capture a cats meow in a can? Okay, Professor Farnsworth.Natalia Godyla:You can tell I'm not the expert on these podcasts.Nic Fillingham:So hang on, did you work out how many cats you needed in a single location to produce a loud enough meow to hurt somebody? Do you know the answer to this question?Natalia Godyla:No. No. I was more focused on total and I don't also know the answer to the question.Nic Fillingham:All right, to all the mathematicians out there and audiologists who have dual specialties into the capturing of cat meows into cans, and then the math required to multiply them into a focused beam of uber meow as a rudimentary weapon, please send us an email, security unlocked@microsoft.com. Oh, Oh, segue, segue. We have email, we have email. We got messages from people who've been listening to the show and they send some very nice things, which is great. And they also gave us some topics they would like us to cover on the show, and we're going to cover one of them today.Nic Fillingham:Shout out to Ryan and to Christian and to Tyler who all asked us to continue to thread on adversarial ML and protecting AI systems. We're doing that exactly today on this episode. We have Andrew Marshall joining us, who is going to absolutely continue to thread that Xia started a couple episodes back talking about protecting AI systems in the MDDR report, and then who are we talking to Natalia?Natalia Godyla:Sam Schwartz. So she is a security PM at Microsoft and works directly with the Microsoft Threat Experts Team to deliver managed services to our customers. So she helps to provide threats Intel back to customers and is working on scaling that out, so that more and more customers can benefit from the billions of signals that we have, that we then apply to the data that we get from customers, in order to help identify threats. On to the podcast.Nic Fillingham:Welcome to the Security Unlocked Podcast, Andrew Marshall. Thank you for joining us. Andrew Marshall:Thank you. It's great to be here. Appreciate you having me on today.Natalia Godyla:Yeah, definitely. So why don't we start off by chatting a little bit about your role at Microsoft. Can you let us know what your day to day looks like?Andrew Marshall:Sure. So I'm a Principal Security Program Manager in the Customer Security and Trust Organization at Microsoft. My role is a little bit different from a lot of people who are security engineers. I'm not part of a product group. Instead, I work across the company to solve long-tail security engineering problems that maybe one particular group may not have the authority to lead all up. So I do a variety of different things, like killing off old cryptographic protocols, where we have to bring the entire company together to solve a problem.Andrew Marshall:And lately, I'd say the past two or three years in particular, my focus has been AI and ML. In particular, the security issues that are net new to the space, because it brings an entirely new threat landscape that we have to deal with. And we have to do this as an entire company. So it was another one of those cross-company security engineering challenges that I really enjoy to tackle.Natalia Godyla:And what does the ML landscape look like in Microsoft? So if it's cross-company how many models are you looking at? How many different groups are using ML?Andrew Marshall:It's a really all over the place. And by that, I mean everybody's using it. And it really is pretty much in universal usage across the engineering groups. And while there's been a big focus to everybody, whether it's in Microsoft or elsewhere, everybody's been interested in jumping on this bandwagon. But as the past couple of years, we've started to see that there are specific security issues that are unique to AI and machine learning, that we're only now, as an industry, are starting to see come out of the world of research-driven, proof of concept contrivances, where somebody created a research paper and a vulnerability that they had to make a bunch of leaps to justify. The pivot is occurring now from that into actual weaponized exploitation of these attacks.Andrew Marshall:So what we're trying to solve here from a security perspective is with this worldwide rush to jump on the AI and ML bandwagon, what is the security debt around that? What are the new products and features and detections and mitigations that we need to build as a company to solve these issues for ourselves and for the world? One of those things is really focused on education right now, because we've published a series of documents that we made, we can publish them externally. We've got a machine learning threat taxonomy, which covers the intentional and unintentional threats that are specific to machine learning. We've got some documents that were built on top of that. One of which was called Threat Modeling AI/ML Systems and Dependencies.Andrew Marshall:And this is a foundational piece of security engineering education work that's being used at Microsoft right now. The issue being security engineers, who have been... you can be a great security engineer, with tons of experience. You could have been doing this for 15 years, or more, but it most likely also means you don't have any data science expertise, or familiarity. So security engineers and data scientists are not two skillsets that often overlap. Ann Johnson calls them, "platinum unicorns", because that's just this mythical creature that nobody really seems to see. But the idea here is that we want all of our security engineers across the company to be intimately familiar with these net new security threats, specific to AI and ML.Andrew Marshall:But here's the problem with all of that. This is such a nascent field, still, especially machine learning specific InfoSec, that if you are going to address these problems today, what you need is you need automation. You need new development work to be able to detect a lot of these attacks, because of the way that they occur. They can either be direct attacks against our model, or they can be attacks against the data that is used to create the model. The detections are very primitive, if they exist at all, and the mitigations are very bespoke. So that means if you find a need to mitigate one of these machine learning threats right now, it means you're probably going to have to design that detection or that mitigation specific to your service in order to deal with that issue. That's not a scalable solution for any company.Andrew Marshall:So where we need to be is we need to get the detections and mitigations for these machine learning specific threats, get them to be transparent, on by default, inherited by the nature of using the. Platform where it just works under the hood, and you can take it for granted, like we take for granted all of the compiled in threat mitigations that you get when you build code code in Visual Studio. So for example, Visual Studio, if you build code there, you inherit all of these different compiled in threat mitigations. You don't have to be a security engineer or know anything about this stuff, but you get all of that goodness just by nature of using the platform. It's on by default and you're oblivious to it. And that makes it easy to use. So, that's where we need to get with this threat landscape too. That's just a very exciting, very challenging space to be a part of.Nic Fillingham:Well, I think we're done. Thanks very much, Andrew. No, joking. Wow, so much there. Thank you for that intro. So I think my first question is this conversation we're having is following one that we have with Sharon Xia recently talking about the machine learning insecurity section that was in the recently published Microsoft Digital Defense Report. You're referring to the threat modeling AI systems and dependencies work that's up on the docs page. We'll put a link to that in show notes. When we spoke to Sharon, she really called out upfront, and I think you've just really Nic Fillingham:Emphasize that the sort of awareness... This is a very nascent topic. And especially at the customer level, awareness is very low and there needs to be awareness in this field. So I think what is Microsoft doing... First, maybe what is Microsoft's role in promoting awareness of this new category and what are we doing there?Andrew Marshall:So we have a role on a couple of fronts here, both within the company and more broadly, within industry and with different governments and customers around the world. So our responsibility is to act... Internally, we'll help shaping not only the educational efforts within the company, but also the research and engineering investments that are made in order to address these issues and solve these problems in this space. There's a policy shaping side of that as well, which is working with governments and customers around the world to help them shape meaningful, actionable policy. That policy in any kind of space can be a dumping ground for good intentions. So whenever people are working on some kind of new policy or some kind of new standard, we always want to make sure that everything is as practical and as actionable as it can be with... And has to be really crisp because you can't have ambiguous goals. You have to have exit criteria for all of these things.Andrew Marshall:And the reason I'm elaborating on that is because my team in the company owns the security development lifecycle. And we're very, very careful about new security requirements that get introduced into that so much so to the point that we try not to introduce new security requirements there, unless we've got some kind of automation already ready to roll for people to use. And that way, we can just tell them, "Hey, this is now a mandatory thing that you have to do, but it's really just run this tool and fix these errors. It's not some kind of new manual attestation or big painful exercise to go through." And that's how we can keep adapting the SDL policy. On the responsible AI side and AI and ethics, we've got... This responsible AI standard that we're working on is basically the guiding principles around responsible AI for Microsoft in terms of how we deal with bias and fairness and transparency and reliability and safety issues as they relate to AI, as well as to security. And this is another element of policy that's being shaped within the company.Nic Fillingham:So you mentioned that very few of these guidances have been automated. Obviously, one of the goals is probably, I assume, to get them automated into toolsets and into SDL. So let's... I'm going to put a customer hat on. I'm a customer of Microsoft. How should I feel about the work that Microsoft is doing to secure its own AI and ML systems? So obviously, we're practicing what we preach here and putting these guidances into place. How is success being measured? Or what are the metrics that we're using to, be it manually or automated, to make sure that our own AI and ML systems are protected?Andrew Marshall:We're spinning up significant research and engineering investments across the company specifically to tackle these kinds of problems. Part of that is largely security. And it's part of this broader series of AI and ethics investments that we're making, but the security issues in particular, because we know that we've got customers reporting these kinds of things, and because we know that we've got our very own specific concerns in this space, we're putting together roadmaps to deal with these kinds of issues as specific sets of new product features and threat detections and mitigations in this space.Andrew Marshall:We understand that you can't catch any of these things manually. It takes automation to catch any of this stuff. So that gives us a roadmap of engineering investments that we can prioritize and work directly with engineering groups across the company to go solve that. And the idea here being that when we deliver those solutions, they're not just available to Microsoft services, but they'll be made available to customers of Microsoft as well.Natalia Godyla:So, Andrew, how are we seeing these attacks start to evolve already? So if you could talk us through a couple of examples, like data poisoning, that would be awesome.Andrew Marshall:Oh, I'd love to. So data poisoning is something that we've seen our own customers impacted by because as we point out in our threat modeling guidance, there's a tremendous over-reliance on using public uncurated data feeds to train machine learning models. Here's an example of a real situation that did happen. So a customer was aggregating trading data feeds for a particular futures market. Let's just say it was oil. And they're feeding these training data feeds from different trading institutions, brokerages, or trading websites or whatever. They're taking all this stuff over a secure channel, they're generating a machine learning model from it. And then they're using that ML model to make some really high consequence decisions like is this location a good place to drill for oil or bid on rights by which you can drill for oil? Or do we want to take a long position or a short position in the oil futures market?Andrew Marshall:So they're essentially trusting the decisions that come out of this machine learning model. And what's the nature of futures trading data feeds there's new data every day, so they're constantly incorporating this new data. Talking about the blind reliance on this untrusted data, even though it was over a secure channel, one of the training data providers was compromised not in a way that resulted in the website being shut down, but what happened was their data was contaminated. The data that they were sharing with everybody else. Unless you're actively monitoring for something like this as the provider of that data, there's no way that you're going to know that you're sending out that data to everybody else.Andrew Marshall:So if the providers are unaware of the compromise, then the consumer of the data is also going to be equally as oblivious to the fact. So what happens is over time, that data became trusted high confidence garbage within that trading data model. So then that led to these decisions like drilling for oil in the wrong place or longing the futures market when they should have been shorting it and vice versa. So the point here is without automation to detect that kind of data poisoning attack, you don't know anything went wrong until it blows up in your face.Natalia Godyla:It really gives you perspective because I feel like normally when you're hearing about cyber attacks, you are hearing about data being stolen and then sold or money itself being stolen. But in the case that you just explained, it's really about altering decision-making, it wasn't just direct money stealing.Andrew Marshall:That was an interesting case because we're also thinking, all right, well, was it a targeted attack against the consumer or the people building machine learning models? How did the attacker know that? Were they looking to see what kind of outcomes this would generate? Is this the only place that they were doing that? Of course the data provider doesn't know. That's one of the more interesting, more insidious attacks that we've seen because we've got to create new types of tools and protections in order to even detect that kind of stuff in the first place. So you're looking for... As your machine learning models are being built, you're looking at taking on new data and looking for statistically significant drift in certain parts of the data that deviate from what looks normal and the rest of your data, and we're looking at ways of solving that. And that's an interesting space. So, yeah.Natalia Godyla:So you noted that one of the potential reasons that the threat actor was playing around with that ML system for that customer example was because they were also just trying to figure out what they could do. So if it's so nascent than threat actors, are they in a similar place as us? Are they ahead of us?Andrew Marshall:Well, we've already had that pivot from contrived research exploits where people are just trying to show off. We've already had that pivot into actual exploitation. So I don't know how to go back and attribute the level of attacker sophistication there. I don't think it was actually... In the attack that I mentioned here, the oil company scenario, that was compromised through traditional security vulnerabilities of that data provider. And I think the jury is still out on the final attribution of all of that, as well as the level of attacker sophistication or if... What would be even more interesting than all of that is really what other customers of that data provider were compromised in this and building machine learning models that were contaminated by that data. Think about hedge funds, who else was compromised by this and never even found out? Or who else had a model blow up in their face? That'd be a very interesting thing to see.Nic Fillingham:The question I wanted to wrap up with, Andrew, is make me feel like we're on a good path here. Like, can we end on a high note? We talked about a lot of very serious scenarios and the consequences for adversarial ML. And obviously it's very important and very nascent, but should I feel like the good guys are winning? Should I feel like we've got good people on this? We're making great progress? That we should feel confident in AI and ML systems in-Andrew Marshall:Yeah, absolutely.Nic Fillingham:The second half of 2020?Andrew Marshall:That's our entire focus with the AI and ethics and engineering and research group. We are bringing the entire weight of Microsoft to bear around these issues from a research, engineering, and policy perspective. And we want to solve all these issues so that you do have trustworthy interactions with all of our products. And that's an essential thing that we realized collectively as a company that has to happen where people won't use these kinds of products. If it doesn't generate an outcome that you can trust is going to be accurate and free of bias and something that you can rely on, then people just won't use those things. So we've got the AI and security centers of gravity working across the company with research and policy experts to tackle these issues. It's a fascinating time to be a part of this. I think that... I just had my 20 year anniversary last month, and I think this is about the most fun I've had period in the past 20 years working on this stuff now.Nic Fillingham:It wasn't the launch of Windows Vista?Andrew Marshall:I have so many horror stories from that. We really don't want to air those.Nic Fillingham:Well, that's awesome. Gosh, what was I... I had this great question I was going to ask you and then the Vista joke popped in and now my brain is mulched.Natalia Godyla:I love how that took priority.Nic Fillingham:Like the most intelligent question I'm going to ask the entire interview and it's like just a joke bonk.Andrew Marshall:I have some very, very funny stories from Vista, but none that are appropriate for here.Nic Fillingham:Well, we may have to bring you on another time, Andrew, and try and sanitize some of those stories because the statute of limitations has surely run out on having to revere every single release of Windows. Surely we can make fun of Vista soon, right?Andrew Marshall:I'm sure we can.Nic Fillingham:So, Andrew, final question, where do you recommend folks go to learn more about this space and keep up to speed with any of the advancements, new taxonomy, new guidelines that come out?Andrew Marshall:I would definitely keep tabs on the Microsoft Security blog. That's going to be the place Andrew Marshall:It's where we drop all of the new publications related to anything in this space and connect you with security content more broadly, not just AI and ML specific, but yeah, the Microsoft Secure Blog, that's where you want to be.Nic Fillingham:Great. Thanks Andrew Mills for your time. We'll also put a link up to the guidelines on the doc's page.Andrew Marshall:All right. Thank you very much for having me today. It's been great.Natalia Godyla:And now let's meet an expert in the Microsoft Security Team to learn more about the diverse backgrounds and experiences of the humans creating AI and tech at Microsoft.Natalia Godyla:Hello everyone. We have Sam Schwartz on the podcast today. Welcome Sam.Sam Schwartz:Hi, thanks for having me.Natalia Godyla:It's great to have you here. So you are a security PM at Microsoft. Is that correct?Sam Schwartz:That is correct.Natalia Godyla:Awesome. Well, can you tell us what that means? What does that role look like? What is your day-to-day function?Sam Schwartz:Yeah, so I support currently a product called the Microsoft Threat Experts and what I am in charge of is ensuring that the incredible security analysts that we have, that are out saving the world every day, have the correct tools and processes and procedures and connections to be the best that they can be.Natalia Godyla:So what are some of those processes look like? Can you give a couple examples of how you're helping to shape their day to day?Sam Schwartz:Yeah. So what Microsoft Threat Experts does, is it as a managed threat hunting service provided by Microsoft defender ATP product and hopefully in the next six months to a year, we'll also support all of Microsoft 365 defender products, which is Office, Azure, MCAS and MDTP.Sam Schwartz:And what they do is our hunters will go through our customer data in a compliant safe way, and they will find bad guys, human adversaries inside of the customer telemetry. And then they notify our customers via a service called the targeted attack notification service.Sam Schwartz:So we'll send an alert to our customers and say, "Hey, you have that adversary in your network. Please go do these following things. Also, this is the story about what happened, how they got there and how you can fix it."Sam Schwartz:So what I do is I try to make their lives easier by initially providing them with the best amount of data that they can have when they pick up an incident.Sam Schwartz:So when they pick up an incident, how do they have an experience where they can see all of the data that they need to see, instead of just seeing one machine that could have potentially been affected, how do they see multiple machines that have been affected inside of a single organization? So they have an easier time putting together the kill chain of this attack.Sam Schwartz:So getting the data and then also having a place to visualize the data and easily make a decision as to whether or not they want to tell a customer about it, does it fit the criteria? Does it not? Is this worth our time? Is this not worth our time? And then also providing them with a path to, with that data quickly create an alert to our customers so that they know what they're doing.Sam Schwartz:So rather than our hunters, having to sit and write a five paragraph essay about what happened and how it happened, have the ability to take the data that we already have, create words in a way that are intuitive for our customers, and then send it super quickly within an hour to two hours of us finding that behavior.Sam Schwartz:So all of those little tools and tracking, and metrics and easier, like creating from data, creating words, sending it to the customers, all of that is what I plan from a higher level to make the hunters be able to do that.Nic Fillingham:And to better understand the scale of what's happening here, like with a typical customer, what is the volume of signal or alerts or, I'm not sure what the correct taxonomy is, but what's the volume of stuff that's being monitored from the customer and then is being synthesized down to a bunch of alerts that then go and get investigated by a hunter?Sam Schwartz:So I don't have a per customer basis, but we have about, I think it's either 450 customers currently enrolled in our program. And unfortunately, we can't take everyone that would like to join us. Our goal is that we will eventually be able to do that, but we don't have enough people and we're still building our tooling to allow us to scale.Sam Schwartz:So with our 450 customers, we have every month, about 200,000 incidents that get created and we then bring that down. So some of those incidents don't get investigated because they don't meet our bar. Some of those incidents get investigated, but aren't interesting enough to actually have an alert created. And some of them even, although the alert is created, it's not actually interesting enough to send, or we've already sent something similar and it's not worth it.Sam Schwartz:So from those 200,000, we send about 200 to like 250 alerts a month about, but it also depends on the landscape. Like it depends on what's going on that-Nic Fillingham:And if I go even higher up the funnel, so before the 200,000 is it, what's the right taxonomy, is it an alert?Sam Schwartz:Incidents. We call them incidents.Nic Fillingham:... What's above an incident. What is, because I assume it's just tons and tons and tons of network logs and smaller signals that end up getting coalesced into an incident. Is that correct?Sam Schwartz:Yeah. So what we do is we call them traps. So what they are is they're queries that run over data that finds something super interesting. And you can think about these as similar to alerts that customers get, but much, much, much lower fidelity.Sam Schwartz:So for us, for our products, a trap, if it fires a hundred times and of that a hundred times, 99 of them are false positives, 99% of them are not super helpful for the customer, we're not going to send that to the customer. That's bothering them 99 times that they don't need to be bothered. But for our service, our whole thing is that we are finding that 1% that our customer doesn't know about.Sam Schwartz:So we have extremely low fidelity traps. Some of them are high fidelity that it can run a thousand times and only one time is it important? We want to see every a thousand times because that one time is worth it. So we have traps, I think we have about 500 of them. Some of them return thousands of results a day. Some of them won't return results for months.Sam Schwartz:And if that gets a hit, then those are the things that get bubbled up into our incidents. We cluster all of those trap results into the incidents, so that's ensuring that our hunters get all the information that they need when they log on, so the signals are massive. There's a massive amount. I don't even have a number.Natalia Godyla:I have literally so many questions.Sam Schwartz:Oh my God, happy to help.Natalia Godyla:So you said earlier, there's a bar for what the Microsoft Threat Experts will focus on. So what is in scope for them? What meets the criteria?Sam Schwartz:We are focusing on human adversaries. So we're not focusing much on commodity malware, as much as we are focusing on a hands-on keyboard attacker. So there are some traps that are, some of them are commodity malware, but paired with other traps so paired with other signals, that could be a hands-on keyboard person. And those are things we look at, but then maybe some of the traps on their own don't meet a bar for us to go look at.Nic Fillingham:Is that because commodity malware is basically covered by other products, other services?Sam Schwartz:(Affirmative). It's covered by our defender ATP product in general. So our hunters wouldn't be adding. Our whole point is that we have hunters who are adding context and value to the already incredible ATP product. And since ATP is already alerting and covering that, we'd rather find the things that aren't being covered.Nic Fillingham:So Sam, let's go back in time a little bit, so tell us about how you found yourself in the security space and maybe it's a separate story maybe it's the same story and how you got to Microsoft. We'd love to learn your journey, please.Sam Schwartz:It is the same story. Growing up, I loved chemistry.Nic Fillingham:That's too far back.Sam Schwartz:I know.Nic Fillingham:Oh, sorry. Let's start there.Sam Schwartz:I loved Chemistry. I loved like molecules and building things and figuring out how that all works. So when I went to college, I was like, I want to study chemical engineering. So I through my education became a chemical engineer, but I found that I really liked coding. We had to take a fundamentals class at the beginning and I really enjoyed the immediate feedback that you got from coding. Like you did something wrong, it tells you immediately that you messed up.Sam Schwartz:And also when you mess up and you're super frustrated and you're like, why didn't this work? Like I did it right. You didn't do it right, it messed up for a reason. And I really liked that. And I thought it was super interesting. And I found myself like gravitating towards jobs that involved coding.Sam Schwartz:So I worked for Girls Who Code for a summer. I worked for a Dow Chemical Company, but in their robotics division. So I was still like chemical engineering, but I got to do robots. And then when I graduated, I was like, I think I want to work in computer science. I don't like this chemical engineering. It was quite boring, even though they said it would get more fun, it never did. We ended up watching water boil for a lot of my senior year of college. And I was like, I want to join a tech company.Sam Schwartz:And I looked at Microsoft and they're one of the only companies that provide a program management job for college hires. So a lot of PM positions because there's a lot of high level thinking, coordinating and collaboration. A lot of PM positions are one of those, like you need experience, but in order to get experience, you have to do the job and it's like one of those weird circles and Microsoft allows college hires to do it.Sam Schwartz:So when I interviewed, I was like, I want to be a PM. It sounds fun to get to hang out with people. And I ended up getting the job, which is awesome.Nic Fillingham:Is that all you said in the interview? Just, it sounds fun to get to hang out with people?Sam Schwartz:Yes. I was like, this is it, this is my thing. What they did is they, in my interviews, they asked me a bunch of, they asked me a very easy coding question, I was so happy. I was so nervous that I wasn't going to get a pass that one, but that was easy. And then they asked me a design question. They asked me, "Pick your favorite technology." And me, I'm sad to say it. I feel like I'm better now looking back on myself, but I'm really not good with technology in general.Sam Schwartz:So they're like pick your favorite technology. And I was like, I'm going to pick a chemical engineering plant because I didn't know anything. So I picked an automation plant as my favorite technology. And they asked me a lot of questions around like, who are the customers? What would you do to change this to affect your customers? Who gets changed? How would you make it better?Sam Schwartz:Then I was talking specifically about a bottling plant, just because that's easy to understand. And I left that interview and my interviewer was like, I didn't know, he said, "I didn't know anything that you were talking about, but everything you said made perfect sense because it's about how can you take inputs, do something fun and then have an output that affects someone. And that's everything that we do here. Even though it's a bit off you skated and you have a bunch of data and bad guys and hunters hunting through things, it's taking an input and creating something great from it."Sam Schwartz:And that's what we learned in our chemical engineering world. And I ended up getting this job and I walked on my first day and my team and they're like, "You're on a Threat Intelligence Team." I was like, "What does that mean?" And-Nic Fillingham:Oh, hang on. So did you not know what PM role you were actually going to get?Sam Schwartz:No. They told me that I was slated for the Windows. I was going to be on a Windows team. So in my head like that entire summer, I was telling people I was going to work on the start button just because like, that's what... I was like, "If I'm going to get stuck anywhere, I'm going to have to do the start button. Like that's where my-"Nic Fillingham:That's all there is. Windows is just now [crosstalk 00:35:45]Sam Schwartz:I was like that what... I was guaranteed, I'm going to get the star button or like Paint. Actually, I probably would have enjoyed Paint a lot, but the start button and I came and they were like, "You're on Threat Intelligence Team." And I was like, "Oh, fun." Sam Schwartz:And it was incredible. It was an incredible start of something that I had no idea what anyone was talking about, when they were first trying to explain it to me in layman's terms, they're like, oh, well, there's malware and we have to decide how it gets made and how we stop it. And I was like, what's malware? I was like, you need to really dumb it down, I have no idea what we're talking about. And initially when I started on this threat intelligence team, there were only five of us. So I was a PM and they had been really wanting a PM, and apparently before they met me were happy to get a PM, but weren't so happy it was a college hire. They're like-Nic Fillingham:Who had never heard of malware.Sam Schwartz:We need structure.Nic Fillingham:And thought Windows was just a giant anthropomorphic start menu button.Sam Schwartz:They're like, we need structure, we need a person to help us. And I was like, hi, nice to meet you all. And so we had two engineers who were building tools for our two analysts and it was, we called ourself a little startup inside of security research inside of the security and compliance team, because we were figuring it out. We were like, threat intelligence is a big market, how do we provide this notion of actionable threat intelligence? So rather than having static indicators of compromise, how do we actually provide a full story and tell customers to configure, to harden their machines and tell a story around the acts that you take to initiate all of these. These configurations are going to help you more than just blocking IOCs that are months old. So figuring out how to best give our analyst tools, our TI analysts, and then allow us to better Microsoft products as a whole.Sam Schwartz:So based on the information that our analysts have, how do we spread that message across the teams in Microsoft and make our products better? So we were figuring it out and I shadowed a lot of analysts and I read a lot of books and watched a lot of talks. I would watch talks and write just a bunch of questions. Then finally, as you're around all these incredibly intelligent security people, you start to pick it up, and after about a year or so I would send meetings and I would listen to myself speak and I was like, did I say that? Was that me that one, understood the question that was asked of me and then also was able to give an educated answer? It was very shocking and quite fun. And I still feel that way sometimes, but I guess that's my journey into security.Natalia Godyla:Do you have any other suggestions for somebody who is in their last years of college or just getting out of college and they're listening to this and saying, heck yes, I want to do what Sam's doing. Any other applicable skills or tricks for getting up to speed on the job?Sam Schwartz:I think a lot of the PM job is the ability to work with people and the ability to communicate and understand what people need and be able to communicate that in a way that maybe they can't communicate. See people's problems and be able to fix them. But I think a lot of the PM skills you can get by working collaboratively in groups, and that you can do that in jobs, you can do that in classes. There's ample opportunity to work with different people, volunteering, mentoring, working with people and being able to communicate effectively and connect to people and understand, be empathetic, understand their issues and try to help is something that everyone can do and I think everyone can be an effective PM. On the security side, I think reading and listening. Even the fact that, the hypothetical was someone listening to this podcast that are already light years ahead of I was when I started, but just listening, keeping up to date, reading what's going on in the news, understanding the threats, scouring Twitter for all the goodness going on.Nic Fillingham:[inaudible 00:03:59].Sam Schwartz:That's the way to stay on top.Nic Fillingham:Tell us about your role and how you interface with data scientists that are building machine learning models and AI systems. Are you a consumer of those models and systems? Are you contributing to them? Are you helping design them? How do you fit into that picture?Sam Schwartz:So a little bit of all of the things that you mentioned, being a part of our MTE service, we have so many parts that would love some data science, ML, AI help, and we are both consumers and contributors to that. So we have data scientists who are creating those traps that I was talking about earlier for us, who are creating the indicators of malicious anomalous behavior that our hunters then key off of. Our hunters also grade these traps. And then we can provide that back to the data scientists to make their algorithms better. So we provide that grading feedback back to them to have them then make their traps better. And our hope is that eventually their traps, so these low fidelity signals, become so good and so high fidelity that we actually don't even need them in our service, we can just put them directly in the product. So we work, we start from the incubation, we provide feedback, and then we hopefully see our anomaly detection traps grow and become product detections, which is an awesome life cycle to be a part of.Nic Fillingham:I want to change topics then, but this one's going to need a little bit of context setting because you are famous inside of Microsoft for anyone that has completed one of our internal compliance trainings. I don't even know how to describe this to people that haven't experienced it. Natalia, we've both done it. So there's this thing at Microsoft called Standards of Business Conduct, it's like a internal employee compliance. This is how you should behave, this is how you should function as a responsible employee and member of the Microsoft family, but then also how we work with customers and everything. And it's been going on for a few years. Sam, you had a cameo, you were the only non-professional actor in the recent series, that's correct?Sam Schwartz:I was, I was, I'm famous, I will be signing headshots when we're all back in the office.Nic Fillingham:So tell us about how did this happen?Sam Schwartz:So I, as anyone who has seen the Standards of Business Conduct videos, I wouldn't call them a training, I would call them a production.Nic Fillingham:An experience. Or production.Sam Schwartz:An experience, yeah. An experience.Nic Fillingham:They're like a soap opera. It's almost like Days of Our Lives. They really stir the emotion and we get attached to these characters and they go on wild journeys in a very short space of time.Natalia Godyla:I was just watching an episode and I literally got stressed.Sam Schwartz:Yeah, you're so invested in these characters and their stories and you're rooting for them to do the right thing. And you're like, come on, just be compliant. And in my first week on the job I was telling, I watched this training as everyone who starts Microsoft has to do and I was telling my team that I was obsessed with the main character who has his own trial and tribulations throughout the entire series. And I just thought it was fun and I was like, how do I get on it? That was my thing when I first joined, how do I get on Standards of Business Conduct? And every year, Microsoft is super passionate about giving, giving back, donating money, and every October we have this thing called the Give Campaign where every employee is encouraged to give back to their community.Sam Schwartz:And one of the ways that they do is they have an auction. So some of the auction things are, you get lunch or golf with Satya, or you get assigned, I don't know, computer or X-Box from Phil Spencer or whatever it is. I made those up.Nic Fillingham:You get to be the Windows start button for a day.Sam Schwartz:You get to be the Windows start button for a day. And one of those is a cameo in Standards of Business Conduct. And you can donate a certain amount of money and there's a bid going, where the person who donates the most money is at the leaderboard and then if you donate more money, you got on top. So a silent auction before giving back and donating. And I saw that last year on the gift campaign, but I didn't think much of it. It had a high price tag and I didn't want to deal with it. And then a couple of months later, I had just gotten back from vacation and my skip level was like, hey, I missed you a lot, let's get lunch. And I was like, okay, great, I love that.Sam Schwartz:And he was like, I want to go somewhere fun, I want to go to building 35, which is the executive nice cafeteria building at Microsoft, which is not near our office. And I was like, okay, weird, he wants to go to another building for lunch, but we can go do that. So I went with him and it was five to 10 minutes into our lunch and these people come up to our table and they're like, can we sit with you? And I'm looking around and there are tons of tables, I'm like, what are these people encroaching on my lunch for? I just want to have lunch and chat and these people want to come sit at my table, but of course, we're going to let them sit at our table. And I look over at the guy who's sitting next to me and it's the main character from Standards of Business Conduct. It is the actor, it is-Nic Fillingham:It's Nelson.Sam Schwartz:It's Nelson. And I fan girled over him for a year and a half now, I've seen all his work, I'm a huge fan.Nic Fillingham:Please tell me it was a Beatles on their first tour to America moment. Please tell me there was screaming, there was fainting.Sam Schwartz:I blacked out.Nic Fillingham:That's the picture in my head.Sam Schwartz:I don't remember. I don't remember what happened because I actually blacked out. And there's a video of this and you can see my body language, when I realized you can see me grab the arms of the chair and my whole body tenses up, and I'm looking around frantically, like, what's happening. And the woman who was sitting next to my skip-level, she actually created Standards of Business Conduct and she's in a lot of the videos, her name is Rochelle. And she's like, your team has pulled together their money and bought you a cameo in our next Standards of Business Conduct. And I turned around and my entire team is on the balcony of the cafeteria filming me and it was very cute and very emotional. And I got to see Nelson and then I got to be in Standards of Business Conduct, which is awesome. And it was a super fun experience.Nic Fillingham:So in the Microsoft cinematic universe, what is your relationship to Nelson? Are you a colleague?Sam Schwartz:We're all on the same team.Nic Fillingham:So you're a colleague.Sam Schwartz:We are on the same team.Nic Fillingham:So forever, you're a colleague of Nelson's.Sam Schwartz:Yeah, I am. And he knows who I am and that makes me sleep well at night.Natalia Godyla:Thank you, Sam, for joining us on the show today, it was great to chat with you.Sam Schwartz:Thank you so much for having me. I've had such a fun time.Natalia Godyla:Well, we had a great time unlocking insights into security, from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us at MSFTSecurity, or email us at SecurityUnlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe.Natalia Godyla:Stay secure.
11/25/2020

The Mechanics of Digital Crime

Ep. 6
Ever wonder why it's so difficult to really secure a network, systems or data? Cyber criminals arestepping up their game, even as security gets stronger and stronger, andthey’reusing all sorts of new techniques to breakthrough enterprise walls.In this episode, hostsNic FillinghamandNatalia GodylaspeakwithDonal Keating, Director of Innovation and Research for the Microsoft Digital Crimes Unit,aboutone of thekey findings in the latestMicrosoft Digital Defense Report:howattackers are adapting and becoming more sophisticated.Plus how social engineering is revealing the true weakest linkin any security plan -- and it’s something you might not expect.Then they dive into what it’s like to hunt threats withMichelle Lam,who brings fresh eyes to every security problem she faces atMicrosoft. She explains whynot spending time in a SOC early in her careerhelps her spot potential attacks others might miss, andwhy she’s so passionate about helping serve under-represented communities and inspiring the next generation of security professionals.In This Episode, You Will Learn:How cyber attackers are using the cloudWhy humans are the weakest link in every security systemThe new steps cyber criminals are taking to get people to trust themHow threat hunters look for malicious activityHow networking helps young security professionalsSome Questions We Ask:What new threat trends are emerging?How should security professionalsprepare for newthreats?What is a homoglyph?Why is threat hunting a uniquely human-based activity?Resources Microsoft Digital Defense Report, September 2020Microsoft Security BlogTranscript(Fulltranscriptcan be found athttp://aka.ms/SecurityUnlockedEp06)Nic Fillingham:Hello and welcome to Security Unlocked, a new podcast from Microsoft, where we unlock insights from the latest in news and research from across Microsoft security, engineering and operations teams. I'm Nic Fillingham.Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft security, deep dive into the newest threat intel, research and data science.Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft security. If you enjoy the podcast, have a request for a topic you'd like covered, or have some feedback on how we can make the podcast better-Natalia Godyla:Please contact us at securityunlocked@microsoft.com or via Microsoft Security on Twitter. We'd love to hear from you. Hi Nick, welcome to another episode. How's it going?Nic Fillingham:Hi, Natalia, I'm a little angry, actually. I'm a little cranky. I don't know if I've said on the podcast before, I live on sort of a small farm, about 30 minutes East of Seattle. And we've got some farm mammals, we've got piglets, recently, they were born in the spring. And this morning the piglets found our delivery of fresh fruit and vegetables from CSA and they ate them all. They ate $75 worth of beautiful organic fruit and veggies, that was meant to last us for the next month. So I'm having pork for Thanksgiving.Natalia Godyla:Those are the brattiest pigs.Nic Fillingham:Yeah well we initially... Their names when they were born, they were super sweet and we called them June and July, my daughters called them that, but we've renamed them to Beavis and Butt-Head because they are stupid jerks.Natalia Godyla:Wow, that's harsh.Nic Fillingham:You think they listen to the podcast? I have given both of them iPhones. Apart from that I'm good, how are you Natalia?Natalia Godyla:Wow, I mean, I can't compete with that story. I'm definitely not at war with one of my piglets.Nic Fillingham:You're in Boston, Massachusetts, I think... You're not downtown, you're in more of the leafy green, sort of, oldie [inaudible 00:02:04] part, aren't you?Natalia Godyla:I am, I'm near Cambridge dealing with equally bratty, but amusing animals. While I don't have the farm set up you have, I have the Somerville turkey.Nic Fillingham:The Somerville turkey? Is that a ghost of a Turkey?Natalia Godyla:Right, it sounds like the headline to a scary movie.Nic Fillingham:Yeah, it's like a turkey shaped poltergeist, what is that?Natalia Godyla:It's just the turkey that causes mayhem in our little neck of the city.Nic Fillingham:Is the turkey's name Somerville or is that the neighborhood?Natalia Godyla:Oh, that's the neighborhood.Nic Fillingham:Does the Turkey have a name?Natalia Godyla:I don't know if it deserves a name.Nic Fillingham:And what does it do, how does it cause mayhem? Is it tipping over trash cans and spray painting swear words on sides of people's houses?Natalia Godyla:I think you might be mixing up a hoodlum with a turkey. No, it blocks traffic and is a great source of distraction for everyone doing remote work in Boston right now.Nic Fillingham:I mean, because you live so close to the storied Cambridge University, I can only assume that a turkey is a much more sophisticated, intelligent turkey. And when it's blocking traffic, it's pulling out traffic cones, it's setting up fake road work, à la Ghostbusters Two.Natalia Godyla:Yeah, this was a very unexpected turn, but I'm impressed at the short number of leaps until we got to Ghostbusters.Nic Fillingham:Hey man, I can get to Ghostbusters in two leaps. Doesn't matter what the topic is. And speaking of turkeys and Thanksgiving, these two turkeys, that's you and me and Natalia, we are very thankful for our guests that joined us on episode six of Security Unlocked. First up, we continue our exploration of some of the topics in the Microsoft Digital Defense Report, the MDDR. Donal Keating is joining us to talk about the increase in sophistication in cyber attacks, and so what does that mean to have seen an increase in sophistication in cyber attacks over the last sort of 12 to 18 months? And some of the sort of high level observations that are in the report, that's a great conversation.Natalia Godyla:And we have Michelle Lim on the show today, threat Hunter at Microsoft. She'll be sharing her path to security and how industry organizations and mentorship have helped her identify new skills and interests within this security space. It's really great to hear how she's leaned on the community to help drive her career and her passions for the cybersecurity realm.Nic Fillingham:And happy Thanksgiving to everyone celebrating in North America. Everyone else happy late November, early December to you. We hope you enjoy the podcast.Nic Fillingham:Welcome to the podcast, Donal Keating.Donal Keating:Hi.Nic Fillingham:Thanks for your time. So Donal we like to start the podcast by asking our guests to give sort of a brief introduction to themselves. What's your job at Microsoft, but sort of what does that look like day to day?Donal Keating:So my role is I'm director of innovation and research for the digital crimes unit, and I generally accepted I have the best job in Microsoft. But what it really means is I sit between a group of people who have regular investigative and analytic jobs and the lawyers who take the cases that we build up and what I consider the data hacking. So we have access to lots of data, lots of crime mechanics, and it's my job, really, to figure out techniques to unveil the criminality and see if we can assist an attribution or mitigation against a particular crime. I'm just sort of the new guy on the block when it comes to new types of crime or new patterns in cyber crime.Nic Fillingham:Are you the Oracle, if I can use a DC universe analogy, or do you prefer a different... What's the superhero role that best fits what you do?Donal Keating:Glue. I'm just incredibly inquisitive glue. And I know very little and it's great having... I feel like a three-year-old going around Microsoft asking, "What does that do?" You need to be inquisitive in this [inaudible 00:00:06:05].Nic Fillingham:That's sort of what Natalia and I are doing on the podcast. That sounds awesome. So Donal, thank you for joining us. In the conversation today, we want to talk about, really one of the biggest headlines coming out of the recently released Microsoft Digital Defense Report for 2020. So it's a report that came out in September. Tom Burt, who leads, I think the organization you're a part of, customer security and trust. He authored the blog post announcing the report and sort of the big takeaway... The big headline there was that, in this last period, cyber threat sophistication has increased and we've never seen it sort of this sophisticated. And so we've invited you onto the podcast today to really help us unpack this idea of cyber threat sophistication and the fact that it is increasing. So if I could start with sort of a pretty big question, cyber threat sophistication is increasing. What does that mean? How do we think about that? How do we measure that? What does it mean for folks out there to know that cyber threats are increasing in sophistication?Donal Keating:Yeah, that's a good question and the way I would... The reason, first of all, the sophistication of this cyber crime is increasing, is largely that the sophistication of the defense has increased significantly. So as more workloads run on cloud environments, operating systems become more secure. People would become more security conscious, there is just more technology in the production area. Criminals by their nature need to adapt to that challenge, so in one area and what I would call traditional hacking where people are trying to gain remote access to a device. They have pivoted away from trying to find zero day exploits and they've actually pivoted to some human engineering. Now, the human engineering may be to get the malicious workloads to run on machines to unlock them and allow malware to be installed.Donal Keating:So that's one area that they need to have got more sophisticated just to get around the defenses. But the second area of where we see sophistication, is cyber crime is now a business. And as a business, there was specialization in that business. So you have very specialized people who will develop malware, ransomware, their specialization and the distribution of that. People who have droppers, people who have networks, botnets, where they will use those botnets to do other things such as proxy attacks, enterprises, proxy attacks on other types of resources.Donal Keating:Even within that, we see a level of automation that we have not seen in the past. So what we would call machine on machine activity is certainly evidence of some of the attacks that we see. But even in the final stages of a cyber crime attack, where it comes to either the ransomware, the exfiltration of data, or just the pure stealing of money out accounts as a result of phishing. The way that money is being muled has also increased. Now, not at the same rate as the sophistication we see in the phishing lures or in the methods of getting people's credentials, because the old saying goes, people used to hack into a computer system, now they log in. A lot of what cyber criminals are doing initially on the attack, is getting some set of credentials to get onto the environment, and then do what they do best, which is do cognizance work across the organization to see more people get more credentials and basically map out the network.Natalia Godyla:With that, can you talk us through a couple examples of how these threats have changed or what new emerging trends are coming out?Donal Keating:Let me give you an example, so banks obviously need to have a significant amount of protection for people logging in, so remote banking. So there's normally a control that says from a given IP address, there can only be, for a given user, there can only be a certain number of login attempts. Now, if you're like me, that's almost guaranteed to be five login attempts because I can never remember what my password is, but I know it's some combination of something. So it is not unusual for normal behavior to be one IP address, one username, three, four, five login attempts. Therefore, any protections that the bank put in place to make sure that the people who are hacking, it needs to meet that criteria, you don't want to disable the customer. And those controls very often called shape controls, will limit the amount of traffic coming into the bank from any one IP address.Donal Keating:So I have seen a case doing what's called credential stuffing. So that's a single IP address with a single username and then multiple attempts to log in. So the attack that we saw, the bank had that control, how many attempts it had been set up at over 20 attempts per hour were allowed. And the bank realized they were having this credential stuffing attack. So what they did is they reduced the number of login attempts that were allowed. And within about an hour, this particular attack dropped down to 14 attacks per hour. Now this was not one IP address, there were 400 IP addresses per hour, probing the banking system. And as the banking system can change their controls, this network of machines adjust to their controls. They also need to do one other thing, the bank had controls as to where those IP addresses had to be located.Donal Keating:The criminals had organized a botnet to deliver the traffic via proxies only in the region where they would be accepted. So they had done two things, they had modified the rate at which they were probing the username, password combination, and they were coming from the location that they were expected to come from. In cyber crime, that's becoming quite a common pattern, that you're not getting the IP addresses from halfway around the world, the login attempts are coming from the area that you expect them to come from. It starts to become quite difficult for defenders to defend against. Now, more barriers will be put up and the cyber criminals will figure a way to get around that, but the improvement in protections and the more security that is applied, requires these cyber criminals to become more inventive in the way they do their thing.Nic Fillingham:So is that rapid agility, that ability to respond? Is that impart the sophistication increase that we're seeing the fact that, to use your example there, that those attackers were able to ascertain that the number of permitted tries per hour was reduced from 20 to 15, and the ability for them to identify that and then adjust their attack. That's in some way, what you're seeing in sophistication increase, whereas in the past, either that wouldn't have happened or it might've taken them weeks or even months to make that change?Donal Keating:Well, two things, one is they are now using cloud Donal Keating:That would resource us to do this. So the attack is not coming from a PC somewhere, this is a battery of VMs set up to behave in a particular way. Their ability to deploy VMs at scale, give them instructions at scale to do these things is a thing that first of all, it just wasn't available previously. But the fact that they are now using the sophistication of technology that large enterprises use to commit crime is indication to me of increasing sophistication. For instance, there are many automated systems to take down. So there's lots of defenders in the world and they see traffic coming from things that they understand are malicious. There are many, many systems to communicate that threat intelligence across companies and those things such as a URL, a malicious URL can be taken down relatively quickly. But if the domain has the ability to stand up thousands of URLs per hour through automation, it becomes a machine on machine war.Natalia Godyla:And on top of the speed and scale, it seems like there's also sophistication in the level of deception. You noted earlier that now it looks like a common user, they can spoof it. So can you talk a little bit more about that? So how does the ability to bypass our detection feed into them being more sophisticated?Donal Keating:Well, let me give you an example. The weakest link now certainly, in security systems, are the humans. So one of the things that most security systems are very good at is recognizing malware, when it can see the malware itself. So for instance, you have a macro embedded in a document, basically that can be detected relatively simply. Well, if you then encrypt that document and send it through an email, the mechanics of detecting the malicious payload is hampered by the fact that that document is encrypted. But then what you need to do is you need to socially engineer the person receiving the document to enter a password and deploy the malicious payload. And that's where I'm saying, people log in rather than hacking anymore. They can assemble enough information about somebody to make an email coming, even from an unrecognized sender to be sort of believable and to encourage a conversation.Donal Keating:And it's not a single email. If you're being targeted, like if you're a CFO or an admin of a system or something, they can be quite persistent over time. They can develop a relationship with that person and then eventually bingo, the malicious payload gets delivered. And they can send that in two parts. They can send an email and say, "Here is the password for the document that I am going to send you." That then, the human reaction to that is, okay, now I am expecting a document from this person. The document comes in and you have the password, that's social engineering.Donal Keating:Now, there are lots and lots of lists of username passwords. And what they tell everyone is, do not share passwords across different systems, especially your private stuff and your work environment. Well, if you're like me and you have a terrible memory, one password is a really attractive proposition. And you may not go with just one password, you make it really clever and add a one, two, three, four at the end of the password. But for people who are looking at thousands and thousands of passwords and millions of passwords, because they've been leaked, they can understand the patterns that people use.Donal Keating:The example is, if I'm trying to hack someone in Microsoft, I'm going to put the word Seahawks somewhere in the dictionary attack, because apparently that's what humans do. It's like, there are certain keywords that people trigger off and think, Oh, nobody can think of Seahawks. And I'm in Seattle. So let's say one individual is compromised in the company, that allows them then to log in to that account and then watch traffic. So what will they do? Someone might change their password, they don't want to be sitting on the email all the time. So what they will do is, they're going to your email preferences and they will forward emails that contain particular words. I've seen an attack where anything that has the word payment, invoice or bank in the email to forward it out to an external Gmail account. Then I don't need to get back into that account anymore because all of the emails containing those keywords are now being sent to me out on a disposable Gmail. I get to see all that email traffic.Donal Keating:So now I have one half of a conversation. And this is where the sophistication becomes really important. Somebody sends in an invoice, we'll say for payment. Well, when that invoice for payment comes in, now, someone has a template of an email that contains an invoice and all of the language. I take the person, the email who sent that invoice in, and I generate a homoglyph of it, meaning a domain that looks almost identical to the sender. Very often it can be even just a different TLD. So instead of microsoft.com, it could be Microsoft dot GZ. And I can use exactly the same username.Donal Keating:So now what I do is I insert a new mail into the chain, so I have the previous thread because they've been harvesting email from that person. And I now put it in my new email and says, whoops, there's a correction on the previous invoice, please change the banking information to this email. And we've seen this in phishing attacks. That sort of thing can be very pernicious. And that is quite widespread. That behavior of monitoring the email, the registration of a homoglyph, and then the conversion of a payment to a different bank account. We see that quite a bit now.Natalia Godyla:So how are we thinking about response to these new threats? What's next for security to combat them?Donal Keating:Well, all the time in the background machine learning, AI is getting smarter and smarter and smarter to protect the assets. And that's why in a lot of the cases that I talked about the objective is to get the username password, to commit that crime that is to login, not to hacking. Now, once they log in, they can do a lot of things. They will deploy remote access tools onto the network to enable them to do a lot of other things like the deployment of ransomware for instance. You need access to the system to encrypt everything. But that first step nearly always is the human element, the engineering, the human element to crack it open.Donal Keating:And, it's a bit like with COVID-19, we're told to wear a face mask, wash our hands and keep six feet apart. The things that we tell people to do are not new or exciting. Make sure you're using multifactor authentication, keep unique passwords for each site, make backups. All of those things, it's good hygiene. But for instance, the use of multi-factor authentication, I've not verified it myself, but I've seen statistics that say that in excess of 90% of username password compromises would have been thwarted if people had been using multi-factor authentication. So-Nic Fillingham:[crosstalk 00:22:15] our ID team will we'll quote 99% or greater. It's pretty significant.Donal Keating:Yeah. so that to me is the wearing a face mask and washing your hands of protection from cyber crime. I have a small carve out for nation state. If the Russians or the North Koreans want to go after you as an individual, you need to tiptoe very carefully. There's all sorts of nastiness that can be done to you as an individual. But the reality is for most targets, it is this people access, username password combination, they log in and then they start the progressive taking over the account to do whatever it is they do. The worst being ransomware.Donal Keating:It's not unusual. So, you talk about increasing sophistication. Ransomware was a big thing and then it took a hit. Why did it take a hit? Because people had deployed ransomware that were really destructive ware. They encrypted stuff and there was no keys existing. So suddenly everyone says, "Well, there's no point in paying a ransom because I'm not going to get my stuff back." So then the criminals had to go and do something else to prove that no, no, no. Really we can decrypt your stuff. So it's a kind of a marketing campaign.Natalia Godyla:There's something very comical about the fact that the hackers had to get people to trust them that they were going to do what they say they're going to do.Donal Keating:Oh, absolutely. Yeah, yeah, this is business no different from any other business. You get a bad reputation for something, you got to fix the reputation, or you got to get another way of leveraging people to do what you want them to do. And that's why I say there are people who are specialized in thinking up these social engineering things. They may not be coders at all, they may not know how to turn on a laptop, but they understand how humans work. There's other people then who are geniuses at writing the malicious payloads, writing the PowerShell scripts, [ossucating 00:24:23] the PowerShell script so as normal detection won't pick them up that.Donal Keating:This is a whole stack of various things with various levels of sophistication and increasing sophistication. But the criminal will tend to go to the softest part of the ecosystem to make their money.Natalia Godyla:You mentioned that part of the challenge right now is that users are just getting smarter and so the hackers are responding in turn. If our users have been taught cybersecurity education on what is a phishing email, how is the evolution of education going to happen or what's next for education for the users so that they can prepare for this next wave of social engineering attacks?Donal Keating:A whole bunch of interesting things tumble out of that question. The first one is we used to always say, go look for mistakes in the phishing email. If it looks like bad English, it's probably phishing or whatever. I actually heard at a conference that they were sometimes deliberately put into an email to trigger the spiny senses of anyone who is halfway security savvy. And the reason was, the person who fell for the phish was then going to be more gullible. They were trying to cut down the amount of traffic that was coming to them for someone who would do... I'm talking specifically about something like tech support fraud, where you'd get an email that your computer was about to run out of its license key or it had some horrible vicious malware on it, and you needed to contact this number.Donal Keating:They would actually put in the sort of deliberate clues to anyone who was savvy. The result then, the people who were calling that number, were going to be much more gullible. So you also have to understand what is the goal of the criminal? And the phishing emails yes, they are getting much, much more sophisticated. But we especially in cloud, when you're looking at O 365 advanced threat protection, that description that I just told you, if something coming from microsoft.com and then another email comes in from microsoft.gz, we actually have exactly that detection running. These look alike domains, where you haven't communicated with that domain before, advanced threat protection will regard that as a high risk email.Donal Keating:So for-Nic Fillingham:That's a homoglyph, right Donal? You mentioned-Donal Keating:Homoglyph, that's exactly [crosstalk 00:27:03]. Yeah homoglyph. It means something that looks like another thing. So, the classic example are the Microsoft spelt and O, you replace it with a zero. The I you replace with a one. And this business has become during the election for instance, people will look for the registrations of all of the legitimate vote Arizona or whatever. it was votecolorado.com I think, and of course, someone registered votecolorado.co I think it was. It looks exactly like you would expect. The response to something like that for government especially, you should only be standing up state material on a dot gov domain.Donal Keating:So there are lots and lots of things that we need to educate people. The IRS, for instance, will never ask you to pay your income tax with iTunes Donal Keating:[inaudible 00:00:28:00]. You would wonder how does that scheme ever work?Nic Fillingham:Yeah.Donal Keating:But they say that [crosstalk 00:28:08][inaudible 00:28:07].Nic Fillingham:I've always wondered what the sto... I've never had one of those phone calls, because I really want to hear the logic from the person that's trying to tell me what happened to the IRS as an institution where they now are relying on the consumer retail supply chain and the company Apple. And that's the only way they're able to accept funding. I want to hear that story straight from the person trying to try and pull the wool over my eyes.Donal Keating:One of the things we do actually is, we call these people. So, every time we get the numbers, there was one that [crosstalk 00:28:41] -Nic Fillingham:[inaudible 00:28:41] home.Donal Keating:... we do actually what we call test phone calls. So, if you look at some of the other, I know this is not the subject of the podcast but, we've recently had big raids in India where 10 call centers were raided. All running tech-support scams. Taking people who thought they had something on their computer and paying subscriptions of to $300 a year to keep your computer protected. They are unsophisticated crimes. But the sophistication of persuading someone that they do have a problem is sophisticated.Nic Fillingham:Awesome. Well, Donal, thank you so much for your time. Again, the report that we're referencing here at the top of the conversation is the Microsoft Digital Defense Report. It's about 38 pages of fascinating insights into the state of cybersecurity. And a lot of the topics that Donal touched on in this conversation are elaborated on in much more detail there. We'll put the link in the show notes. Again, Donal, thank you so much for your time.Donal Keating:Very happy to be here. Thank you.Natalia Godyla:And now, let's meet an expert from the Microsoft security team. So, [inaudible 00:29:49] more about the diverse backgrounds and experiences of the humans creating AI and Tech, at Microsoft. Today we're joined by Michelle Lam, a threat hunter at Microsoft. Well, thank you for joining us, Michelle.Michelle Lam:It's a pleasure. Thanks for having me.Natalia Godyla:Yeah, of course. Well, can you start the show by just telling us a little bit about your day-to-day? What do you do at Microsoft? What is your day-to-day look like?Michelle Lam:Sure thing. So, I could tell you about the boring things, which is that, I look at a bunch of data and spreadsheets. And I look at them and I say, " Bad things happened, or everything is fine and people are off doing their normal things." But I guess the more complicated story to my work is that, what I look for is patterns in data that might indicate malicious activities. So that, might, could be anything from human-operated ransomware, to new malware strains, or even just new pivots in activity in general. So, things that we can feed into the rest of the Microsoft ecosystem for security.Natalia Godyla:And threat hunting is a relatively new space, correct?Michelle Lam:Yes it is. But I think it's interesting, because the concept of threat hunting has existed, but it's always been in other realms and security. So, if you think about things like security-tracking or security operation centers already looking at alerts and whatnot, or on the idea of incident response, the concept of threat hunting is already baked into a lot of these more traditional spheres of security. So, yes, it is new, but I think it's always existed in one form or another.Natalia Godyla:Do you feel like it's become a stand-alone part of security now? So it's been baked into these different aspects of security in the past, but now we need it as a stand-alone function?Michelle Lam:I think that really depends on where you're at? What kind of organization you're in? And what are you trying to do with that data? Because, it doesn't make sense to go hunting for data and the deep, deep sea of data that exists. If you have data that you need to analyze for a purpose, I think that's what threat hunting is really great for. For me, I'm looking for data because I want to figure out," what context can I give it that will be helpful to a customer? Or to the rest of Microsoft as a whole?" I think if you ask that question to anyone else in any other organization, then it's a different story because what part of that data is interesting to you is different for everybody, depending on your sector, depending on your organization, depending who you are even.Nic Fillingham:And what is that sort of, Stat focus area for you, Michelle. How do you scope down that near limitless sea of data for looking for threats?Michelle Lam:That's a fantastic question. I think I'm really interested in looking at different techniques that already are well known in the industry. So, things like using PowerShell, using Scripts, different ways of disabling security mechanisms. Those are techniques that already exist and can be used in one-off occasions. But what I'm really interested in, and when I look for this data is how I can correlate all of these little things that might happen one at a time, in a benign case. But if they happen all together, how can I combine that and say, "Is this related to a specific activity group? Or is this someone who's doing a penetration test? What sort of things can I identify about how they were executed or how they're launched? And [inaudible 00:33:24] can I make that connection to something else and provide that context elsewhere?"Nic Fillingham:Would you mind telling us about your journey into security? And then how you found yourself working for Microsoft?Michelle Lam:Sure thing. So I guess my story, even entering security really has to start with this journey of me entering tech as a whole. So, I myself, I come from a low-income family, and a family of immigrants. And so it was really interesting for me to decide what my career path was going to be as I started this journey of, "okay, well, I'm leaving high school. Where do I go?" And the direction that I was going to take was in the business direction. And I ended up deciding, with the encouragement of a few of the teachers that I'd had at the time to go into computer science. I won't lie, I was a little motivated by money, who isn't? But when I actually got into college and I discovered what you could really do in the field, I was really intrigued.Michelle Lam:And I tried to figure out, " What does it take to be more technical? And what else is out there?" So, while I was at college, I actually joined a security club. And there were a couple of students there that helped mentor me for the process of writing my own code, to do the very simple things like encrypting or decrypting data. And that moved on into me actually getting internships and learning how to code and ending up at Apple and working in cryptography and wondering, "what the heck am I doing? This is so cool, but I have no idea what I'm doing." So, my entry into cybersecurity was really fueled by this curiosity of, "I have no idea what I'm doing, but I'm going to continue to do it". And for me, that continued up until my last year of college. When for a lot of low-income and first-generation college students, there's this very common pattern of, it takes you a little bit longer to graduate from college because no one you've ever known has been through this process.Michelle Lam:And for me I was, to be frank, I was scared. I didn't know what it would mean for me to go out into the industry. So, I wanted to figure out what I wanted to do. And I wanted to figure out what to do in security. So, I actually attended a Women in CyberSecurity Conference, and I attended a talk by these two women that I really admire in the industry, Malware Unicorn and Maddie Stone. And they were super friendly and they did this course on Reverse Engineering and Assembly. And I was like, "Oh my Gosh. This is so cool. This is a field where I don't necessarily have to be coding, but I can put a lot of that low-level knowledge to use that I've learned in college and I can figure out what malware does. I can solve a problem."Michelle Lam:So I really took that into consideration as I moved forward. And I ended up teaching a course for my senior project about reverse engineering. I didn't know very much at the time, but that is what I decided to teach. And I also took an internship that was based in Incident Response and Computer Forensics at a government laboratory. And it was a super weird internship to have. It's not normal, I think for a lot of my peers to have that experience of, you go to a government lab, and it's a very different experience than what you expect. And you also reverse malware and you figure out what the baddies do. So, it's a little hard to explain to your peers, but I absolutely loved it. And I figured out, "This is what I want to do when I grow up. When I exit college and I graduate this is going to be it."Michelle Lam:So, that's my short story of how I got into security. And from there, it was a bit of a pivot before I ended up at Microsoft itself. So, after college, I had decided to go down this route of, " I can do a little bit of incident response. Okay, I'm going to take a job in incident response." So, I moved to Atlanta to take our role in incident response consulting, where I learned a lot. And they did a bunch of little things, but I didn't really know if I was advancing myself or learning about the baddies in the way that I wanted to. And it so happened that I attended a conference that's very focused on reverse engineering called REcon, which is in Montreal. And I met a few people that I'd actually met at some other security conferences when I was a little more junior in my college career.Michelle Lam:And I was like, "Well, what's going on?" And they're like, "Hey, I'm at Microsoft. I do cool things. You should come here and do cool security things too." And I was like, "But, are you sure?" And they're like, "yeah just chat, it'll be fine." Long story short, a few months later, I took a job offer from Microsoft, for my current team, The Microsoft for Experts team. And here I am getting to hunt on and look at really interesting data. So for me, it's been this really interesting journey of exploring and running into this field, and trying to figure out, how do you enter it without a ton of mentorship from those around you?Nic Fillingham:If someone listening to the podcast sees a bit of themselves in your story here, what would you recommend for how they maybe, go and find some of those support groups, maybe some of those mentors, maybe some of those industry bodies that could help them out early on in career, to get some of these experiences? Is there any tips or tricks you'd want to pass on?Michelle Lam:Yeah. So, I would say the biggest things for me were building a really strong network over social media. So that doesn't mean, go out and tweet all the time because, I certainly don't, but I definitely found a lot of really resourceful things on Facebook groups and Twitter groups. Even some of the internships that I actually applied to and got offers from, were things that were shared on a Facebook group for like Women in Security or Women in CyberSecurity. I only found out about a lot of conference sponsorships for following different Twitter feeds and seeing, "If I follow a bunch of these people, someone at some point is going to share some way that I can attend DEF CON or another conference for free or for reduced rate or some form of sponsorship. So, that's been really important for me as I grew my career and I definitely plan on giving back at some point because, I would not be here if it weren't for that.Natalia Godyla:It's interesting because I think for many of the people that we've chatted with, it's been a little bit more of a winding journey to security. But in your case you started with Comp Sci but you ended up thinking about security already when you were in school. So, how was that experience seem different than some of your other colleagues who have started in other backgrounds and have then made their way to security? Do you feel like it's been helpful to know that security was your path when you were in college? How does Comp Sci factor into it?Michelle Lam:For sure. So, in a way I do feel like it's been really helpful for me to join security and find out about security so early on, because, I feel like I've been able to learn a lot and be able to put a lot more of, I guess, some of the foundational computer science skills into use. Things like learning assembly which in college, if you're a college student right now and you're taking any assembly course, you're like, "I'm never going to write in this super low-level language. Why am I doing this?" Well, it so happens that when you work in this industry, you want it. Or if you take compilers. Compilers is surprisingly useful in security.Michelle Lam:So, I guess, what I think about a lot in terms of my career progression in comparison to some of my peers, is that I do feel a bit of a disadvantage sometimes because, I'm still quite junior in my career. I'm maybe two or three years out of college at this point, so there's still plenty that I have to learn, but I do feel that I don't have that traditional security experience. A lot of folks on Twitter and in the traditional security spheres, talk about this concept of, "You need CIS Admin experience to be a security person. You need to know all of these things. You need to have worked for 10 years, 15 years in security before you can become a threat hunter." And I'm like, "Did I make a career mistake?" To be honest, I have imposter syndrome about it quite a lot. But, if you think about it, everyone has this different Michelle Lam:Take on what they're looking for when they're threat hunting. And what's valuable for me, coming from such a junior and such an almost indoctrinated security experience, is that I see these things and I see that they look bad, but I have a different way of relating to the data in which I might say instantly, "This is bad, and here's why," or "This looks weird," and someone's like, "No, you're wrong." And I'm like, "Well, you're just saying that because it looks like something you've used before. But I've never seen it and it looks malicious."Michelle Lam:So I think it's all about, there is a joy and a need for us to have different perspectives when we're hunting across data, and when we're looking across data. Because everything looks different to everyone, especially in this industry. And it's about, how do you take those arguments and how you condense it down to, "It's not argument. It's us trying to understand the data," that's really important.Natalia Godyla:So Michelle, how does AI and ML factor into your role? How do you leverage those tool sets to help our customers?Michelle Lam:We actually use AI and ML in several different detections that we use. Whether that be ranging from the antivirus in the AV side of things, to things like Windows Defender for endpoint. We might be looking at different signals and putting those together in different ways to figure out, if users are performing this type of recon several times in a row, that's malicious, that looks like exploration activity, right? There are other ways that we're looking at using it that might involve... We see this particular activity group perform this activity in sequence. When we see that, that's an indication to us that there is maybe this activity group is on this machine.Michelle Lam:And that's really interesting data for us to have, especially as we hunt and we track that data because maybe we're not completely sure the history of what we've looked at in security, I think, has always been very indicator of compromised base. It's been very focused on, we see these hashes, we see these files, we see these IP addresses, but what happens in a world when you can't really use that information anymore to hunt? For me, I'm really interested in when I see this behavior, how can I use that? I think that's something where AI and ML is super powerful and super helpful for us as we figure out like, if I were to move away from a world of IOCs, this is where we would go and this is how we would build a detection in order to actually catch a group in action.Nic Fillingham:We've already spoken to a few folks on the podcast, Michelle, that are working on behavior based detections and try and leverage ML and AI to do that. I'd love your perspective on your role as a threat hunter and what makes threat hunting as a process, and as a task, and as a role, what makes that sort of a uniquely human-based function, as opposed to simply a bunch of algorithms out there running in the cloud?Michelle Lam:I think there's two different ways to think about this. And one of them is that, well, how did the algorithms get created? You still have to teach the algorithms how to use that data. We are working with several data scientists to actually figure out how do we feed your algorithm that data that actually says that this is tied to an actor. And you can't do that without actually having a human to hunt across that data and understand what it means.Michelle Lam:I think the second component to that question is that attackers are human too. If they weren't human, then it would probably be a lot easier for us to catch them, and maybe we wouldn't be having this conversation, and maybe I wouldn't be having this job. But because attackers are human, we have to pivot ourselves to align with them. You can't expect machines to catch everything that a human is doing, but if we have humans that are looking at other humans activity, we might be able to predict and start learning off of what they're doing and build that into our algorithms so that algorithms can assist us to do the heavy lifting while we look for the new things that are happening.Nic Fillingham:I love it. That was a great answer.Natalia Godyla:This is a bit of a big picture question, but it sounds like a lot of your path to security has really brought you to this role to threat hunting. What would be next for you? Are you interested in continuing to pursue a career in threat hunting, or are you looking to explore other aspects of security down the line?Michelle Lam:I think that's a really wonderful question and it's tough for me to answer, being so early on. I think about a lot of the questions that you get asked about when you're pretty junior in your career, right? I won't lie, everyone has asked me, "What's your dream in five years? What do you want to do in five years?" And I'm like...Natalia Godyla:Every time you come home for the holidays.Michelle Lam:Yes. So, I don't know. I think about this a lot and I have to say, I actually do think I'm in my dream position right now. It's a different question of where I want to take my role and what I want to do with it really, because I love hunting across data. I love finding weird things. Like, what does this do? And how can I learn what it's doing? And, do I [crosstalk 00:05:11]?Nic Fillingham:What's the weirdest thing you found? What was your, like, you woke up in the middle of the night with like, "Oh my God, that was so weird?" Has anything stuck out?Michelle Lam:I want to say that I could answer that, but I'm not sure that I can actually share it, so it will just have to be a mystery.Nic Fillingham:Can you hint at something that doesn't jeopardize any OPSEC?Michelle Lam:No, that's kind of the joy about being a threat hunter. I don't want to share too much, I don't want to tip anybody off.Natalia Godyla:What big problems are you passionate about solving in cybersecurity? Are there any challenges that you're seeing that you'd like to tackle throughout your career?Michelle Lam:That's such a hard question to answer, because I feel like I am tackling a lot of really big problems as it is, fighting the fight against human operator ransomware is huge. But I think if there's anything that really is important to me in the way that I was raised and how I got into this career, it's about how do we make security and option for those who security might not have occurred as a first option? How do you make sure that security shows up for those that are underrepresented communities?Michelle Lam:Because it's not just a matter of physical security, but cyber security is so incredibly important for these communities. How can you make sure that they have access to it when they need it? There are a lot of scenarios that these communities have to reach out and figure out how they can get support in tough times in these kinds of situations. I would love to figure out what does that look like for me and for others.Natalia Godyla:I feel like this comes back to what you said earlier about all the communities that you can reach out to. It's always an aspect of you reaching out to try and find these communities. I think that proves out that some of these resources or niche are difficult to find right now, and that you have to put the effort into doing it. So just easing that access.Michelle Lam:For sure. And I think that's something that I've always struggled with, is this idea of how do I balance my career, progressing in my career versus helping the communities that I've come from. I've done work in the past, a volunteer with organizations like Girls Who Code, and we've brainstormed quite a bit internally of how do we volunteer our efforts to actually teach underrepresented communities, people of color, women who are younger, who might not traditionally come from a tech career path? How do we teach them these cybersecurity skills? Because we're constantly running out of cybersecurity professionals and the only way to solve it is to grow the base of cybersecurity professionals that exist. So how do we teach them and how do we introduce them to this field in a way that makes them feel like they belong?Michelle Lam:I feel like that's a really important problem to solve, especially because I come from a place where had I not gotten lucky at college and ran into a club full of cybersecurity people, maybe I wouldn't be here. And for me, that's scary to imagine because I love what I do. And I love that I get to feel like I'm saving the world. So what does it mean if I teach others to do that? How do I do that? That execution is... I don't know, the idea of that is so interesting to me and I think there's a lot of impact that I could have.Nic Fillingham:Michelle, are there any organizations you want to plug?Michelle Lam:I would like to talk a bit about Blackhoodie, which is this really awesome organization that was founded by a couple of ladies off of the Twitter security community. It's really a community of women who are teaching these reverse engineering workshops that are meant to be technical, and to really teach you about technical low-level skills that could get you into reverse engineering or into the security community. All of the women that I've met from being a part of Blackhoodie have been absolutely amazing. I stay connected to them to this day, and I've even taught a course for them at a previous Microsoft conference, BlueHat. If you are a lady listening to this, I would super recommend that you go check them out on Twitter and see if they've got any courses coming up that you might be able to attend because they're free and they're taught by some really, really intelligent woman across the security industry.Nic Fillingham:What do you like to do in your free time, michelle?Michelle Lam:That's a really great question. My favorite-Nic Fillingham:Afar from quarantine for eight months.Michelle Lam:Okay, fair. Quarantining is a fantastic hobby. My hobbies are drinking lots of bubbly water, playing with my puppies, and fashion. I love fashion. Someday, if I'm good enough, I would love to compete with Jessica Payne in Malware Unicorn. We'll see if I get there, but I want to have a security idle fashion competition.Nic Fillingham:As in where you make clothes? No, what would that look like?Michelle Lam:I don't know. I guess we could all just attend a security conference and wear ball gowns and I don't know, compete against each other. I'm not sure what it would look like.Nic Fillingham:Tell us about your puppies.Michelle Lam:Yes, I have two puppies. One of which was obtained during coronavirus, her name is Kali, after Kali Linux. Very secure. And our other pup is Nelly, who is a beautiful rescue.Nic Fillingham:Do they have an Instagram account?Michelle Lam:No. I mean, even if they did, I'd like to maintain a little bit of OPSEC, so maybe not. Sorry.Nic Fillingham:Well, Michelle, we're very happy that you found your path to both security Microsoft and thank you for doing the work that you do and best of luck helping others find their path as well.Michelle Lam:Thank you.Natalia Godyla:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us @msftsecurity or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe.Natalia Godyla:Stay secure.
11/11/2020

Protecting Machine Learning Systems

Ep. 5
In this episode, hostsNic FillinghamandNatalia Godylaspeak withSharon Xia, a principal program manager for cloud and AI at Microsoft, about the role machine learning plays in security. They discuss four major themes, outlined in theMicrosoft Digital Defense Report,including how to prepare your industry for attacks on machine learning systems, preventing attack fatigue, democratizing machine learning and leveraging anomaly detection for post-breach detection. Then they speak toEmily Hacker, a threat intelligence analyst at Microsoft, about her path from professional writing to helping find and stop attacks. In This Episode, You Will Learn:How to prepare for attacks on machine learning systemsThe dangers of a model poisoning attackWhy it’s important to democratize machine learningHow a humanities background helps when tracking threatsThe latest methods attackers are using for social engineering Some Questions We Ask:Why are most organizations not prepared for ML attacks?How do youassess the trustworthiness of an ML system?How can machine learning reduce alert fatigue?What kind of patterns are analysts seeing in email threats?Why is business email compromise treated differently than other threats? Resources Microsoft Digital Defense Report, September 2020 Microsoft Security BlogTranscript(Full transcript can be found at http://aka.ms/SecurityUnlockedEp05)Nic Fillingham:Hello, and welcome to Security Unlocked, a new podcast from Microsoft, where we unlock insights from the latest in news and research from across Microsoft Security Engineering and Operations teams. I'm Nic Fillingham.Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft Security, deep dive into the newest threat intel, research and data science.Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft Security. If you enjoy the podcast, have a request for a topic you'd like covered, or have some feedback on how we can make the podcast better...Natalia Godyla:Please contact us at securityunlocked@microsoft.com or via Microsoft Security on Twitter. We'd love to hear from you.Nic Fillingham:Hello, Natalia. Welcome to another episode of Security Unlocked.Natalia Godyla:Hello, Nick. How's it going?Nic Fillingham:It's going really well. We've got some early data, some early data hot off the presses from our listeners. I thought we might jump straight into that, instead of finding out what smells have permeated my basement. Is that-Natalia Godyla:Great to hear it.Nic Fillingham:Yeah. So, we just got some data coming out of the various podcast hosting platforms, and we have been listened to in over 60 countries, which is, I mean, that's amazing. That's if my math is correct, that's a quarter of all sovereign nations on earth. So that's pretty cool. Right?Natalia Godyla:Yeah, we're making headway. I feel like global just makes it sound like such a big deal. We're currently listened to in Estonia, Kazakhstan, the UK, both of our father slash motherlands Australia and Poland. So, it's great to see the representation. Thank you all.Nic Fillingham:I want to list a few more, because I just want to make sure that the few listeners that I think are there, they're getting a shout out. Myanmar, Azerbaijan, Albania, Haiti. Thank you so much to all of you listening to the podcast. On today's episode, we speak first with Sharon Xia, who is the Principal PM in the Cloud Security team. This will be the first of five or six interviews we're going to have over the next few episodes with authors and contributors to the Microsoft Digital Defense Report, the MDDR. You can download that at aka.ms/whackdigitaldefense. This is what I like to call the spiritual successor or the successor to the Security Intelligence Report, the SIR, which is a document that Microsoft has produced for the last 15 years on trends and insights in the security space. Natalia you've read the report. What would you say to folks that are sort of thinking of downloading it and giving it a read?Natalia Godyla:Well, first off the machine learning attack section is definitely one to read. It's fascinating to read about the new attacks that there are, model poisoning, model inversion, we'll touch on them in future episodes. So I'll leave it at that, but lots of new goodness, and just in general, the MDDR is a huge effort within Microsoft. It's highly collaborative and it brings together a ton of experts who really know their stuff. And so you'll see just that breadth of knowledge and intelligence when reading the report and in all of our upcoming episodes, since we'll be spotlighting, a number of experts who were contributing to the report, we also, in addition to the MDDR, we'll have Emily Hacker on the episode who is a threat analyst, and she'll talk about her journey from literature major to cyber security realm.Nic Fillingham:Awesome. We hope you enjoy the episode, Sharon Xia. Thank you so much for joining us. Welcome to the security unlocked podcast.Sharon Xia:Hey everybody, thank you for inviting me.Nic Fillingham:Oh, you're very welcome. We're happy to have you, could you give us sort of a brief introduction to yourself? What's your title? Tell us about what you do. Day-to-day in your team, sort of the, the mission and goal of your role and the team that you run.Sharon Xia:Sure. So I'm the principal program manager, which manages the PM team in Azure security data science team. And we have six PMs with 30 data scientists. Our day to day work is using machine learning to write threat detections and other features that protecting Azure, protecting our customers and also protecting machine learning models.Nic Fillingham:So that's a team of 30 data scientists, sort of machine learning experts, that are protecting all of Azure and Azure customers. Is that right?Sharon Xia:That's right. So actually including more than Azure customers, because our products and our solutions applies to on-prem system, as well as, as a crowds like AWS and the GCP.Natalia Godyla:Microsoft had recently published the Microsoft digital defense report, in which we talked about machine learning and security. And as I understand that you contributed to this report, and one of the themes was something you just touched on, which was preparing your industry for attacks on machine learning systems. So can you talk a little bit about how the cybersecurity space is viewing these machine learning attacks? What's happening? What are the measures organizations can take to protect themselves against these attacks?Sharon Xia:Yeah, as we all know, machine learning takes an increasingly important role in the operations and in our day to day life, right? It applies to not only like a facial recognition or voice, or even apply in many medical devices or analysis.Sharon Xia:So it's just embedded in our day-to-day life nowadays. But to the attacks, cyber attacks to the machine learning system and the machine learning models, we're just getting to know these. And it's more and more prevalent, based on our research. We did a survey to 28 large customers, enterprises, 25 told us they have no idea what are the attacks. You know, it's there. And to the machine learning system. So that's kind of alarming, right? And for example, the model poisoning attack, and real world example is, attack can manipulate the training data to make a street sign classifier, that to learn, to recognize a stop sign as a speed limit. So that's really dangerous if you think about it, right. If you're driving a Tesla and you're supposed to stop. I'm not saying Tesla is vulnerable to this attack, but this is kind of an example of a model poisoning attack.Nic Fillingham:So, we talked about the report. So the digital defense report, the Microsoft digital defense report that was released, it's a pretty lengthy document. It's full of a lot of incredible guidance. You and your team specifically contributed. And what we're talking about on the podcast today to the section within the state of cyber crime, which is called machine learning and security. And as you, as you just touched on that, the very first of the four trends that are called out there is simply just awareness, and preparing. I want to just touch on that stat that you mentioned just a minute ago. So you surveyed 28 organizations, 25 of those 28 just said that they don't have a plan for, they don't have tools. They're not prepared for adversarial or ML. Is that an accurate takeaway?Sharon Xia:Yeah. So what do we, we seen at this moment is a security team and the machine learning team are running on two parallel orbits right now. So they know to not interact, that they are doing their own things, not aware of security on machine learning system. Yeah. So the first step we, we have been putting a lot effort is the community awareness. And we definitely need community help to pull those orbits together. Finally, interact, right? So that's a call to the community. Like that's a raised that awareness and walk together to first aware of these, then due to some tools, trainings to get our defense up, you have red team and a blue team, right? So they'll get our defense up to the speed.Nic Fillingham:You mentioned a few types of sort of attacks there against models, model stealing, I think is relatively self-explanatory. Model inversion is interesting the way you explained it, it sounds like it's the ability to sort of reverse engineer or extract the data out of a model. The one that I sort of want to touch on here is, is sort of model poisoning. So you, you explained it as poisoning a model so that instead of seeing a stop sign, if it was trying to identify road and traffic signs, it may see something else. It may see a speed limit or something. How does that happen? How do we know how model poisoning works? Have we seen it in action? Have we been able to sort of post-mortem any successful model poisonings to understand how it actually happens?Sharon Xia:Yeah. There are multiple ways to have the model poisoning happening because the- like I described, it's about manipulating the training data, right? So if you have access to the training data directly, you could manipulate it, but that- on purpose that needs some machine learning knowledge to do it right? So you can also, let's say if at a first glance, you don't really have the access to the poisoning data, but then you have access to the network. So you can do a traditional main, the middle attack, to disrupt the training. And there are two kinds, integrity attack or availability attack. So if you disrupted the training model to run the training effectively, this is basically kind of attack from availability point of view. And if you change the data, like the street sign classifier, to make it read to us a speed limit, that's called a kind of integrity attack.Sharon Xia:So there is some multiple ways to do that.Natalia Godyla:So how are we thinking about assessing the trustworthiness of an ML system? It sounds like it's clear that we're still at the awareness stage and we're partnering with organizations to build out frameworks. What elements are we bringing into these frameworks or standardizations to measure trustworthiness of ML systems and identify whether they've been impacted?Sharon Xia:Yeah. We came up with kind of an amendment to our item, Microsoft, an amendment to our security development to ripe cycle. One of the process is the threat modeling. So we have machinery needs threat detection, the threat modeling for machine learning systems. That's at a specific guidelines, questions, how do you do threat modeling on a machine learning system to identify, those potential attack surfaces and the potential risks in the development process? So that's the first step we are taking to, this is also part of a awareness effort, right? When you are doing the regular threat modeling, and you are asked for these questions, for example, if your data is poisoned or tampered with how would you know? Right? So then the follow-up question is, do you have telemetry to detect a skewed data in quality in your training data? Right. And are your training from user supplied inputs? If Sharon Xia:Yes. Right. What kind of input validation or sanitization are you doing or if your training is against an online data store. So what steps do you take to ensure the security of those connections? There are long list of questions we ask in our, regular threat modeling like that. We actually published the document a while Microsoft security engineering site, it's a public documentation, was all these questions for the community to referenceNic Fillingham:Sharon, what should, Microsoft customers know about how we are securing our AI systems and machine learning models that are in production. Obviously we're doing everything we can, we're investing heavily, but this is a very new area.Sharon Xia:Right. Yeah. So like I said, at the very beginning, we work with Microsoft scale, right. And the incorporating and the Scott battery register, they all aware of the effort. So we will work with the responsible AI at the Microsoft white. Also, we have an ISA working group that focus on, responsible AI and the ad was three AI. So it's a Microsoft's effort to make sure at our engineering part, we are building a secure machine learning system.Natalia Godyla:And aside from protecting our machine learning systems, how are we taking this technology, taking machine learning and applying it to our security solutions so that we can empower security teams?.Sharon Xia:Good question, we're building solutions, detections in our cloud native, SIEM product, Azure Sentinel. So it's not being released yet, but we are working on it so that, our customers can use the tech knowledge based on our experience, our study and to apply it to their machine learning systems, to at least, detect those attacks to their machine learning system. And another end is we have grad team actively, doing red teaming activity to the machine learning system. And we also keep learning the new attack techniques in that way.Nic Fillingham:Got it. So we've covered that first trend here, which is really about awareness of this new category, of this new sort of threat of attacks on machine learning systems. I might move on to that the second of the four trends that are in the report and that one is talking about leveraging machine learning to reduce alert fatigue. Can you talk a bit about that trend for us, what happened in 2020, or sort of in the last sort of 12 months around how ML has advanced in the use of ML to help reduce alert fatigue?Sharon Xia:Yeah. So, when you look at the security operations, the security analysts in every organization at dealing with a lot fatigues. I think if you are working in security operation field, you have to deal with salient alerts from different products like Enterovirus or Pareto Network, firewalls, and then EDR solutions, XDR solutions while for, all these kinds of security solutions, just sending alerts like a thousand alerts. So a typical, security analyst in the security operation center for an S 500 enterprises, they get about, 2000 alerts. They have to deal with daily that's obviously cause lots of issues, right? So on the other end, if you're not able to go through all these alerts and you may drop off the real attacks, but all these alerts, there are lots of false positives. So there is a survey saying some products generate more than 50% false positives, or even 70% false positives that really preventing the defender team, the SOC analysts, to deal with the two attacks, real threats.Sharon Xia:So one of the reason why are all these false positive is because the tradition or low based approach doesn't adapt to the change of the environment. The advantage of machine learning is it learns that new environment, right. And adapts to the change of the environment. And so we are looking at the Azure Sentinel, we have this machine on threat detections called a fusion. Fusion Technology use three different machine learning algorithms and a power, provide a graph and use kill chain and use different machine learning algorithm. We basically incorporating signals from multiple products, multiple sources like your identity management system, your firewall, your EDR, your end points, also sources of data and the lock, all these anomalies and chain the men together in the sense of the kill chain, threats and the coaching sense and fired like a high fidelity alerts.Sharon Xia:So give you an example. If you find a suspicious login from a tall Barraza meaning, an anomalous IP address, then this is maybe not that suspicious. But it's not meant a high fidelity, like this account is compromised or this login is malicious, right. But then if you follow by unusual mass download or setting up a mailbox, forwarding law in outlook and the forward, all the company, business email to a Gmail or something like that, those activities, if you chain those activity together, you can see obviously there is something like a data ex filtration or seek to attack, depending on different signals, right? So this is how we use machine learning to alert, reduce alert, fatigue and give you high confidence and high fidelity alerts. Allow the security analysts to focus on, these, their energy to investigate and mitigate those threats.Natalia Godyla:The volume of signals and the need for specialized skill sets, data science skills to develop these ML models. That brings us to a third theme, which is democratizing ML. So can you talk a little bit about, what our ask is to the security community and how we view democratizing ML as a next step in the progression.Sharon Xia:In a way we've seen in the industry, we're short of security experts. We are definitely short of, data scientists to build good, high quality threat detection. We need to boost knowledge. Security knowledge, as well as machine learning knowledge and going further. We also need domain knowledge, which I mean, industry domain knowledge is if it's a financial industry or healthcare or energy, or Microsoft, we have Saudi security experts, right. For IT, information technology. We also have, hundreds of data scientists like my team, have certainly different full-time data scientists. So we also work like across the team, we work with our threat intelligence team, we work our security analysts team leverage their knowledge. So when you use the product we produce at a Microsoft like this threat detection, it's the result of multiple teams, multiple efforts, all the expertise in there, but we don't claim we know everything.Sharon Xia:And like I said, a generic machine learning, not algorithm may work well in one environment, but less effective in another environment because of some special circumstances in that organization. And we fully realize, there is a lack of resource of data scientists in the enterprises. So what do we want to do is enable security analysts. Experts in security and that they are domain expert in their organization. To be able to improve the beauty in machine learning models, being our products, for example, Azure Sentinel to include quality of the model produce better signal in their environment. So this is the effort of democratizing machine learning in the SOC ML. So we are building this interface and this technology and in the product. So security analysts can customize our machine learning models without any machine, any knowledge.Nic Fillingham:And Sharon that leads us to sort of the fourth and final sort of big trend that's in the report. And again, this is the Microsoft digital defense report, 2020, which you can download@ak.ms. Whack, digital defense and Sharon, that sort of final trend that's discussed here is about leveraging anomaly detection for post-breach detection. We had Dr. Josh Neil on the podcast. I think in our second episode, his team is actively involved in this area. Can you talk a little bit about the sort of final trend that's called out in the report?Sharon Xia:Yeah. So behavior changes over time, right? And that's the beauty of machine learning. So, machine learning model, we observed the normal behavior. And then we signal if there's anomalous behavior happens, unusual activities, and these are important for the post-breach detection. If we observe anything abnormal happening, we stitch all these abnormal together and then find those strong attack, irrelevant incidents. So there are the supervised machine learning models and the unsupervised machine learning models. And when we found out, because supervised machine learning models requires labelling and this put lots of demand on our customers. So we are actually now switch to more and supervise the message to attack, detect those behavior changes or abnormal behavior changes that will automatically adjust in a profile, a user or a machine or IP. We call those, all of them entities in the customer environment and they learn those normal behavior versus abnormal behavior. So that's how we, use anomalies to detect those post-breach detections. And because of these kinds of unsupervised machine Sharon Xia:Model. Most of the models, we are able to do streaming fashion because it doesn't require training. So to be able to do streaming fashion, which is bring us to the meantime, to detect in the milliseconds, right? This is important. If you can detect a potential compromise in near real time, we want to do that, right. Otherwise like "Oh," nine months later, or maybe two days later, you'll find a compromise, right. So-Nic Fillingham:If it's not instantaneous, it's sort of useless.Sharon Xia:Right, I know, yeah. So this is really a truly important advantage in tech knowledge. We are able to detect those anomalies in real time or near real time and stitch them together as quickly as possible.Nic Fillingham:Well thank you, Sharon. There's a lot in the five pages of the machine learning and security section of the report, there is a lot of content to cover and we've really just touched on each of those four trends.Nic Fillingham:I highly encourage folks to download the report. We'll make sure the link is in the show notes. If you're someone that can hear links and remember them and put them into your browser, it's Aka.ms [Whack 00:27:21] Digital Defense.Sharon Xia:Yeah. What I wanted to say is it's very exciting that we are working on really this important area, and protecting our customers with machine learning technology, right? And there are lots of new areas, new territory we haven't explored. So I would really call for the community together to work with us and to innovate in this area, so our customers are better protected.Natalia Godyla:That's great. Yeah, it'll be a group effort. Well Sharon, thank you for joining us today. It's been great to hear about the progress we've made and the progress we are making in machine learning and security. So really appreciate you walking us through this and sharing the great work your team is doing.Sharon Xia:Thank you for the opportunity.Natalia Godyla:And now, let's meet an expert in the Microsoft security team to learn more about the diverse backgrounds and experiences of the humans creating AI and tech at Microsoft. Today, we're speaking with Emily Hacker. Thank you for being here, Emily.Emily Hacker:Thank you for having me.Natalia Godyla:Well, let's kick things off by just talking a little bit about your day job. So can you tell us your role at Microsoft and what your day-to-day looks like?Emily Hacker:Yeah, definitely. So I am a threat intelligence analyst on the TIGER team on Microsoft Defender. And I spend my days doing a variety of things. So specifically, I have a focus on email threats. So I gather a lot of information about email threats from open-source intelligence, from telemetry, from internal teams. And I combine all of these sources to try and find the email threats that are impacting our customers the most, and to put in proactive measures to stop those from impacting customers.Nic Fillingham:I want to know what the TIGER team is. What's a TIGER team?Emily Hacker:A TIGER team. It does stand for something, Threat Intelligence Global-Nic Fillingham:Is it a backronym? Were you all sitting in a room, and you're like "We need a cool name"?Emily Hacker:Oh, for sure. Definitely a backronym. It was definitely a backronym.Nic Fillingham:Someone's like "Tigers are cool"?Emily Hacker:Yeah, I feel very confident.Nic Fillingham:So you made it work.Emily Hacker:Yeah.Nic Fillingham:You made it work, but it's not necessarily memorable?Emily Hacker:No, we do have a lot of tiger imagery and logos and stuff related to our team now. And so we know what animal we are, but we might not know what we do.Natalia Godyla:I love that you guys went all in on it.Nic Fillingham:Are there any other teams based on animals of the Serengeti?Natalia Godyla:No, oh the Serengeti. So there's a fishing org that I've dotted a line to that we recently backronymed as well. And now it's Osprey, like the bird. So I'm like a member of the animal kingdom here.Nic Fillingham:Yeah, that's like a seagull, isn't it?Emily Hacker:I think they're pretty scary looking though. I think that was [crosstalk 00:30:27].Nic Fillingham:It's also the name of the big Marine helicopter I think in the British Navy.Emily Hacker:The helicopter, yeah. And that's what I usually think of first. I think it's the one, the helicopter that maybe folds up or something.Nic Fillingham:That's got the wings that fold out? Is that right? It's sort of like half a plane?Emily Hacker:Yep. Mm-hmm (affirmative).Nic Fillingham:It's like a VTOL, is it a VTOL?Emily Hacker:It's fancy looking for sure.Nic Fillingham:Got it. Well, this has been a great conversation. Thanks, we're done here. No, I think you were... I'm sorry, I derailed us by asking what TIGER stood for.Natalia Godyla:I was going to start with a rather broad question, so I'm glad we did TIGER first. So you spend your day-to-day on email threats. Do you see any patterns that... like to elucidate the audience on?Emily Hacker:So patterns, I mean we see a lot of different techniques and patterns and stuff that we're tracking for sure. I think with... We look at both malware threats being delivered by email, and we look at phishing, like credential theft, threats being delivered by email. And one of the things that I would say, maybe a pattern that I've noticed is that a lot of times the techniques that we see between the two are kind of different. So it's usually noticeable to us if we're looking at certain techniques that is definitely malware versus fishing.Emily Hacker:And then we've also recently expanded more of our deep dive into business email compromise, which often is completely wholly different from the other two types of threats that I just mentioned.Natalia Godyla:Can you describe why business email compromise is often treated wholly different? What is the distinction between that and the other two threats?Emily Hacker:Yeah, definitely. So business email compromises a lot of times is totally different from malware and phishing, because it won't contain any links or attachments. So it's totally social engineering based, which is interesting to me. Personally, I find it super interesting because it's basically just the quote unquote "Bad guys" if you will, tricking people into wiring them money.Emily Hacker:So when we're looking at malware threats, a lot of times they're going to use links or attachments that lead to obviously malicious code being downloaded onto the machine. And the emails themselves might be... We've seen completely blank emails. We've seen emails that use really generic lure, such as "Please do the attached invoice." Of course, the attached invoice is fake. And with phishing, similar we'll see lure such as... Actually we see a lot of they're like "Please join this Zoom call or this teams call or whatever."Emily Hacker:They're going to try and make the recipient click on the link. But with business email compromise, it's totally done in email. So the threat actor will just send an email. A lot of times they will either compromise as the name suggests, they will compromise one of the accounts of a individual who works at a victim company in accounting or wire transfers or that kind of job. And they will send emails from that account. Or another thing I've seen is they will have some kind of methodology of watching emails on a victim's email network. So either via some [o-off 00:33:27] phishing that they had done earlier, or perhaps they got credentials to the email inbox. But then when it actually comes time to send the malicious email, rather than using the user's email, they'll create one that looks almost identical, but just change a couple of characters.Emily Hacker:So they might register a domain. For example, if someone was trying to use my email address instead of "Microsoft.com", they might register "Micros0ft, with a zero.com", And then use my exact username. So to an unsuspecting victim, a reply to a thread will look exactly like it came from me, but then the malicious emails themselves aren't going to contain links or attachments. They're literally just going to be the bad guy saying like, "Hey, can you wire me these hundred thousand dollars or more, send it to this bank account?" And since there's already a level of trust with the victim, because it's usually coming either from a legitimate email account that they're used to doing business with, or one that's faked to look very similar to it, these are super successful.Emily Hacker:The people are wiring money to attack our accounts. And there's no malicious code involved. There's no phishing link involved, it's completely social engineering. Sorry, that was a really long answer. I got apparently really into that, sorry.Nic Fillingham:Emily, I wonder if you could tell us how you found your way to Microsoft. Have you been in security for a long time? What was path into your role and how did you find yourself in the security industry?Emily Hacker:Definitely. So it's definitely a bit of a roundabout interesting story. So it goes back a ways to when I first went to college, I guess. So I have a degree in English and communications and a minor in journalism. And I had every intention of being a newspaper reporter. I worked for my school's newspaper for a while. And then I worked for the city newspaper, for the city that I went to college in. And upon graduation, I decided maybe I wanted a job that had a little bit more normalcy. I really loved newspaper reporting, but it was a lot of late nights in the newsroom and stuff. So I ended up going into technical writing, and my first job out of college, I was actually writing software manuals. So it was pretty dry stuff, I'll admit. Where I was writing the manuals that people would refer to if they were having trouble.Emily Hacker:This was specifically for software for car dealerships, where the stuff I was writing was like "Press the F5 key to submit", or like that level of manuals, those very dry manuals. And I wasn't all that excited by that work. Some people love it and I understand why, but I didn't. So I was lucky that a girl that I had worked with at that job, I had only worked with her for a couple of months and she had gotten another job. Well, she contacted me about 10 months later and said that she had gotten promoted and wanted to hire me to backfill her. And she said it was a tech writing job, but it was totally different from the type of tech writing that we had been doing previously at the company. So I gave it a shot. I applied and I went to work with her.Emily Hacker:And what it was was I was actually the tech writer for a threat intelligence team at an oil and gas company, but it was my first foray into security. And it was not something I even knew was a thing honestly before, I didn't realize cybersecurity was kind of a field that people could work in. And it was very exciting to me. And I remember the first year or so that I worked there, everything was new and exciting, like "Oh my God, threat actors, what are those? This is so exciting. Nation States, Oh my God, this is a thing that's real." And it just all seemed like this movie script, except it was real. And after a bit of doing the editing and stuff for their reports, the reports that I was editing were very interesting to me. And I would ask questions because I needed to, to understand the report in order to edit it.Emily Hacker:But also just because I was legitimately interested, like "How did you do this analysis? What is this?" And I quickly decided I liked their job better than mine. So, I decided I was going to learn from my coworkers. And I am extremely lucky that the team of threat intelligence analysts that I was working with are some of the best people I've met in my life at that job and were super open to helping me learn. If I would say like "Hey, what are you working on? Can I kind of sit with you and learn from you?" Everyone was always just like "Yeah, let's do it, let me show you what I'm doing, blah, blah, blah." So I learned from them, and eventually, there was a time where we were a little short-staffed, as is common in security. And we were in charge of checking the phishing email inbox.Emily Hacker:So when users at the oil and gas client that I was working for would submit potentially suspicious emails, they would all go to an inbox that we had to analyze to determine if they were malicious or not. And it was a time-consuming job, and we just didn't have enough people on the team to do it and the rest of our work. So I kind of volunteered to help out. And that was how I got to learn how to do actual analysis. And I had job duties related to analysis. So I learned pretty much completely on the job from my coworkers. And then from there, I did that for about a year, maybe a little bit more after that. And I decided I wanted to move to Seattle, I was living in Texas during that.Emily Hacker:And I was very interested in living up here in the Pacific Northwest. So I left that job and got a job as a security researcher at a security vendor here in Seattle. So it gave me that other side of security that really allowed me to see the full picture of both having worked at a SOC, having worked at a vendor. And then I did that for just over a year. And this position at Microsoft opened up and I actually applied, Emily Hacker:I don't want to say as a joke, but I didn't think I was going to get the job.Nic Fillingham:As a stretch.Emily Hacker:Yes. It would be like if I applied to be president of the United States or something. It's one of those, where I'm like, "Oh, wouldn't that be great to submit the application," thinks never again about that moment. And then I was shocked to say the least when I got called for an interview and even more shocked when I got offered the job. So that was back in March. So I've only been here for a few months and I am loving it obviously so far. And what is really exciting to me is how this job is kind of, I get both the focus of having in-point telemetry like I did at my first job and phishing email telemetry. And then I also have a wider birth of just a lot of data and open source intelligence like I did it at my second job. And now I have them both here as well as getting to work with some of obviously the smartest people in the industry. So it was very exciting and I still am a bit amazed that I work here.Nic Fillingham:When you were writing manuals in for the car dealership and probably thinking about what was going to happen in the future, was there a little kernel, was there a little nugget of, it'd be awesome to be a company like Microsoft and doing cool nation state security, investigatory stuff?Emily Hacker:Absolutely not. I didn't even know that this was a job opportunity. The fact that this is a job that people do and now that I do. When I had first graduated and gotten my first job out of college, there was just so much about the world that I didn't know, but there was so much about careers that I didn't know. I didn't even know this was an option. And I do remember distinctly, I wasn't a huge fan of that job, but I didn't know what else was out there. And it just feels, everything's very overwhelming when you're 22 years old and you're like, "What is life like? Is this what I have to do forever?" So I'm just glad that I now know that this is an option.Nic Fillingham:What is life? Guess what? You keep asking that question. I'm afraid it's continually one you keep going back to. In a good way though. Do you find yourself bringing your technical writing skills, your formal sort of literature training? Do you find you're bringing that into this current role?Emily Hacker:Yes.Nic Fillingham:Are you writing a lot of reports and does that help you?Emily Hacker:Amazingly so much so that I think that this is something that people who work in technology don't always think about, but I work in threat intelligence and a large, extremely important facet of threat intelligence is communicating that intelligence to decision makers. If you know what's the intelligence but you're unable to communicate it, it's useless. So we write a lot of reports. I have a lot of those skills from my previous work. So writing a report is not difficult for me. It's something I've literally used to do for a living and knowing exactly how to phrase technical situations in a way that everybody, including non-technical people can understand is something I'm very good at because I have historically been a non-technical person. So it's something that is very useful to me.Emily Hacker:The other people who work on my team are also very good at it. But my point in that is that a lot of them have tech backgrounds. They have degrees or jobs where they have worked in technology. And so they have that tech skillset, but they have to learn the writing and communication on the job. And I have the writing communication and I had to learn the tech skill set on the job. And now all of us are good. We all do the job and we're all very good at it and we all have our things that we specialize in and we can help each other. But the point being when it comes to working in security or technology and hiring for security or technology, there's a large swath, if you will, of skillsets that are needed and nobody's going to have all of them for the most part. So finding people that have some of them, they can be trained up in the other ones, even if the ones that they're being trained up in are the technology ones.Nic Fillingham:Yeah. So have you found yourself in the same way that your colleagues were sort of helping you in the early days? Learn, fill in gaps, if you will, with you sort of being sort of somewhat new to the industry? Have the tables now turned? Are you now helping your colleagues be better communicators and helping them in their ability to pass this intelligence on into way that people understand?Emily Hacker:Yeah, I think so. So I definitely have edited a few of my colleagues reports before they went on to the formal editing process and just kind of taking the time to sit with them and be like, "This is what I'm changing and why." Either A, it's grammatically incorrect and let me explain to you what grammatically correct would be, or I'm saying this is unclear and we can make it more clear by saying this or this is too technical, only a handful of people reading this are going to know what this means and we need to simplify it to layman's terms. And I think people appreciate it. I hope. Either that or I'm like the red pen girl who just comes in and ruins everybody's reports and they're all terrified to see me coming. But I do think that they appreciate it.Nic Fillingham:What do you like to do Emily?Emily Hacker:Yeah, I do things.Nic Fillingham:Good answer.Emily Hacker:Okay. Believe it or not, I live in the Pacific Northwest, so I like hiking. I know. So does everybody in the entirety of the Pacific Northwest, but I actually really like hiking and that's why I moved here from Texas. So that's something that I greatly enjoy. I do things at home. Oh my God. I actually had made a list. This is sad. But at one time I made a list of things I do for fun, because when people ask this question, I always forget. I like writing. I did go to school to be a newspaper reporter. I still like writing. So it's my goal one day to get a novel published, but they may never come. And I play music. So I play several instruments and I like running. Do I like running? I run whether or not I like it. It's questionable.Nic Fillingham:Does anyone really like running?Emily Hacker:I don't think so.Natalia Godyla:I actually immediately want to ask what genre novel would you write?Emily Hacker:I think I would write a mystery, detective novel, because I'm really into true crime, which also everybody. But I like watching a lot of stuff about true crime, but then I'm also really... Am I admitting this? Probably. I'm also really into paranormal stuff and Big Foot and ghosts and what are they doing? And whether or not I believe in them, it's usually no, but they're interesting stories. And I feel like there's this very interesting intersection of detective stories and paranormal that is the X-Files, but could also be a novel one day. So let's just wait and see.Natalia Godyla:From your background, Emily, and your hobbies it seems you've got a lot of creativity either in writing or music. So what are your final thoughts on how creativity comes into play in the cybersecurity industry or in your day-to-day job?Emily Hacker:That's a really good question. And I think it's super important, especially in intelligence, which is all I can speak to because it's really all I've worked in in security. But one of the key aspects of working in threat intelligence is seeing a bunch of different data points. I might have a couple of data points here from open-source intelligence. I might see something weird on a machine and I might have an email and being able to connect the dots. And while that's not always something a machine can do, otherwise, we'd all been replaced by now. But it does require this level of creativity and this level of being able to remember, or kind of be like, "I wonder if I could connect this email to this thing that's happening with this machine."Emily Hacker:I was talking about detective novels earlier and I think that there's an aspect of that that kind of comes into play here too that's also an aspect of creativity, where you have to put the pieces together. You have to be able to see something once and then three days later when you have a malicious email in front of you be like, "Oh my God, this reminds me of this things from three days ago." There's also this level of creativity. I feel like that helps a lot of us. I was just talking about this with one of my coworkers yesterday, actually, about how one of the things that makes everyone on my team so successful, it is this level of, it's not by itself creativity, but I think it's an output for really creative people is this tenacity of when I see something I have to get to the bottom of it.Emily Hacker:And I think that I'm not just going to like run one query and be like, "Oh, computer told me it's X." I'm like, "But what is X? How do I get to the next part? What is it? How do I connect it to this Y over here? Do X and Y both connect over here to A maybe? Are they connected to as actor?" It's this level of just making a story out of the information that's presented to me that helps me, I feel like, be successful as an intelligence analyst. And I feel like there's a level of creativity to that that I honestly didn't think about until I've been in the industry for a while.Natalia Godyla:Yeah. I think you see a lot of unending curiosity with security folks as well. Like you said, as soon as you get one answer, it just opens up another question.Emily Hacker:Exactly.Nic Fillingham:So, Emily, you joined Microsoft in March of 2020, is that correct?Emily Hacker:Yes.Nic Fillingham:So you joined just as the mandatory work from home order was coming to place?Emily Hacker:Yeah. I've never ever been into the office.Nic Fillingham:Wow.Emily Hacker:Well, okay. I went into the office on day one to pick up my laptop and then went home, but I started after the work from home. So I've never met, well, I never met a lot of the people I work with in person. People always talk about the good old days of being on the office. Apparently there's a fridge that has bubbly water in it. One day I'll maybe drink some bubbly water.Nic Fillingham:It's a myth. It doesn't exist. We just tell that to people when they join the company and when they come in for the first time-Emily Hacker:Then they start and then they just make you work from home where you can buy your own bubbly water.Nic Fillingham:Yeah. Hey, where is this bubbly fridge? There's a fridge with bubbly water. No, it doesn't exist. You've been duped. So hang on. So I want to backtrack a bit because you talked about how you've got awesome colleagues and they've really helped you, so your experience completely through remote work.Emily Hacker:Yeah, it is.Nic Fillingham:So you've been able to join a new company, joined a new team, been supported and had sort of great experiences with colleagues through a hundred percent remote experience.Emily Hacker:Yep.Nic Fillingham:That's fascinating.Emily Hacker:I think one of the things that's been helpful is that there's a lot of new people on my team. So my team grew significantly around the time that I started. So me and another guy started on the same day and then four weeks later, another woman started and then over the summer we had two more people joined. And so we were in this together. And so it helped us. We all were in the same. It wasn't like everybody else knew each other and I was the new person, like, "Hey guys, let me join your conversation." We were all new. And so that helped a lot. But even the existing people on the team have been really, I don't know what word I'm trying to go for here, but they've been really open, I guess, to this remote work situation.Emily Hacker:The number of Teams calls, screen shares I've done where I'm just like, "Help. I don't understand what this means." And anybody I talk to is willing to sit on the other end of the Teams call and just walk me through what's happening. It has been honestly incredible. I'm really grateful for my team. I would like to go into the office one day, but I'd rather not be sick and I am glad that Microsoft is taking precautions. So considering the circumstances, things have definitely been going really well.Nic Fillingham:That's awesome. Well, Emily Hacker, thank you so much for being on Security Unlocked. We will work out how to send you a case of bubbly water.Emily Hacker:Thank you. Maybe then I won't go thirsty.Natalia Godyla:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us at msftsecurity, or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe.Natalia Godyla:Stay secure.