Security Unlocked

Share

Below the OS: UEFI Scanning in Defender

Ep. 24

All of us have seen – or at least, are familiar with – the antics of Tom and Jerry or Road Runner and Wile E. Coyote. In each one the coyote or the cat set up these elaborate plans to sabotage their foe, but time and time again, the nimble mouse and the speedy bird are able to outsmart their attackers.  


In our third episode discussing Ensuring Firmware Security, hosts Nic Fillingham and Natalia Godyla speak with Shweta Jha and Gowtham Reddy about developing the tools that allow for them to stay one step ahead of cybercriminals in the cat & mouse game that is cyber security.  

  

In this Episode You Will Learn: 

The new capabilities within Microsoft Defender to scan the Unified Extensible Firmware Interface (UEFI)

How the LoJax attack compromised UEFI firmware

How UEFI scanning emerged as a capability  


Some Questions that We Ask: 

Has UEFI scanning always been possible? 

What types of signals is UEFI scanning searching for? 

What are the ways bad actors may adjust to avoid UEFI scanning? 


Resources:  

Shweta Jha’s LinkedIn: 

https://www.linkedin.com/in/jhashweta/ 


Gowtham Reddy’s LinkedIn: 

https://www.linkedin.com/in/gowtham-animi/ 


Defender Blog Post: 

https://www.microsoft.com/security/blog/2020/06/17/uefi-scanner-brings-microsoft-defender-atp-protection-to-a-new-level/ 


Nic Fillingham’s LinkedIn: 

https://www.linkedin.com/in/nicfill/ 


Natalia Godyla’s LinkedIn: 

https://www.linkedin.com/in/nataliagodyla/ 


Transcript

[Full transcript can be found at https://aka.ms/SecurityUnlockedEp24]


Nic Fillingham:

Hello, and welcome to Security Unlocked, a new podcast from Microsoft where we unlock insights from the latest in news and research from across Microsoft's security, engineering, and operations teams. I'm Nic Fillingham-


Natalia Godyla:

And I'm Natalia Godyla. In each episode we'll discuss the latest stories from Microsoft Security, deep dive into the newest threat intel, research and data science.


Nic Fillingham:

And profile some of the fascinating people working on artificial intelligence in Microsoft Security.


Natalia Godyla:

And now, let's unlock the pod.


Natalia Godyla:

Hello Nic. Welcome to Episode 24. How's it going with you today?


Nic Fillingham:

Going well, thank you, Natalia. Yes, uh, welcome to you and welcome to listeners to Episode 24 of Security Unlocked. On today's podcast, we speak with Shweta Jha and Gowtham Reddy from the Microsoft Defender for Endpoint engineering team about capabilities in MDE to scan down into the UEFI layer. Now this is the third of three conversations we have that started back in Episode 11 with Nazmus Sakib where we talked about secure core PCs and, and firmware integrity. Then in Episode 14 we spoke with Peter Waxman about the Pluton processor and some of the new work that's happening there to imbed more security tech into sorta silicon onto the actual CPU die itself. And today we're sort of rounding that conversation out with Shweta and Gowtham to talk about how Microsoft Defender for Endpoint can now scan down or can scan down into the UEFI layer. You're gonna hear a bunch jargon, a bunch of technical terms like, I guess, UEFI. That's, we, we could start there.


Natalia Godyla:

Yes. And UEFI is the Unified Extensible Firmware Interface, so it is the software interface that lies between an operating system and firmware, and is an evolution of BIOS. And we'll also talk about MosaicRegressor which, for those of you that don't know, is the second ever UEFI rootkit which was discovered in 2020, but was used in an attack against NGOs in 2019. And, and for me, the timeline is shocking, second ever in the past year. Normally we hear about the continuous increase of a certain type of attack over the years, and here we're just at the second ever.


Nic Fillingham:

Yeah. It's a real interesting part of the conversation where we talk about the history of BIOS attacks, firmware attacks, UEFI attacks, and to learn that this has been sort of traditionally a pretty challenging area for attackers to, to breech and compromise. But, you know, Shweta and Gowtham have been, you know, very much ahead of the curve and, and being ahead of, of attackers in, in being able to develop these new capabilities to, from the operating system, scan down to the UEFI layer and look for malware, look for compromise. And it's a, it's a fascinating conversation. Again, it's sort of a completion of three episodes starting with Episode 11 and 14. So if you haven't listened to those, I recommend you add them to the queue. But I guess on with the pod.


Natalia Godyla:

On with the pod.


Nic Fillingham:

Welcome to the Security Unlocked podcast. Shweta Jha and Gowtham Reddy, welcome both of you. Thanks for being here.


Gowtham Reddy:

Thank you.


Shweta Jha:

Thank you so much for having us. We're so very excited.


Nic Fillingham:

I'm very excited, too. Now this is gonna be the third conversation in a sort of a mini series that we're running here on the podcast. We started with Nazmus Sakib who introduced us to the idea of secure core PCs and we talked about some of the challenges of firmware integrity and keeping firmware safe. Then we spoke Peter Waxman in another episode to learn about Pluton, the history of, of that technology and sort of what's coming for the Pluton processor. And today we're actually gonna talk about some new capabilities, or newish as of 2020, in Defender to scan down into the UEFI layer. Before we jump into to all that, let's just do some introductions for the audience. Shweta if we could start with you. Who are you? What is your role? What do you do day-to-day at Microsoft? Tell us, what you like the audience to know about you?


Shweta Jha:

Absolutely. Thank you, Nic. My name Shweta Jha. I am a program manager with Microsoft Defender for Endpoints, and I've been building security solutions features and products, and I'm super excited about it because security is the need today for our, uh, customers. And a few of the features that I built with my team were part of anti-tampering. Investment that we did, EDR block as part of be able to blocking and containment. And then we are gonna talk a lot about UEFI scanner. So pretty much around building security solution and features in this team and helping our customers.


Nic Fillingham:

Fantastic. And, and Gowtham, welcome to the podcast. If you could also introduce yourself. Uh, tell us about your role. What does your day-to-day look like?


Gowtham Reddy:

Hi. This is Gowtham Reddy. I'm an engineering manager in Microsoft Defender, uh, Endpoint. So before engineering manager, so I was working as an engineer in the same team for last six years. So I work on, uh, many of the rootkit technologies, the Defender, uh, has and, uh, the remediation technologies to remediate many of the malwares that are present on the system. I have been where I working on this fantastic team, developing like durable protection features that were, and compliment the ever changing malware fields.


Nic Fillingham:

That's great. So, again, welcome to both of you. Thanks for your time. One of the things we do on the, uh, Security Unlocked podcast here is we, we don't necessarily cover the latest announcements. We, we sort of look back over the last sort of three to six months for interesting sort of technology, interesting advancements in, in security technology, and we bring experts on to, to talk about these new features and capabilities after them sort of being in the wild. Today we're talking about the UEFI scanning capabilities that are in Microsoft Defender, and there's a blog post that, that both of you helped author back in, in June of 2020, which feels like a decade ago, but I guess it's more like six or seven months. So I wondered if one of you might be able to just walk us through. What was that announcement made in that blog post? What was sort of the news? And then I think maybe if the other one or maybe just following, I'll, I'll leave it to how we, how we split this up. But what was announced back in June? And sort of what's happened since then? How have those new capabilities sort of rolled out and what are we seeing with customers actually using them?


Shweta Jha:

So I, I guess I can get us started, and then I'll hand it over to Gowtham definitely to talk more on the technical details and the, the attacks that we see in the wild, and that's why we kind of built this UEFI scanner. So as you understand, this is a journey, right? We built a layered defense in that security solutions. And when we build any security solution, we need to make sure that we take a holistic approach. So if you look at the operating level of security solutions, we've been getting pretty great at operating level security solutions. And it's not only Microsoft. If you see other security providers as well, they have been doing great, too.


Shweta Jha:

So what does that mean? It means that because the operating system level security solution is really great, it does making difficult for attackers to not get detected at that level. It's a constant battle, so they have been looking into other means where they can go into the system undetected, and that's where if you look at the data, you would find that in recent past the attacks across hardware and firmware level has been on the rise. So we built UEFI scanner keeping in mind that we should be able to detect those type of attacks, because those type of attacks are not only very dangerous, but often time they are not detected. They persist even if you reboot the system. So the nature of these type of attacks is very dangerous, and keeping that in mind, we decided to build UEFI scanner.


Gowtham Reddy:

So I can add like why we did, uh, build the UEFI scanner. So because of the operating system security features that Microsoft is constantly working on, the bad guys are trying to go in, down and down in the layered architecture. And so some of the traits of the ia64 went onto the BIOS, tampering the BIOS and, uh, tampering the MBR, the master board required and, uh, VBR based bootkits. So Defender has evolved into that space of counting the MBR and, uh, detecting the bootkits and the boot time.


Gowtham Reddy:

So as a logical evolution the bad guys are, uh, from the stage of Colonel to the MBR, MBR to the UEFI. So we were anticipating that this kind of evolution is quite possible and the UEFI implants were not very far. So that's the time we found the first UEFI implant called LoJax. So that was a triggering point when we completely committed to ourselves to expand our root kit technology, to detect any kind of rotates presence in the UV. So that was our core idea of expanding or rotating to the layer much below the operating system. So there were some challenges


Natalia Godyla:

If you don't mind me jumping in, I had a question around that. So...


Gowtham Reddy:

Mm-hmm (affirmative)


Natalia Godyla:

... the way you're framing it is that when we started to notice the threat landscape moved to this layer, we decided to invest in this type of technology. What about the technology itself? Had there always been this opportunity to tackle UEFI scanning, or is there something new that we're leveraging in order to solve this problem Now that might not have been around beforehand?


Gowtham Reddy:

That's a good question. So there was always a chance to exploit the UEFI, but it's about the timing of the attackers to get at target this space because the rest of the platform and ecosystem is getting more and more secure. So the UEFI is not new. So it was there a decade ago, but the implants are new because of the advances in the operating system.


Nic Fillingham:

So Gowtham, tell us about the LoJax attack that happened. Was it the first or it was one of the first detected compromises of the UEFI firmware? Can you tell us some more about, about that? If folks aren't familiar with it like me?


Gowtham Reddy:

Mm-hmm (affirmative).So that definitely some theoretical researcher driven, before the LoJax, but the LoJax is a fast known exploitation instance where we know we found it in the wild. It is quite possible even before that a UEFI implant demonstrated in many of the black hat conferences, but those are theoretical in nature. So the research had access to the device and they demonstrated it. But LoJax's is the one where from operating system level. So a particular malware, I would say it as a root kit, which has tried to intrude from kernel mode to the UEFI, and they have installed a UEFI driver. So if we consider the operating system as a drivers, even the firmware itself had some drivers. So they were able to install a driver which actually in turn drops the another kernel mode driver, advanced operating system boots up. it's about the boot sequence.


Gowtham Reddy:

So first the firmware starts running and it initializes all the system, and then it invokes operating system. So in the LoJax's case, after the firmware is completed, it has already dropped the kernel driver on the operating system, if it is not present. So that means by the end of the firmware sequence, so we have a presence of a kernel driver. And when that kernel driver starts, that is a user mode, malware starts, kicks in. So this keeps repeating even after you were re-install the wares, even if you change the hard disc, the same pattern will be fought. So that's how the LoJax's type work.


Nic Fillingham:

And I wonder, do we know, what was the breakthrough that made LoJax possible? UEFI has been around for a while. UEFI for probably predates LoJax. And obviously before UEFI, there was sort of the more standard sort of BIOS that probably most folks are familiar with. Can we talk a little bit more about how LoJax came about and sort of what maybe changed or what the breakthrough was on the attacker side?


Gowtham Reddy:

I would say that there were a couple of open source read-write drivers, which has a capability to access the firmware, using a special interface called SPI. SPI is a something called serial peripheral interest. So using the serial peripheral interface, any kernel driver can instruct the platform hardware layer to read and write any content in the flash. So I think like many of the security industry knows a driver called a read drive, everything, they call it as RWE. So this is the driver using which anybody can read any offset, any device memory, and write. I think this is, the prevalence of this kind of open source tools might be help attackers to develop this kind of ecosystem of all the sequence of the malware, the root kits.


Shweta Jha:

In addition to what Gowtham said, definitely the work that researchers were doing in this space, it always starts with researcher trying to do something and then attackers trying to find other means. So here are the things. Attackers usually do exploit things that are not done in a right way. So in this case, for example, if there are certain configuration that you need to, or your partner needs to make sure that those are in place, for example, rewrite where you are not providing writing access, just the reading access, and so on.


Shweta Jha:

So typically in all these type of attacks would see that misconfigured devices are exploited the most, and that misconfiguration happens at the time when the devices are getting built. So that is another factor why these attacks are very successful, because there are misconfigured devices, because while building the devices, somebody messed to configure it and right way. And if you look at the journey, that's where you have a secure core PC, which is designed be secured, making sure that the things that are needed to protect the computer against these types of attacks that are there out from the first day.


Natalia Godyla:

So my question is about the application of this new technology. So I really appreciate you walking through that attacker workflow. So what type of signals is UEFI scanning, looking for? What is it using to enrich the context of the existing end point data?


Gowtham Reddy:

That's a very good question. So basically the level of details that UEFI scanner can get is enormous. So this is the area where like the defender has a content scanning. So, uh, we have, uh, extended our content scanning to every file that is present inside the firmware. So this help the defender research to write any kind of content scanning signatures to detect any bad content. So that means in this case, if research knows any implant, so we have a capability to scan the 600 million devices to know if any of our customers have impacted with the specified malicious file.


Gowtham Reddy:

And this is just one part of our UEFI scanner. And the other part of it is detecting any anomalous behavior inside the firmware. For example, in many of the supply chain attacks like Solarigate, it's quite possible that some of the OEMs channels were compromised and the deliver the firmware updates with the malicious modules in it.


Gowtham Reddy:

So in this case, our UEFI scanner collects all the metadata about the new for- firmware update and we run heavy amount models in our cloud. And that will tell us if there is an unknown anomaly that exists in this particular firmware update. Instead of a known malware implant so that the UEFI scanner has the two capabilities. One is detecting a known malicious implant, and the other one is anomalous from where presence of a fax. So in this case, we act both ways.


Nic Fillingham:

What does an anomaly look like in this context?


Gowtham Reddy:

Anomalies look like, for example, if you have a firmware is a, firmware is a file system, like a typical drive. A presence of an driver file, probably a hedge P driver file or an unsigned driver file. On a Dell OEM is constrained to the anomaly. Because we have trained the model of all the known Dell firmwares with them, a ML model. So any new image with the unexpected file, it will be immediately flagged.


Nic Fillingham:

And why is ML the sort of approach you've taken here versus sort of heuristics? I would have thought that there's a pretty limited set of content. They could make up sort of firmware and firmware instructions. Obviously, I don't know anything about this space, so I'll caveat that there, but, um, could you talk about why ML versus heuristics versus something else?


Gowtham Reddy:

In the days of, uh, BIOS, so you are a expectation was right. The bast consists of a series of micro code,


Gowtham Reddy:

... which is, uh, very limited. And, uh, in the context of UEFI, you have a full file system, uh, which has, like, uh, thousands of files; individual files. And, uh, this causes... Uh, creates, uh, basically a huge amount of, uh, the vectors space, which to scan or to collect the metadata.


Gowtham Reddy:

So it's not just simple collection of mecra- microcodes. It contains the drivers, it contains the services, it contains a lot of other things. It's a file system like NTFS.


Nic Fillingham:

Got it. So because UEFI is, as you say, a file system as opposed to... What was BIOS? BIOS was not a file system? BIOS was, uh, sort of a discreet, sort of, low level executable?


Gowtham Reddy:

Yeah, i- i- it is just a sequence of, uh, microcode instructions that will be run on the firmware. So basically, i- it has a s- uh, fi- se- set of microcodes.


Nic Fillingham:

So the machine learning models that you reference, w- where are they running? Are some of them running locally? Are they all running in the Cloud? Is it a mixture of the two?


Gowtham Reddy:

They're all running in the Cloud for now. So we have MDATP Cloud services where we run all this clo- uh, demo models. So our models are really very effective. So recently, we got in, uh, so- so, uh, the UEFI alert by, uh, mal- model. Apparently, it's a kind of, um, true positive because, um, there was a Microsoft engineer who was working on a hardware space.


Gowtham Reddy:

So he take, uh, firmware. And he kept a developer driver and he flashed on his own device. And, uh, our UEFI scanner immediately caught it and we... the security administrator got an alert and there was an investigation happen. So we are pretty ready to catch any kind of such things now.


Natalia Godyla:

So we all know it's a cat and mouse game with the threat actors. So what is the team anticipating in terms of how the actors will adjust their processes to evade this new UEFI scanning technology?


Gowtham Reddy:

That's a good question. We're trying to validate something in a- a lower level of trust, the lower level of ring other than the kernel. So definitely, there is a chance that attacker can modify the firmware presence. Uh, he can spoof the content when defender tries to scan. So this is quite, uh, possible. But we are already working on mitigating that kind of an attacks.


Nic Fillingham:

So now that this feature, these capabilities, have been live in the product for, uh, I guess over six months at this point, w- what have you learnt? What have you seen in the telemetry? What have you seen in the types of attacks and, I guess, even sort of false positives that have- have come through from- from this new, uh, capability?


Gowtham Reddy:

Uh, that's a very good question. So we learnt a lot of things. The UEFI file system has never scanned before. So we got some false positives on the content that we scan but we immediately fine-tuned our signatures.


Gowtham Reddy:

Back in... Six months before, when we published a blog, we only know the first UEFI known implant called LoJax but often we share... There was a second implant called Public. That's called MosaicRegressor and our UEFI scanner has well detected the MosaicRegressor implant. Uh, the- the telemetry count was small. So we did, uh, able to detect the mi- MosaicRegressor.


Nic Fillingham:

So in this first six months, as well as the LoJax campaign, uh, what's the taxonomy here? How do we f- refer to it?


Gowtham Reddy:

Uh, we can consider... W- we are, uh, tracking them as an UEFI implant malware or UEFI rootkit. So this is the category we are looking at. So right now, we have, uh, LoJax and we have a MosaicRegressor as, uh, two big families in this space.


Nic Fillingham:

Big families. Got it.


Shweta Jha:

Yeah, about MosaicRegressor, I wanted to add a little bit more just to complement what, uh, Gowtham mentioned, how powerful this tool is. And how powerful this particular feature is. So if you read through the MosaicRegressor, uh, breach, it was a nationwide targeted attack.


Shweta Jha:

This was targeted for diplomats. And this attack, as Gowtham described, first they would insert one module. Uh, that one module would get undetected and then that module would try to do other stuff, like try to, uh, get in touch with command and control and get another, uh, module and so on.


Shweta Jha:

So the entire c- chain is so very interesting. And I'm glad that we built this feature and we were able to detect it because it's so powerful. Most of the security solution, they're not able to detect because they don't have this, uh, such great capabilities.


Shweta Jha:

But look at the way this attack was carried. It was pretty much targeted, pretty much nationwide for a few countries, originated from one country. So the sophistication level in the nature itself speaks for it and I'm glad that we, as in our product, we have this capability which can even, you know, unknown, first seen, it can detect those type of attacks as well.


Natalia Godyla:

In the process of developing this new technology, where were there false starts? What techniques did you try but didn't work to solve this problem?


Shweta Jha:

Little bit on the journey, right? We have been working on it. Um, so Gowtham explained about how we have rootkit, bootkit level and then we went to the UEFI site and we had to be extremely careful because it's, like, uh, it has a high integrity and high severity of going wrong.


Shweta Jha:

So we had to be very careful making sure that the running system is not damaged and at this point, I'll hand it over to Gowtham because he can explain, in detail, each and every pieces that we took into consideration to making sure that our customers' device remain intact. So go ahead Gowtham.


Gowtham Reddy:

Yeah. Thanks Shweta. So, uh, we have indeed explored, uh, many mechanisms like accessing the PCI space from the operating system itself, which we didn't continue to proceed because of some of the pushback from the kernel team to update the haul.


Gowtham Reddy:

So actually, uh, to accessing any peripheral device from the PCI bus, there are a couple of complications because the peripherals have, uh, specific implementation of Reads and Writes, the bus Reads and Writes. So, uh, the approach we took was, uh, using the SBI interface, which is pretty much, kind of, an, uh, universal interface which is developed by Motorola by a long time ago.


Gowtham Reddy:

So luckily, what worked in our favor was most of the Intel p- s- uh, chipsets, they support the SBI based access. So they support the SBI, uh, using which we can use the memory map mechanisms to access the PCI space.


Gowtham Reddy:

So basically, here, what happened was instead of directly using the hardware primitives, we used, uh, software primitives because the chipsets are well supporting the SBI interface. So that's how we landed in our approach.


Nic Fillingham:

I wanted to circle back to the use of machine learning here in- in solving this problem. How big are the signal sets that you're getting to train the model? How big is the model?


Nic Fillingham:

Is the model that you use here, to detect anomalies in the firmware layer, is it as sophisticated and large as something as, like, looking for malware on endpoints? Or are we talking, like, a much sort of smaller more, sort of, n- nuance. No, that's not the right word. Sort of a smaller bespoke model?


Gowtham Reddy:

Uh, I can take that question. So u- usually, uh, in the endpoint when- when applying the malware, um, in machine learning models, we heavily focus on the individual file properties, like file headers, file footers and some file p- properties and so on.


Gowtham Reddy:

But UEFI case, we built a brand new machine learning model based on the properties of the UEFI image itself. So thanks to David, from our MDATP team. So he come up with a model where... which takes input signals as specific to the UEFI firmware image.


Gowtham Reddy:

To give some examples, each firmware drive has a lot of GUIDs, called firmware GUIDs. And then they have some properties called, uh, file types and properties. Every property that we took was specific to the firmware. So they are not generic to the specific malware files that we see regular malware detections. So these are highly tailored to the signals from the UEFI firmware image.


Nic Fillingham:

And were you able to reuse some of the anomaly detection


Nic Fillingham:

Algorithms are purchased from other parts of the defender engineering org, or did you have to sort of build a brand new model and a brand new way to detect anomalies?


Shweta Jha:

Yeah. So, we definitely used our existing infrastructure. So, as you know? Uh, we have a massive backend system where we get tons of signals and we run tons and tons of AI and ML model to detect the anomalies and to detect the new trends and so on. So, as Gordon was talking, for this particular UV, AI and ML model, even though where we had to tweak it to make sure that we capture the inputs that are UV specific, the models were used, the pipeline to collect the data that were used and the channel where we surface it to our customers. So, if you look at the end to end story, the way we do things are we detect, we remediate, and we also notify to our SecOps that, "Hey, these are the things that happened in your environment." And that goes in the form of alerts or incidents and so on. So, we used exactly same infrastructure, same pipeline, but specific to UV.


Natalia Godyla:

So, I know a little earlier in this episode, we talked about the learnings after being in market. What about the impact to SecOps teams? Do we have any early numbers to talk through about what this has raised for our customers?


Shweta Jha:

That's a great question. We do see here and there, though the number is not pretty high on the implant, but we do see in numbers there, like, as Gordon mentioned about a mosaic regression. We did find that and there are few others also. But I think the most important aspect of this unique feature is that, just a little bit forget about this feature and see that today's world, today, there is no UV scanner, the security admins or SecOps, they, they don't know what is happening at this level. They have tons of device in their organization. And these devices are at this level is completely black box for them, because they don't know whether it is configured well. They don't know if there are implants there. They don't know if there are vulnerabilities that could be exploited.


Shweta Jha:

So, there's the power of this UV scanner. One is, you know, so we, we built a solution keeping in mind that we will not only detect, we will bring these, these things where they don't have visibility today to understand what is going on. So, the focus area, and then the objective that we have is to detect the implant, either using the heuristic detection or the AI, ML but also read through each and every configuration that are happening at this level and the vulnerabilities that exist at this level and bring that to the, SecOps attention, so that when they look at it, they can take appropriate action to remediate it. So, that's the next step. And that is the work right now, we are currently doing. We do not have, in the form of report, we do see it in our data and we want to make sure that these are available to our SecOps. But just to tell you, there are tons and tons of misconfigured device out there. And it's, it's a little tricky.


Gowtham Reddy:

To add more about the misconfiguration. So, it's about like the PC settings, like a UV, the BIOS read-write or whatever the settings we'll use to see in when we go to the BIOS in the past. So, the UV must be configured well to support the secure boot, to use the TPM and to use any of the hardware provided features, it must be configured well. If it is misconfigured, you won't get any protection. So, if you have a helmet in your backseat, when you are driving, it won't help you. So, you had to keep it on your head.


Shweta Jha:

(laughs). That's a great analogy.


Nic Fillingham:

That leads us to, what is the guidance here for Sec admins and security teams out there? How do they enable this functionality? Is it on by default in, in certain places? What do we need to do to make sure that, that customers are getting the full protection from this capability?


Shweta Jha:

So, uh, this, this feature is enabled by default on all the devices. Um, we made sure that this is available. And the great news is that it is not only, you know, Windows 10, it is available for servers, download as well. So, that's the power that we have in our solution. Ultimately, if you look at what is the future that are gonna look like, secure core PC is the future we should be heading towards. But because enterprises and customers are not there yet, uh, we have UV scanner to compliment it. The other thing, if we have to talk about the futuristic roadmap, right now, we built the scanner for UV, but there are other network devices like network adapter and things like that. There is a scope to extend these types of capability to those devices as well, because those, there is a possibility to get those devices exploited too. So, that's something we are considering to work through.


Nic Fillingham:

Got it. So, just to confirm there, so, this new capability is on by default in any device that is being protected by the defender service. Is, is it, is it as simple as that or is there sort of more to it?


Shweta Jha:

Yes. Any device which is having defender antivirus running.


Natalia Godyla:

Thank you for that. That was super helpful. And thank you both for joining us on the show today.


Shweta Jha:

Thank you, Natalia. It was pleasure to be here and talking with our customers. Thank you so much for hosting us.


Gowtham Reddy:

Thank you Natalia and Nick for hosting us. So, it's been wonderful time talking to you about UV scanner. Thank you so much.


Nic Fillingham:

Thank you both for your time. Thanks for bringing great innovation to the security space.


Shweta Jha:

Absolutely. It's a constant journey and we're on it.


Natalia Godyla:

Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.


Nic Fillingham:

And don't forget to tweet us @msftsecurity or email us @securityunlocked@microsoft.com with topics you'd like to hear on a future episode. Until then, stay safe.


Natalia Godyla:

Stay secure.

More Episodes

5/5/2021

Ready or Not, Here A.I. Come!

Ep. 26
Remember the goodoledays when wespent youthfulhours playing hide and seek with our friends in the park?Wellit turns out that game of hide and seek isn’t just for humans anymore.Researchers have begunputting A.I. to the test by having it play this favorite childhood gameover and overandhavingthe softwareoptimize its strategiesthrough automated reinforcement training.In today’s episode,hosts Nic Fillingham and Natalia Godyla speak with Christian Seifert and Joshua Neil about their blog postGamifying machine learning for stronger security and AI models,and how Microsoft is releasing this new open-sourcedcode to help it learn and grow.In This Episode, You Will Learn:What is Microsoft’sCyberBattleSim?What reinforcement learning is and how it is used in training A.I.How theOpenAIGym allowed for AI to be trained and rewarded for learningSome Questions We Ask:Is an A.I. threat actor science fiction or an incoming reality?What are the next steps in training the A.I.?WhowastheCyberBattleSimcreated for?ResourcesOpenAIHide and Seek:OpenAIPlays Hide and Seek…and BreaksTheGame! 🤖Joshua and Christian’sblog post:Gamifying Machine Learning for Stronger Security and AI ModelsChristian Seifert’sLinkedIn:https://www.linkedin.com/in/christian-seifert-phd-6080b51/Joshua Neil’sLinkedIn:https://www.linkedin.com/in/josh-neil/NicFillingham’sLinkedIn:https://www.linkedin.com/in/nicfill/NataliaGodyla’sLinkedIn:https://www.linkedin.com/in/nataliagodyla/Microsoft Security Blog: https://www.microsoft.com/security/blog/Transcript[Full transcript at https://aka.ms/securityunlockedep26]Nic Filingham:Hello and welcome to Security Unlocked! A new podcast from Microsoft where we unlock insights from the latest in news and research from across Microsoft Security Engineering and Operations Teams. I'm Nic Filingham.Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft Security. Deep dive into the newest threat intel, research, and data science.Nic Filingham:And profile some of the fascinating people working on artificial intelligence in Microsoft Security.Natalia Godyla:And now, let's unlock the pod.Nic Filingham:Hello, Natalia! Hello, listeners! Welcome to episode 26 of Security Unlocked. Natalia, how are you?Natalia Godyla:Thank you, Nic. And welcome to all our listeners for another episode of Security Unlocked. Today, we are chatting about gamifying machine learning, super cool, and we are joined by Christian Seifert and Joshua Neil who will share their research on building CyberBattleSim, which investigates how autonomous agents operate in a simulated enterprise environment by using high-level obstruction of computer networks and cyber-security concepts. I sounded very legit, but I did just read that directly from the blog. Nic Filingham:I was very impressed.Natalia Godyla:(laughs)Nic Filingham:If you had not said that you read that from the blog, I would've been like, "Wow". I would to like to subscribe to a newsletter. Natalia Godyla:(laughs)Nic Filingham:But this is a great conversation with, with Christian and Joshua. We talked about what is reinforcement learning. Sort of as a concept and how does that gonna apply to security. Josh and Christian also walked us through sort of why this project was created and it's really to try and get ahead of a future where, you know, malicious actors have access to some level of automated, autonomous tooling. Uh, and so, this is a new project to sort of see what a future might look like when there all these autonomous agents out there doing bad stuff in the cyber world.Natalia Godyla:And there are predecessors to this work, at least in other domains. So, they used a toolkit, a Python-based Open AI Gym interface to build this research project but there have been other applications in the past. OpenAI is, uh, well-known for a hide-and-seek. There is a video on YouTube that shows how the AI learned over time different ways to obstruct the agent and the simulated environment. Things like, blocking them off using some pieces of the wall or jumping over the wall.Nic Filingham:The only thing we should point out is that this CyberBattleSim is an open source project. It's up on GitHub and attained very much want researchers, and really anyone who's interested in this space to go and download it, go and run it, play around with it, and help make it better. And if you have feedback, let us know. There is contact information, uh, through the GitHub page but you can also contact us at Security Unlocked at Microsoft dot com and we can make sure you, uh, get in contact with the team. And with that, on with the pod?Natalia Godyla:On with the pod!Nic Filingham:Welcome to Security Unlocked, new guest, Christian Seifert. Thanks for joining us and welcome returning guest, Josh Neil, back to the podcast. Both of you, welcome. Thanks for being on Security Unlocked.Christian Seifert:Thanks for having us!Joshua Neil:And thanks, Nic.Nic Filingham:Christian, I think as a, as a new guest on the podcast, could we get a little introduction for our listeners? Tell us about, uh, what you do at Microsoft. Tell us about what a day to day look like for you.Christian Seifert:Sure, so I'm a, uh, research lead on the Security and Compliance team. So our overall research team supports a broad range of enterprising consumer products and services in the security space. My team in particular is focused on protecting users from a social engineering attack. So, uh, think of, like, fishing mails for instance. So we're supporting Microsoft Defender for Office and, um, Microsoft Edge browser.Nic Filingham:Got it, and Josh, folks are obviously familiar with you from previous episodes but a, a quick re-intro would be great. Joshua Neil:Thanks. I currently lead the Data Science team supporting Microsoft threat experts, which is our managed hunting service, as well as helping general res... cyber security research for the team.Nic Filingham:Fantastic, uh, again, thank you both for your time. So, today in the podcast, we're gonna talk about a blog post that came out earlier in this month, on April 8, called Gamifying Machine Learning for Stronger Security in AI Models, where you talk about a new project that has sort of just gone live called CyberBattleSim. First off, congratulations on maybe the coolest name? For, uh, sort of a security research project? So, like, I think, you know, just hats off there. I don't who came up with the name but, but great job on that. Second of all, you know, Christian if, if I could start with you, could you give us a sort of an introduction or an overview what is CyberBattleSim and what is discussed in this blog post?Christian Seifert:As I... before talking about the, the simulator, uh, the... let me, let me kind of take a step back and first talk about what we tried to accomplish here and, and why. So, if you think about the security space and, and machine learning in particular, a large portion of machine-learning systems utilized supervised, uh, classifiers. And here, essentially, what we have is, is kinda a labeled data set. So, uh, for example, a set of mails that we label as fish and good. And then, we extract, uh, threat-relevant features. Think of, like, maybe particular words in the body, or header values we believe that are well-suited to differentiate bad mails from good mails. And then our classifiers able to generalize and able to classify new mails that come in. Christian Seifert:There's a few, uh, aspects to consider here. So, first of all, the classifier generalizes based on the data that we present to it. So, it's not able to identify completely unknown mails. Christian Seifert:Second, is that usually a supervised classification approach is, is biased because we are programming, essentially, the, the classifier and what it, uh, should do. And we're utilizing domain expertise, red teaming to kind of figure out what our threat-relevant features, and so there's bias in that. Christian Seifert:And third, a classifier of who has needs to have the data in order to make an appropriate classification. So, if I have classifier that classifies fish mail based on the, the content of the mail but there is the threat-relevant features are in the header, then that classifier needs to have those values as well in order to make that classification. And so, my point is these classifiers are not well-suited to uncover the unknown unknowns. Anything that it has not seen, kinda new type of attack, it is really blind to it. It generalizes over data that, that we present to it. Christian Seifert:And so, what we try to do is to build a system that is able to uncover unknown attacks with the ultimate goal then to, of course, develop autonomous defensive component to defend against those attacks. So, that gives it a little bit of context on why we're pursuing this effort. And this was inspired by reinforcement learning research and the broader research community, mostly that is currently applied kinda in the gaming context. Christian Seifert:So OpenAI actually came out with a neat video a couple of years ago called Hide and Seek. Uh, that video is available on YouTube. I certainly encourage listeners to check it out, but basically it was a game of laser tag where you had a kinda, uh, a red team and a blue team, uh, play the game of laser tag against each other. And at first they, of course, randomly kind of shoot in the air and run around and there is really no order to the chaos. But eventually, that system learned that, “Hey, if a red team member shoots a blue team member, there's a reward.” and the blue team member also learned while running away from the red team member is, is probably a good thing to do. Christian Seifert:And so, OpenAI kinda, uh, established the system and had the blue team and the red team play against each other, and eventually what that led to is really neat strategies that you and I probably wouldn't have come up with. 'Cause what the AI system does, it explores the entire possible actions base and as result comes up with some unexpected strategies. So for instance, uh, there was a blue team member that kinda hid in a room and then a red team guy figured, “Hey, if I jump on a block then I can surf in that environment and get into the room and shoot the blue team member”. So that was a little bit an inspiration because we wanted to also uncover these unknown Christian Seifert:Unknownst in the security context.Nic Filingham:Got it. That's great context. Thank you Christian. I think I have seen that video, is that the one where one of the many unexpected outcomes was the, like, one of the, the, blue or red team players, like, managed to sort of, like, pick up walls and used them as shields and then create ramps to get into, like, hidden parts of the map? Uh, uh, am I thinking about the right video? Christian Seifert:Yes, that's the right video. Nic Filingham:Got it. So the whole idea was that that was an experiment in, in understanding how finding the unknown unknowns, using this game, sort of, this lazar tag, sort of, gaming space. Is, is that accurate?Christian Seifert:That's right, and so, they utilized reinforcement learning in order to train those agent. Another example is, uh, DeepMind's AlphaGo Zero, playing the game of Go, and, and here, again, kind of, two players, two AI systems that play against each other, and, over time, really develop new strategies on how to play the game of Go that, you know, humans players have, have not come up with. Christian Seifert:And it, eventually, lead to a system that achieved superhuman performance and able to beat the champion, Lisa Dole, and I think that was back in 2017. So, really inspiring work, both by OpenAI and DeepMind.Nic Filingham:Got it. I wonder, Josh, is there anything you'd like to- before we, sort of, jump into the content of the blog and, and CyberBattleSim, is there anything you'd like to add from your perspective to, to the context that Christian set us up on? Joshua Neil:Yeah. Thanks, Nic. I, I mean, I think we were really excited about this because... I think we all think this is a natural evolution of, of our adversaries, so, so, currently, our adversaries, the more sophisticated ones, are primarily using humans to attack our enterprises and, that means they're slow and they can make mistakes and they don't learn from the large amount of data that's there in terms of how to do attacks better, because they're humans.Joshua Neil:But I think it's natural, and we just see this, uh, everywhere and, all of technology is that people are bringing in, you know, methods to learn from the data and make decisions automatically, and it's- so it's a natural evolution to say that attackers will be writing code to create autonomous attack capabilities that learn while they're in the enterprise, that piece of software that's launched against the enterprise as an attack, will observe its environment and make decisions on the fly, automatically, from code. Joshua Neil:As a result, that's a frightening proposition because, I think the speed at which these attacks will proceed will be a lot, you know, a lot more quick, but also, being able to use the data to learn effective techniques that get around defenses, you know, we just see data science and machine learning and artificial intelligence doing this all over the place and it's very effective that the ability to consume a large amount of data and make decisions on it, that's what machine learning is all about. And so, we at Microsoft are interested in exploring this ourselves because we feel like the threat is coming and, well, let's get ahead of it, right? Let's go experiment with automated learning methods for attacks and, and obviously, in the end, for defense that, by implementing attack methods that learn, we then can implement defensive methods that will, that will preempt what the real adversaries are doing, eventually, against our customers.Joshua Neil:So, I think that's, sort of, a philosophical thing. And then, uh, I love the OpenAI Hide-and-Seek example because, you know, the analogy is; Imagine that instead of, they're in a room with, um, walls and, and stuff, they're on a computer network, and the computer network has machines, it has applications, it has email accounts, it has users, it's got a cloud applications, but, in the end, you know, an attacker is moving through an environment, getting blocked in various ways by defenses, learning about those blockings and detections and things and finding gaps that they can move through in, in very similar ways. So, I just, sort of, drawing that analogy back, Hide-and-Seek, it is what we're trying to do in cyber defense, you know, is, is Hide-and-Seek. And so the, I think the analogy is very strong.Nic Filingham:Josh, I just wanna quickly clarify on something that, that you said there. So, it sounds like what you're saying is that, while, sort of, automated AI-based attacking, attackers or attacking agents maybe aren't quite prevalent yet, they're, they're coming, and so, a big part of this work is about prepping for that and getting ahead of it. Is, is, is that correct?Joshua Neil:That's correct. I, I'm not aware of sophisticated attack machinery that's being launched against our enter- our customers yet. I haven't seen it, maybe others have. I think it's a natural thing, it's coming, and we better be ready.Christian Seifert:I mean, we , we see some of it already, uh, in terms of adversarial machine learning, where, uh, our machine learning systems are getting attacked, where, maybe the input is manipulated in a way that leads to a misclassification. Most of that is, is currently more, being explored in the research community.Natalia Godyla:How did you apply reinforcement learning? How did you build BattleSim? In the blog you described mapping, some of the core concepts of reinforcement learning to CyberBattleSIm, such as the environment, that action space, the observation space and the reward. Can you talk us through how you translated that to security?Christian Seifert:Yeah. So, so first let, let me talk about reinforcement learning to make sure, uh, listeners understand, kinda, how that works. So, as I mentioned, uh, earlier in the supervised case, we feed a label data set to a learner, uh, and then it able to generalize, and we reinforcement learning works very differently where, you have an agent that sits within an environment, and the agent is, essentially, able to generate the data itself by exploring that environment.Christian Seifert:So, think of an agent in a computer network, that agent could, first of all, scan the network to, maybe, uncover notes and then they're, maybe, uh, actions around interacting with the notes that it uncovers. And based on those interactions, the agent will, uh, receive a reward. That reward actually may be delayed by, like, there could be many, many steps that the agent has to take before the reward, uh, manifests itself. And so, that's, kinda, how the agent learns, it's, e- able to interact in that environment and then able to receive a reward. And so that's, kinda, what, uh, stands, uh, within the core of the, the CyberBAttleSim, because William Bloom, who is the, the brains behind the simulation, has created an environment that is compatible with, uh, common, uh, reinforcement learning tool sets, namely, the OpenAI Gym, that allows you to train agents in that environment.Christian Seifert:And so, the CyberBattleSim represents a simple computer network. So, think of a set of computer nodes, uh, the, the nodes represent a computer, um... Windows, Mac OS, sequel server, and then every node exposes a set of vulnerabilities that the agent could potentially exploit. And so, then, as, kind of, the agent is dropped into that environment, the agent needs to, first, uncover those nodes, so there's a set of actions that allows to explore the state space. Overall, the environment has a, a limited observability, as the agent gets dropped into the environment, you're not necessarily, uh, giving that agent the entire network topology, uh, the agent first needs to uncover that by exploring the network, exploiting nodes, from those nodes, further explore the network and, essentially, laterally move across the network to achieve a goal that we give it to receive that final reward, that allows the agent to learn.Natalia Godyla:And, if I understand correctly, many of the variables were predetermined, such as, the network topology and the vulnerabilities, and, in addition, you tested different environments with different set variables, so how did you determine the different environments that you would test and, within that particular environment, what factors were predetermined, and what those predetermined factors would be.Christian Seifert:So we, we determined that based on the domaine expertise that exists Christian Seifert:... is within the team, so we have, uh, security researchers that are on a Red Team that kind of do that on a day-to-day basis to penetration tests environments. And so, those folks provided input on how to structure that environment, what nodes should be represented, what vulnerabilities should be exposed, what actions the agent is able to take in- in terms of interacting and exploring that, uh, network. So our Red Team experts provided that information. Nic Filingham:I wonder, Christian, if you could confirm for me. So there are elements here in CyberBattleSim that are fixed and predetermined. What elements are not? And so, I guess my question here is if I am someone interfacing with the CyberBattleSim, what changes every time? How would you sorta define the game component in terms of what am I gonna have to try and do differently every time? Christian Seifert:So the- the CyberBattleSim is this parametrized, where you can start it up in a way that the network essentially stays constant over time. So you're able to train an agent. And so, the network size is- is something that is dynamic, that you can, uh, specify upon startup. And then also kinda the node composition, as well as ... So whether ... how many Windows 10 machines you have versus [inaudible 00:19:15] servers, as well as the type of vulnerabilities that are associated with each of those nodes. Nic Filingham:Got it. So every time you- you establish the simulation, it creates those parameters and sort of locks them for the duration of the simulation. But you don't know ... The agent doesn't know in advance what they will d- they will be. The agent has to go through those processes of discovery and reinforcement learning. Christian Seifert:Absolutely. And- and one- one tricky part within reinforcement learning is- is generalizability, right? When you train an agent on Network A, it may be able to learn how to outperform a Red Team member. But if you then change the network topology, the agent may completely flail and not able to perform very well at all and needs to kind of re- retrain again. And that- that's a common problem within the- the re- reinforcement learning research community. Natalia Godyla:In the blog you also noted a few opportunities for improvement, such as building a more realistic model of the simulation. The simplistic model served its purpose, but as you're opening the project to the broader community, it seems l- that you're endeavoring to partner with the other researchers to create a more realistic environment. Have you given some early thought as to how to potentially make the simulation more real over time? Christian Seifert:Absolutely. There is a long list of- of things that we, uh, need to think about. I mean, uh, network size is- is one component. Being able to simulate a- a regular user in that network environment, dynamic aspects of the network environment, where a node essentially is added to the network and then disappears from the network. Uh, all those components are currently not captured in the simulation as it stands today. And the regular user component is an important one because what you can imagine is if we have an attacker that is able to exploit the network and then you have a defender agent within that network as well, if there is no user component, you can very easily secure that network by essentially turning off all the nodes. Christian Seifert:So in- a defender agent needs to also optimize, uh, to keep the productivity of the users that are existing on the network high, which is currently not- not incorporated in- in the simulation. Nic Filingham:Oh, that's w- that's amazing. So there could be, you know, sort of a future iteration, sort of a n- network or environment productivity, like, score or- or even a dial, and you have to sort of keep it above a particular threshold while you are also thwarting the advances of the- of the agent. Christian Seifert:Absolutely. And I mean, that is, I think, a common trade off in the security space, right? There are certain security m-, uh, measures that- that make a network much more secure. Think of like two-factor authentication. But it does u- add some user friction, right? And so, today we're- we're walking that balance, but I'm hoping that there may be new strategies, not just on the attacker's side, but also on the defender's side, that we can uncover that is able to provide higher level of security while keeping productivity high. Nic Filingham:I think you- you- you have covered this, but I- I'd like to ask it again, just to sort of be crystal clear for our audience. So who is the CyberBattleSim for? Is it for Red Teams? Is it for Blue Teams? Is it for students that are, you know, learning about this space? Could you walk us through some of the types of, you know, people and- and roles that are gonna use CyberBattleSim?Christian Seifert:I mean, I think that the CyberBattleSim today is- is quite simplistic. It is a simulated environment. It is not ... It'-s it's modeled after a real world network, but it is far from being a real world network. So it's, uh, simplistic. It's simulated, which gives us some advantages in terms of, uh, scalability and that learning environment. And so at this point in time, I would say, uh the simulation is really geared towards, uh, the research community. There's a lot of research being done in reinforcement learning. A lot of research is focused on games. Because if you think about a game, that is just another simulated environment. And what we're intending to do here with- with some of the open source releases is really put the spotlight on the security problem. And we're hoping that the- the reinforcement learning researchers and the research community at large will pay more attention to this problem in the security domain. Nic Filingham:It's currently sort of more targeted, as you say, as- as researchers, as sort of a research tool. For it to be something that Red Teams and Blue Teams might want to look at adopting, is that somewhere on a road map. For example, if- if you had the ability to move it out of the simulation and into sort of a- a- a VM space or virtual space or perhaps add the ability for users to recreate their own network topology, is that somewhere on your- your wishlist? Christian Seifert:Absolutely. I think there's certainly the goal to eventually have these, uh, autonomous defensive agent deployed in real world environments. And so in order to get to that, simulation needs to become more and more realistic in order to achieve that. Joshua Neil:There's a lot of work to be done there. 'Cause reinforcement learning on graphs, big networks, i- is computationally e- expensive. And just a lot of raw research, mathematics and computing that needs to be done to get to that real- real world setting. And security research. And in incorporating the knowledge of these constraints and goals and rewards and things that ... T- that takes a lot of domain research and getting- getting the- the security situation realistic. So it's hard. Christian Seifert:In the simulation today, it provides the environment and ability for us to train a Red Team agent. So an agent that attacks the environment. Today, the defender is very simplistic, modeled probabilistically around cleaning up machines that have been exploited. So as kinda the next point on the wishlist is really getting to a point where we have the Red Team agent play against a Blue Team agent and kinda play back and forth and see kinda how that influences the dynamic of the game. Natalia Godyla:So Christian, you noted one of the advantages of the abstraction was that it wasn't directly applicable to the real world. And because it wasn't approved as a safeguard against nefarious actors who might use CyberBattleSim for the wrong reason. As you're thinking about the future of the project, how do you plan to mitigate this challenge as you drive towards more realism in the simulation? Christian Seifert:That is certainly a- a- a risk of this sort of research. I think we are still at the early stages, so I think that risk is- is really nonexistent as it stands right now. But I think it can become a risk as the simulation becomes more sophisticated and realistic. Now, we at Microsoft have the responsible AI effort that is being led at the corporate level that looks at, you know, safety, reliability, transparency, accountability, e- et cetera, as kind of principles that we need to incorporate into our AI systems. And we, early on, engaged the proper committees to help us shape the- the solution in a responsible fashion. And so at this point in time, there weren't really any concerns, but, uh, as the simulation evolves and becomes more realistic, I very much expect that we, Christian Seifert:... be, uh, need to employ particular safeguards to prevent abuse. Nic Filingham:And so without giving away the battle plan here, wh- what are some other avenues that are being, uh, explored here as part of this trying to get ahead of this eventual point in the future, where there are automated agents out there in the wild? Joshua Neil:This is the- the core effort that we're making, and it's hard enough. I'll also say I think it's important for security folks like us, especially Microsoft, to try hard things and to try to break new ground and innovation to protect our customers and really the world. And if we only focus on short-term product enhancements, the adversaries will continue to take advantage of our customers' enterprises, and we really do need to be taking these kind of risks. May not work. It's too ... It's really, really hard. And t- and doing and in- in purposefully endeavoring to- to- to tackle really hard problems is- is necessary to get to the next level of innovation that we have to get to. Christian Seifert:And let me add to that. Like, we have a lot of capabilities and expertise at Microsoft. But in the security space, there are many, many challenges. And so I don't think we can do it alone. Um, and so we also need to kinda put a spotlight on the problem and encourage the broader community to help solve these problems with us. And so there's a variety of efforts that we have pursued over the last, uh, couple of years to do exactly that. So, about two years ago we published a [inaudible 00:28:52] data science competition, where we provided a dataset to the broader community, with a problem around, uh, malware classification and machine risk identification and basically asked the community, "Hey, solve this problem." And there was, you know, prize money associated with it. But I really liked that approach because we have ... Again, we have a lot of d- expertise on the team, but we're also a little bit biased, right, in- in terms of kinda the type of people that we have, uh, and the expertise that we have. Christian Seifert:If you present a problem to the broader research community, you'll get a very different approaches on how people solve the problems. Most likely from com- kind of domains that are not security-related. Other example is an RFP. So we funded, uh, several research projects last year. I think it was, uh, $450,000 worth of research projects where, again, we kind of laid out, "Here are some problems that are of interest that we wanna put the spotlight on, and then support the- the research community p- to pursue research in that area." Nic Filingham:So what kind of ... You know, you talk about it being, uh, an area that we all sort of collectively have to contribute to and sort of get b- behind. Folks listening to the podcast right now, going and reading the blog. Would you like everyone to go and- and- and spin up CyberBattleSim and- and give it a shot, and then once they have ... Tell us about the- the types of work or feedback you'd like to see. So it's up on GitHub. What kind of contributions or- or feedback here are you looking for from- from the community? Christian Seifert:I mean, I'd really love to have, uh, reinforcement learning researchers that have done research in this space work with the CyberBattleSim. Kinda going back to the problem that I mentioned earlier, where how can we build agents that are generalizable in a way that they're able to operate on different network topology, different network configuration, I think is an- an- an exciting area, uh, that I'd love to see, uh, the research community tackle. Second portion is- is really enhancing the simulation. I mentioned a whole slew of features that I think would be beneficial to make it more realistic, and then also kinda tackle the problem of- of negatively impacting potential productivities of- of users that operate on that network. So enhancing the- the simulation itself is another aspect. Nic Filingham:Josh, anything you wanted to add to that? Joshua Neil:Yeah, I mean, I- I think those were the- the major audiences we're hoping for feedback from. But a- al- also like Christian said, if a psychologist comes and looks at this and has an idea, send us an email or something. You know, that multidisciplinary advantage we get from putting this out in the open means we're anticipating surprises. And we want those. We want that diversity of thought and approach. A physicist, "You know, this looks like a black hole and here's the m- ..." Who knows? You know, but that's- that's the kind of-Nic Filingham:Everything's a black hole to a physicist- Joshua Neil:(laughs) Yeah. Nic Filingham:... so that's, uh ... Joshua Neil:So, you know, I think that diversity of thinking is what we really solicit. Just take a look, yeah. Anybody listening. Download it. Play with it. Send us an email. We're doing this so that we get your- your ideas and thinking, for us and for the whole community. Because I think we- we also believe that good security, uh, next generation security is developed by everybody, not just Microsoft. And that there is a- there is a good reason to uplift all of humanity's capability to protect themselves, for Microsoft but for everybody, you know? Natalia Godyla:So Christian, what are the baseline results? How long does it take an agent to get to the desired outcome? Christian Seifert:So the s- simulation is designed in a way that also allows humans to play the game. So we had one of our Red Teamers to actually play the game and it took that person about 50 operations to compromise the entire network. Now when we take a- a random agent that kinda uninformed takes random actions on the network, it takes about 500 steps. So that's kind of the- the lower baseline for an agent. And then we trained, uh, a Deep Q, uh, reinforcement learning agent, and it was able to accomplish, uh, the human baseline after about 50, uh, training iterations. Again, network is quite simple. I wouldn't expect that to hold, uh, as kinda the- the simulation scales and becomes more complex, but that was, uh, certainly an encouraging first result. Joshua Neil:And I think the- the significant thing there is, even if the computer is- takes more steps than the human, well, we can make computers run fast, right? We can do millions of iterations way faster than a- than a human and they're cheaper than humans, et cetera. It's automation. Nic Filingham:Is there a point at which the automated agent gets too good, or- or is there sort of a ... What would actually be the definition of almost a failure in this experiment, to some degree? Joshua Neil:I think one- one is to- to sort of interpret your question as it could be overfed. That is, if it's too good, it's too specific and not generalized. And as soon as you throw some different set of constraints or network at it, it fails. So I think that's a- that's a real metric of the performances. Okay, it- it learned on this situation, but how well does it do on the next one? Nic Filingham:Is there anything else, uh, either of you would like to add before we wrap up here? I feel like I've covered a lot of ground. I'm gonna go download CyberBattleSim and- and try and work out how to execute it. But a- anything you'd like to add, Christian? Christian Seifert:No, not from me. It was, uh, great talking to you.Natalia Godyla:Well, thank you Josh and Christian, for joining us on the show today. It was a pleasure. Christian Seifert:Oh, thanks so much. Joshua Neil:Yeah, thanks so much. Lots of fun. Natalia Godyla:Well, we had a great time unlocking insights into security, from research to artificial intelligence. Keep an eye out for our next episode. Nic Filingham:And don't forget to tweet us at MSFTSecurity, or email us at securityunlocked@microsoft.com, with topics you'd like to hear on a future episode. Until then, stay safe. Natalia Godyla:Stay secure.
4/28/2021

Knowing Your Enemy: Anticipating Attackers’ Next Moves

Ep. 25
Anyonewho’severwatched boxing knows that great reflexes can be the difference between achampionshipbeltand a black eye.The flexing ofan opponent’s shoulder, the pivot of theirhip-a good boxer will know enoughnot only topredictand avoidthe incoming upper-cut, but willknow how to turn the attack back on theiropponent.Microsoft’s newestcapabilities in Defender puts cyber attackers in the ring and predicts theirnext attacks as the fight is happening.On today’s episode,hosts Nic Fillingham and Natalia Godyla speak with ColeSodja, Melissa Turcotte, and Justin Carroll(and maybe even a secret, fourth guest!)abouttheirblogposton Microsoft’s Security blogabout the new capabilities of using an A.I.to see the attacker’s next move.In This Episode, You Will Learn:• What kind of data is needed for this level of threat detection and prevention?• The crucial nature of probabilistic graphical modeling in this process• The synergistic relationship between the automated capabilities and the human analystSome Questions We Ask:• What kind of modeling is used and why?• What does the feedback loop between program and analyst look like?• What are the steps taken to identify these attacks?Resources:Justin, Melissa’s, and Cole’s blog post:https://www.microsoft.com/security/blog/2021/04/01/automating-threat-actor-tracking-understanding-attacker-behavior-for-intelligence-and-contextual-alerting/Justin Carroll’s LinkedIn:https://www.linkedin.com/in/justin-carroll-20616574/Melissa Turcotte’s LinkedIn:https://www.linkedin.com/in/mturcotte/ColeSodja’sLinkedIn:https://www.linkedin.com/in/cole-sodja-a255361b/Joshua Neil’s LinkedIn:https://www.linkedin.com/in/josh-neil/NicFillingham’sLinkedIn:https://www.linkedin.com/in/nicfill/NataliaGodyla’sLinkedIn:https://www.linkedin.com/in/nataliagodyla/Transcript[Full transcript at https://aka.ms/SecurityUnlockedEp25]Nic Fillingham:Hello, and welcome to Security Unlocked, a new podcast from Microsoft, where we unlock insights from the latest in news and research from across Microsoft Security engineering and operations teams. I'm Nic Fillingham.Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories for Microsoft Security, deep dive into the newest threat intel, research and data science.Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft Security.Natalia Godyla:And now, let's unlock the pod. Welcome, everyone, to another episode of Security Unlocked, and hello, Nic, how's it going?Nic Fillingham:It's going well, good to see you on the other side of this Teams call. Although, you and I were in person not 24 hours ago. You were here in Seattle, we were filming some more episodes of the Security Show. I don't think we've really given listeners of the podcast a full, meaty introduction to the Security Show, have we? Do you wanna let listeners know what they can find?Natalia Godyla:We play games and hang out with experts in the industry and we've done everything from building robots with folks, to building blocks, to painting our nails. You can find the Security Show on our YouTube channel, so, YouTube.com/MicrosoftSecurity or you can go to aka.ms/securityshow. We talk with Chris Wysopal, the CTO and co-founder of Veracode on modern secure software development, and Dave Kennedy, who comes to talk to us about SecOps and everything you need for a survival kit in SecOps, so come come check them out.Nic Fillingham:Bad news is you, you have to deal with, uh, Natalia and I on another, uh, media format. But before you go there, make sure you listen to today's episode of Security Unlocked. We have a couple of returning guests. We have Cole and Justin, who have been on before, as well as Josh Neil, who comes on in the, in the last few minutes. And new guest, Melissa. They're all from the Microsoft 365 Defender research team, and they all co-authored a blog from April 1st called Automating Threat Actor Tracking, Understanding Attacker Behavior for Intelligence and Contextual Alerting, which is exactly what it is but I think it buries the lead. Natalia, you had a great TL;DR, what did they do?Natalia Godyla:The team used statistics to predict the threat actor group and the next stage in the attack and really early in the attack, so that we could identify the attack and inform customers so that they could stop it. I think what's really incredible here is, not only the ability to predict that information, but to just do it so early in kill chain. Nic Fillingham:Within two minutes after an attack begin, using this model, Microsoft threat experts were able to send a notification to the customer to let them know an attack was underway. The customer was able to do, you know, the necessary things to get that attack shut down. We'd love, as always, your feedback. Send us emails, securityunlocked@microsoft.com. Hit us up on the Twitters. On with the pod. Natalia Godyla:On with the pod. Nic Fillingham:Well, welcome back to the Security Unlocked podcast, Cole and Justin, and welcome to the Security Unlocked podcast, Melissa. Thanks for joining us today. We have three wonderful guests, with maybe a, a fourth special guest appearing at the end. And today we're gonna be talking about a blog post appearing on the Security blog from April the 1st, called Automating Threat Actor Tracking, Understanding Attacker Behavior for Intelligence and Contextual Alerting. All of the authors from that blog are here with us. Cole, if I could start with you, if you could sort of reintroduce yourself to the audience, give us a little bit, uh, about your role, what you do at Microsoft, and then perhaps hand off to one of your colleagues for the next intro.Cole Sodja:Sure. Will do, thank you. So, hi, I'm Cole. I work in the Microsoft 356 Defender group. I'm a statistician. Primarily my responsibilities are driving, kind of, research and innovation in general, with supporting threat analytics, threat hunting, threat research in general. Yeah, been doing that for about three years now, and I love it, and I that's a little bit about myself, I'll hand it over to Melissa. Melissa Turcotte:All right. My name's Melissa, I work with Cole, so in the same group, Microsoft 365 Defender. I'm also a statistician by background. I've been in the cyber domain for about probably seven years now. I was working for Department of Energy research laboratory in their cyber research group for five years, and I joined Microsoft a year ago. I like all sorts of problems related to cyber. My expertise probably would be in anomaly detection, but anything related to cyber, and there's data in a problem, I like to be involved.Nic Fillingham:And Justin.Justin Carroll:Hey. I also work in the Microsoft 365 Defender team, doing threat intelligence. My main focus is uncovering new threats and actor groups and understanding what they're doing, different modifications to how they're conducting their attacks, and the outcomes of those attacks, and then figuring out the most effective ways to either, communicate that out to customers or action on detection capabilities to stop them from succeeding.Nic Fillingham:Listeners of the podcast will note that you have a super sweet ninja turtles tattoo, is that correct? Justin Carroll:This is accurate, this is definitely accurate. Nic Fillingham:And, and we may or may not have a super secret fourth guest on this episode, who may join us towards the end, who you would, you would know from an very early episode of the podcast, but perhaps we'll keep them secret until the very end. Thank you all for joining us, thank you for your time. Again, we're referring to a, a blog post that, that all of you authored from April 1st. This is a, quite a complex, and, and sort of technical blog post, which I know a lot of our audience will love. Nic Fillingham:I got a little lost in the math, but I, I absolutely was enthralled by what you all have undertaken here. Cole, if I could start with you, can you give us, give us an overview of what's covered in this blog post, and sort of what this project was, how you tackled it, and what we're gonna talk about, uh, on this episode today.Cole Sodja:Yeah. So if I step back, being someone kind of still fairly new in learning, uh, to cyber security, uh, I approached things pretty much with just using data, right? Doing data driven imprints, as I'd say. And through my research, what I started to, um, kinda ask myself is, can we kinda get ahead of cyber security attacks, you know, from a post-breach perspective? Once we see an adversary in a network, can we start to make some predictions, basically, on what they're likely gonna do? Who is the adversary, or is it human operated, is it an automated script, for example. And then if we recognize the adversary, kinda recognize their tactics, their techniques, their procedures, can we say, okay, we're, we're likely gonna see they're gonna ransom this enterprise, for example.Cole Sodja:So I tried to look at it as more of a data mining exercise initially, it's like, can I recognize these type of patterns, and then how predictive are these patterns that we're seeing in terms of what likely is gonna occur. Or put it another way, what type of threat is this, essentially, to the enterprise? So, so that's kinda the background, the motivation. Now, when I started this project, back with Justin and then with Melissa, it started really as let's look for particular, uh, threat actors that we're aware of, that we recognize, that we know about, and see, like, can we start, from a data perspective, classifying okay, is it this group, is it that group, and what does this group tend to do? Cole Sodja:And one of the challenges in that is, is sparsity. Basically, we don't have a lot of labels sitting around out there saying, it's threat actor group A, B, C, D, and so on. We have handfuls of those. Some of these actors, they don't tend to do attacks very frequently, right? They're extremely sparse. So, so one challenge of this, and one the motivation is, how can we actually partner with threat intelligence, for example, and our threat hunters, to try and essentially encode or extract some of their information to help us build models, to help us reason over the uncertainty, essentially. Cole Sodja:And when we say probabilistic modeling, that's what we mean. It's how do we actually quantify this uncertainty, both in what we believe about the actors, or the adversaries in general, as well as what they're gonna do, right, once they've breached your network. So that's kinda how it started, and what this blog's really about is kinda giving a walk-through, essentially, of what we did initially with this research. It started with, and Justin will talk about this in a moment, it started with looking at few, select threat actors that are very serious. Cole Sodja:We started to understand their behaviors more and more and we thought it was a good opportunity, initially, to try and build a model to, again, understand what they're doing, track what they're doing, because they do change their tactics over time, as well as just see if we could get ahead of them. Can we actually notify a customer in advance, before, uh, for example, their organization's ransomed? So, so that's one part of the blog that we'll discuss, and I'll hand it over to my good friend Justin to take it from here.Justin Carroll:So, like, one of the, the main challenges that we kinda face in the intelligence sphere is understanding the particulars of an actor and when they are present in an environment. A lot of times, you'll see the intelligence is really focused on a very particular indicator such as, like, a known IP address that's malicious, or a single behavior. But it's kinda difficult to frequently pivot them out to understand when a suspected attacker is in an environment. A lot of that is due because they don't always do the exact same behaviors when they are compromising... Organization or device. There will be some variation and it basically requires manual enrichment a lot of the times of devices to try and understand the specifics of the attacks and what Justin Carroll:... the final outcomes o- wh- out of that attack, so this opportunity presented one to work with data scientists to, like, really supercharge our efforts so that we could kinda come in understanding a much bigger picture and knowing, essentially, what behaviors that we saw occur and then which ones we might suspect. A lot of times with these human operated ransomware ones, the time to alert, to notify of the expected outcome is often fairly short, in particular with, uh, one of the ones that we worked on to kinda test this method out. We had seen very short instances from time to compromise to ransom, so, um, this was to try and see if we could have a, a highly confident method of enriching that intelligence, um, and then working with other teams to get those alerts out.Natalia Godyla:If I could jump in here for a moment. So, at the beginning of your description, you noted that typically you'd use manual enrichment. Can you talk a little bit about that? So prior to this probabilistic model, how did you go through that manual enrichment process to try to, uh, predict what threat actors they were or determine what stage of an attack it was?Justin Carroll:It would be something along the lines of, let's say, you had intelligence from either a partner team or open source intelligence that says, you know, "These threat actors are using this IP address as part of their attack," and then looking for the presence of that and then finding out what actually occurred on those devices to understand the entirety of the attack, or looking more generically and saying, like, "Okay, we know these attackers like to use a particular behavior as part of their credential theft," and then so looking for all sorts of instances of that credential theft and then kinda continuing to pivot down into one that is leading to the behavior that y- you're looking for. One of the difficulties that you'll see in particular with this and other actors is, like, they will use multiple shared open source tools and payloads. Um, many of them aren't even malware, they're clean tools with legitimate purposes, so it can make it difficult to try and suss out the ones from malicious versus administrative use, so you have to look for that combination of different behaviors to indicate something malicious is afoot.Nic Fillingham:Justin, if I look at the blog, I think it might be the first chapter here, there's a MITRE ATT&CK framework diagram, Figure One, and it, uh, outlines sort of the steps taken here for how this model was able to, with high confidence, identify the, the actor and, uh, send an alert to the customer who was able to shut it down. I wonder if you could sort of, could you walk us through this, these sort of six steps as an example of, of how this work, how this worked in, in sort of real life?Justin Carroll:Yeah. I can walk through basically from a model's perspective, essentially, how it works. Timing, that's more a function of, like, how the attack, uh, typically progresses with this actor. Technically speaking, what the model's really doing is it's encoding each behavior we have, in this case, each MITRE technique in particular in terms of what's the confidence that once we see, for example, initial access follow... Under, let's say, RDP brute force, followed by lateral tool transfer with subset of tools recognized, that particular sequence right there, that's where the model would be like, "Okay, the probability that it's this particular threat actor group conditional on those two things occurring in sequence will be X," and that sequence could occur in a matter of minutes or even days and weeks, dependent on the actor, of course, we're talking about. Justin Carroll:With the, the actor we're showing in this graph, this actor typically will penetrate a network through RDP brute force, but then w- sometimes the, they won't immediately transfer their tools. They might wait a day or two, or sometimes they'll, they'll do it very fast, like, once they basically compromise a log-in then, uh, they'll, they'll go to that machine, there might be some, um, discovery related commands before they transfer or they might just transfer their tools and then that will be the attack box, basically, in which they stage their attack, and then they'll do some additional things.Justin Carroll:So at each step, basically, or each stage of the attack, as we like to call it, the model is basically gonna then update its probabilities and say, "Okay, based on all the information I've seen up to this stage, the probability that it's this actor is P and now, conditional that it's this actor with probability P, the probability that we'll now see, for example, defense evasion and this 'tack will be Q," or, or we could even go further in the attack stage to say, "Now, given all this, what's the probability that we'll see, for example, ransomware or inhibit system recovery in the coming hour? Or in the coming, you know, X time?" Justin Carroll:So the model's able to do that, but it's primarily conditional on the stages it's observed up to a point in time, not so much in terms of the time it takes for the actors to do X.Natalia Godyla:So, in this blog and in our discussion today, we're gearing up to talk about probabilistic graphical modeling as a way to address the challenge that, Cole and Justin, you've set up for us today, and, and for any of our listeners who'd like to follow along in the blog, the blog is titled "Automating threat actor tracking: Understanding attacker behavior for intelligence and contextual alerting" and you can find it on the Microsoft Security blog. I'd love to dive into the probabilistic graphical modeling and perhaps start with a definition of what that means. So, M- Melissa, could you give us an overview of this approach?Melissa Turcotte:Yeah. We have this problem which what they are essentially saying is, we have a collection of things which... I'm a statistician so I often call them variables, but, you know, features, if you will, if that's m- more easy for you to understand, but we, th- these TTPs, th- right. The sets of things that the actors are doing, and we have a collection of them. And given some collection of these, we wanna make a statement about whether or not it's ransomware or whether it's not a specific threat actor, or a group of actors. Right? And this is, this is, like, a perfect, um, example of where probability can help you make these decision, and one thing I'd like to stress is that no one of these features gives you enough information about whether or not it's this actor or this, this group of actors, or it's ransomware, you know, whatever your variable interest is.Melissa Turcotte:It really is the collection of these together that, you know, kind of in Justin's mind, as an analyst, he's, he's making these connections in his head, and I wanna be able to replicate that in some sense, I wanna take into account his knowledge and kind of his decision making process, combined with the data that I have, to make these probabilistic statements about what I think is happening. And graphical models are really great here, probabilistic graphical models in particular, as they kind of provide this joint probability distribution over all these features, and the variable of interest, in this case, is kind of, maybe is it this actor, but not necessarily. I mainly wanna know something about any one of these other features. I may already know it's this actor, and I may wanna be like, "Wh- what are the common things I see this actor do?"Melissa Turcotte:So, so graphical models really shine in this case where you have this collection of things that you are observing, and you kind of want to ask questions about any subset of them. Given some observations of others, and so th- this is a really great tool to use in this setting, and it's also quite interpretable. So if you kind of look, if you're looking at the blog and you see this Figure Two, which is a toy example, but y- you kind of, as a human, you can look at that and you can kind of understand that, "Okay, so I'm seeing transfer tools and lateral movement are related." Um, and you can kind of understand sort of wh- what the relationships the model is making. Um, and so that kind of provides this extra, you know, benefit of this in that, yeah, I can talk an analyst through what this kind of is showing and then i- it's quite interpretable for them even if they don't understand the underlying maths, and that's kind of something we really wanna strive for. Um, you shouldn't have to understand the underlying maths to kind of understand the decisions that are being made.Melissa Turcotte:It's really attractive in this sense, and then the Bayesian networks, why I really like it is kind of, the Bayesian paradigm is... So you, you have, you know, statistics, generally, or data science, you have some data and you're kind of, you know, making inference given the set of data to make statements about things of interest. So the data tells you something about your beliefs and the state of the world, but you have your own subjective beliefs about wh- what you think could and could not happen. The, the Bayesian paradigm kind of combines those two things, so it's, you have your beliefs and then you have what the data is telling you, a- and your ultimate kind of predictions are based on the combination of those things. And generally, the, the way it works is the more data you have, the data will always win through.Melissa Turcotte:So this problem, bringing it back to attacker prediction, is a case where we don't have a lot of data, right? We don't... Companies get attacked... Or we say, companies get attacked all the time but not at the scale at which we collect the underlying data, so like, you know, we have, you know, you as a user are performing actions, logging into computers you use... You know, this shows up in the data thousands of times a day, whereas an attack happens kind of, like, on a monthly scale, so c- the scales of attacks to the data we're getting is just really small, and then when you go into attacks that kind of we've labeled as being attributed to a threat actor, I mean, that's even way smaller. So it's, it's kind of a small data problem, uh, in terms of the number of labels you have.Melissa Turcotte:But what we do have is this analysts who have spent years tracking these people and have their kind of, you know, beliefs about what they do and how they changed over time. And so we Melissa Turcotte:Wanna capture that. We definitely want to include the evidence we see and the data, but we wanna capture that really rich knowledge that we get from the analysts. And so kind of that's where the Bayesian network part becomes attractive because it, it provides a very principled way to, to capture the analysts' expertise, combine that information with the data we're seeing to make these ultimate predictions.Natalia Godyla:For our audience, could you really quickly describe a Bayesian network?Melissa Turcotte:So, a Bayesian network is a way of building a model for a collection of variables whereby the idea is that you have different variables which are related to each other. It, it, it kind of helps draw out or show what those relationships are so, like, in the graph, you know, if there's an arrow from impact... Or from transfer tools to impact that's saying if I see transfer tools, that has a direct impact... I'm gonna use the word impact twice here. Has a direct impact on whether or not I'm going to see impact. So, so it's kind of the way the variables relate to each other and the way the probabilities change according to those relationships. And so a Bayesian network encodes all this information. Nic Fillingham:If I can take another swing at that one... Thank you, Melissa. I'm wondering what were some of the other, uh, techniques that you either considered for this approach? Like, did you experiment with other methods and then ultimately chose Bayesian?Melissa Turcotte:Yes, um, in fact, uh, so the initial kind of... The perhaps most obvious thing to do is to c- to think of decision trees, right? You s- you're, you're, you're seeing, you know, these things over time. Okay, I saw, um, what was the first one? Initial access with this... You don't go as broad as initial access, but I saw initial access using this, you know, minor technique. And so you can kind of think, like, you, you, you have a tree that's kind of... Okay, I saw this, I didn't see this, but I saw this and I didn't see this, so now I think it's this actor. But kind of where this is preferable is the fact that, as Paul says, we don't want to see the whole attack happen before we make a statement about what we think it is. And Bayesian networks work really well in, in the absence of some observed variables. Cole Sodja:Yeah, I'll just quickly chime in. I agree with Melissa. So, I did experiments, for example, with several models including decision trees. Even, um, different forms of Bayesian decision trees like BART for example. And in addition to what Melissa is saying where, for example, predicting the probability that it's threat actor conditioned on certain variables we saw, uh, we might also, as Melissa pointed out, want to say, okay, let's predict, for example, that this threat actor is going to do impact or a certain form of impact. And with decision trees, that means basically you're building multiple decision trees to do that. You can't just build one decision tree... Well, let's put it this way. You can't easily build one decision tree to have multiple target variables. That's something you get for free with the Bayesian network. Another thing I'll say in addition to what, um... To marginalization is the Bayesian network is more general. So, it could actually handle kind of a broader graphical structure. The decision tree is a specific graph. Cole Sodja:So, it kind of already inhibits you, if you will, to learning a certain structure over the data. Whereas the Bayesian nets, they could give you a little more general structure. We could also build these models that are time dependent, what are called dynamic Bayesian networks. That's something much harder to do with tree models. So, it's just a more flexible model as well as I would say. In my experiments, the Bayesian network did perform better on average than the set of decision trees I considered.Nic Fillingham:I'd like to better understand the relationship between this model and folks like Justin. So, is Justin, as a very experienced threat analyst, is Justin helping you define labels and helping you sort of build some of the initial... I'm, gonna get the taxonomy wrong here, so please correct me. But the initial sort of properties of the model? Or is, is Justin, as an analyst, interpreting what you sort of think you have in the model? How, how do I understand the relationship between the analyst and, and how they're providing their expertise into, into this model?Melissa Turcotte:All three.Nic Fillingham:Oh, great. (laughs)Melissa Turcotte:All three things you said is actually correct. So, so hopefully we, we've explained it somewhat well. So, yes. The first stage, right Justin? The analysts are providing us our label data. So, yes. That's the first thing. And then they also help us kind of, you know, you have the raw data, but that's kind of... There's so much data processing that goes... That, that happens before it's kind of... This data's kind of in this tabular forms that's like, yes, we... You know, these are the features we are tracking, so think of your TTPs, the different notes in your graph. Getting the data into that, kind of that schema, the threat analysts help with. So, you know, help define what, what these tactics, techniques, and procedures are that we should track. Like you said, you, you can't be super broad. Lateral movement doesn't really have a lot of meaning, um, to kind of like the different ways in which someone can do lateral movement and how granular w- you want to go. Melissa Turcotte:So, we discuss with the analysts all the time to kind of build up, you know, the ontology, if you will. And then, you know, as a first stage, like I said, it's a small data sample, so we're like... Justin helps inform what the model thinks about in a probabilistic sense. So, you... One thing I might ask him, I, I would be like... If I saw net... you know I'm borrowing from our toy example, but if I saw a network scanning modify system process, transfer tools, but didn't see any of the others, do you think it would be this actor X? Or do you think it would be ransomware? And he would be like, hmm, I would probably 60% certain. I can take that information and encode that directly so that, in the absence of any data, the model would return 60%. It would... If I didn't see any data, it would return what Justin believed was the probability in the presence of a certain number of variables. Melissa Turcotte:And then we kind of see data and we update our beliefs over time based on that. And then, also, after we've kind of trained these things, I go back to Justin and say does this make sense to you? So, he, he's kind of involved in all three, the whole process.Nic Fillingham:Melissa, I think you're telling me you've built a virtual Justin. Melissa Turcotte:We... That, that is what we are literally trying to do. And back it up... And, you know, and back it up with data as well. I'd, I'd like to like... You know, I'm a firm believer that everyone has their subjective beliefs, Justin has beliefs as well. Oftentimes, I can prove analysts wrong. Be like, they think something, I'm like, well, the data is telling me something else. So, we need to figure out, you know, that discrepancy. But, yes. We are essentially trying to build virtual Jus- uh, Justins. Although, like, th- there... I don't think there's any stage upon which we won't need the analysts to constantly feed back in with the new information they have. Nic Fillingham:Got it. And then can it come full circle? Justin, how do you as an analyst, how do you get smarter and better at what you do by what this model is, is telling you? What's the feedback loop look like here for you?Justin Carroll:It's one of those where, basically, using the model kind of super-charged my abilities where, instead of having to look at this very granular kind of like ad hoc, oh, this may be interesting, now I have the instances already serviced to me, and I have a good understanding of what success rate through the kill chain the attacker was able to get. And maybe figure out which ones that I needed to enrich more to understand was there data that we can add into the model because they've done something different that we need to capture and then look for opportunities in that way. So, really, it's basically... It made it where, give or take, sometimes it would take anywhere from 10 to 20 minutes sometimes to try and figure out, like, is this who I think it is? And like, what have they done? What are their goals? To just looking at the result from the model. And within usually seconds, being like, yeah, that looks exactly right. That's... It's confirmed, I think that's spot on. Natalia Godyla:So, Justin, was there something that was the most surprising in working with this model? Something that the model taught you either about threat actors or any details about the features? Justin Carroll:One of the things was kind of reexamining My confidence levels on different parts of the attack. Um, where Melissa was stating, for instance, you know, the data suggesting this and the models coming to this conclusion, uh, you know, thinking that it's this probability, and there would be times where I'd have to kind of reevaluate and think, like, hmm, I might've been missing something or overestimating the prevalence of a particular thing and saying it's related to such. Like, uh, I can tend to get very biased based on my narrow scope of the attacks that I'm looking at and think that it's related to this thing, but the model was able to provide a lot of clarity to some of the behaviors that maybe I didn't think were as confident a signal or extremely confident signal and I wasn't giving them the appropriate weight. That's one of the advantages of using it to understand what the attacker's doing, is I let it do much of the leg work once everything's kind of coded in. And then occasionally, like if we found opportunities where it was like, hmm, this still isn't quite right, then it could be tuned as a c- um, as necessary. Justin Carroll:I think that was probably one of the biggest ones of kind of trying to work through and actually spell out, like, my own thinking processes when I'm evaluating the data. It was something that you just kind of do without thinking, where you're constantly, as an intelligence analyst, looking at data and making conclusions on that data. But you're not usually saying, like, okay, I saw this so I'm gonna give it a 60% probability that it's this. And like, you're, you're just kind of sometimes it's either gut intuition or working on it that way. But actually having the model encode and return back what it was understanding made a, a pretty big impact in trying to understand how my own decision processes work and basically how best to kind of think Justin Carroll:About these different, wide array of attacks that we're constantly investigating.Nic Fillingham:The types of indicators that you're building this model on, again please correct me on my taxonomy here, but you're not looking for, you know, NFO files or like ASCII art or, you know, the actual threat actors name being sort of hidden somewhere in the jpeg that they drop as a, as a for the LOLs, like, they're... You're not looking for a sort of a literal signature of these threat actor groups, you're, you're, what you're, what you're doing is you're, you're seeing the actions that have been taken and without any other way of attributing them to an individual group, you're piecing them together. Nic Fillingham:And as you, as you get more actions and you piece them together based on the, the labels that you get from people like Justin, you're able to, to ultimately have a high probability that it's this threat group actor and they're doing this thing and they're likely to do this thing next. Have I got that right? You're, they're... In no way shape or form are you actually finding a secret text file that has the name, you know, the, the, the handles for all the hackers who are doing it for the LOLs.Cole Sodja:So let me just quickly jump in, you pretty much nailed it. I'll say this, so, we wanted to do both actually, right, because we don't want to restrain the model if it's, if core's gonna add predictive power, so like you said, we're not actually searching, grepping for example, for a threat actor name and some file or image, certainly not that level. But, for example, some of the actors, maybe they have common infrastructure, maybe they use particular types of tools in their attack typically, right? Like, maybe there's a SHA-1 out there they've used a lot in their attack, or, or recurring IP addresses they use as part of brute forcing. Cole Sodja:Those are there, but those are very specific and if you just relied on those, like Melissa was saying, either one or a few of those, you're not gonna generalize. You'll probably miss that attacker, right? But we certainly don't want to exclude it from the model because, um, if we happen to see that, the model will, uh, come back with a different type of probability, right? It'd be like, okay. Now the model might be more confident early, rather than waiting to see how the rest of the kill chain progresses. On the more general side, we probably won't go to the MITRE categories, 'cause they're a little too general, right? But if we go to some of the sub techniques, we don't actually have to look at the particular types of executables, or tools, or IPs used. Cole Sodja:Sometimes just the timing and sequencing is enough actually, to narrow down to, maybe not a particular threat actor, but a group of actors or, more generally, we can say with high competence, you know, this is a human adversary. They're taking this amount of time to do discovery commands, they're, they're doing lateral these type of ways. And the model could recognize that, even without knowing the particular commands, it's just seeing the more general techniques involved, right? So we do a bit of both, actually. We tend to want to rely more on, kind of, the general attacks or indicators as you're saying, that's right. But, we certainly don't want to throw away specifics that are reuse because we could get ahead of the attack much earlier too. So it's a bit of both at the end of the day.Melissa Turcotte:So yes, Nic, if, if, if you have an evil bit, look for the evil bit. You don't need data science for that. Nic Fillingham:(laughs)Natalia Godyla:And how is this model being used today, meaning is this a model that's being used by our internal security team to protect Microsoft and its customers, is it being used by a Microsoft threat experts group or is this actually embedded in some of our solutions today, and our customers are feeling that benefit? And what is the future intent of the model?Justin Carroll:One of those... So, there are multiple uses that are in place for the model. So one of the big things for me, so in my own selfish interest, it's intelligence, it's one of the easiest ways that I can keep tabs on the attacker and continually build new profiles and understand, basically, reports out, this is what they're doing, this is how they're doing it, this is how active they are. Like, are we seeing, you know, large volumes of their attack, are they taking a break, that kinda stuff. Then, the Microsoft threat experts are using it as a signal to help understand attacks early on in the kill chain so that they can get those notifications out ideally before the ransom, which can be quite difficult a lot of the times depending on the adversary and how quickly they seek to ransom. A lot of times there isn't a great deal of time.Cole Sodja:Yeah, there's other products, for example, M365D. So, um, there are plans, uh, it requires some engineering, ultimately, because this is a big product, um, huge customer base and so on. But there are already plans in motion to take what we've built already, as part of this framework, and integrate that into that product. There's other products as well, both from a threat intelligence perspective, and possibly kind of from SOC alerting perspective as well, that I'm in active discussions with other products across Microsoft to do the POC, make sure it works with their data, make sure they're comfortable and then work with their engineering team to at least get that in the plan. Those are ongoing discussion but M365D does have, kinda, I'll say, in their planning cycle, to get this in the product. Nic Fillingham:I wonder if this might be a good time to bring our secret special guest on microphone, Josh, if you're there, I think I might ask, uh, might wonder if you could jump in on this one. I think you've understated the power of what you've built here. From everything that you've just explained, you know, within a couple of minutes of a threat actor getting initial access to have a high probability index to be able to contact the customer and say, here's who we think is inside your network, here's what we think they're gonna do next, so they can shut it down. This is the next level, right? And, and Josh, when we interviewed you on episode three, you were hinting at this, if I'm not mistaken. Is this, is this sort of what you guys have been working on?Joshua Neil:Yeah, I'm so proud that we, that we took it from concept to realized value for the customers and, and at this point we've had that impact with your customers in stopping human operations. And, and so it's really exciting and, and it's, it's on the journey but, you know, if I extract an overall theme from this, it's consistent with that podcast that we had before because I was sort of complaining about AI. And I was sort of complaining about what we see in some of the, in some of the branding and marketing that, that folks do in, in cyber security. And I think this team and, and the work they've done exemplifies the right applications of data driven methods. Joshua Neil:There is no magical, artificial intelligence today. What there is is, and this is a, an experience that all of us on the data science team have had over the, over the past few years, and really for me about 20 years, is we can use data and some mathematics and some computing to begin to automate and accelerate what the humans are doing. And so, by sitting very closely with, and working very hard with the human experts like Justin, we're explicitly encoding their knowledge into models. So that's one thing is that the data science we're doing is to automate some of the stuff they're doing today. But the intention is not to solve the world, not to give our customers a license to solve security, we're, we're not gonna be able to do that. What we are able to do is uplift the sophistication of our customers operations. Joshua Neil:So, you know, what Justin sort of reflected on, uh, he's able to do a more interesting job, a more sophisticated job, because we're taking the data and his knowledge and encoding it and accelerating and automating some of the stuff that he's having to do manually now. And that's where the real nuts and bolts, you know, and the real rubber meets the road here, is that there's no magic gun that's gonna blow away all the adversaries with, with AI. What there is is hard work between data scientists and threat expertise to uplift their capabilities and accelerate their effectiveness in the face of the adversary. And that's what I would like to get across to the, to the listeners, is that by hard work and careful and close collaboration between data science and threat expertise, that's how we really make progress in this space.Nic Fillingham:Thank you so much Josh. And I just wanted to quickly clarify, from a previous comment from Cole, so this model is in use now, correct? Folks like Justin, Microsoft threat analysts, they are using this model now to make the model better, and to be able to get that additional information and those confidence levels in, in, in doing their analyst work. And so Microsoft threat expert customers are directly benefiting from this work, as of today. That's correct, is it?Joshua Neil:That's correct. We've sent targeted attack notifications to customers based on this model.Nic Fillingham:You've all been very, very, generous. Natalia Godyla:Thank you for that. And, and thank you to the whole team here for joining us on the show today. Melissa Turcotte:Absolutely.Cole Sodja:My pleasure.Joshua Neil:It was a lot of fun as always. And, and thank you, Nic and Natalia for this.Natalia Godyla:Well, we had a great time unlocking insights into security, from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us at MSFTSecurity or email us at securityunlocked@microsoft.com with topics you'd like to hear on future episode. Until then, stay safe...Natalia Godyla:Stay secure.
4/14/2021

Inside Insider Risk

Ep. 23
Throughout the course of this podcast series, we’ve had an abundance of great conversations with our colleagues at Microsoft about how they’re working to better protect companies and individuals from cyber-attacks, but today we take a look at a different source of malfeasance: the insider threat. Now that most people are working remotely and have access to their company’s data in the privacy of their own home, it’s easier than ever to access, download, and share private information.On today’s episode, hosts Nic Fillingham and Natalia Godyla sit down with Microsoft Applied Researcher, Rob McCann to talk about his work in identifying potential insider risk factors and the tools that Microsoft’s Internal Security Team are developing to stop them at the source.In This Episode, You Will Learn:• The differences between internal and external threats in cybersecurity• Ways that A.I. can factor into anomaly detection in insider risk management• Why the rise in insider attacks is helping make it easier to address the issue.Some Questions We Ask:• How do you identify insider risk?• How do you create a tool for customers that requires an extreme amount of case-by-case customization?• How are other organizations prioritizing internal versus external risks? Resources:Rob McCann’s Linkedin:https://www.linkedin.com/in/robert-mccann-004b407/Rob McCann on Uncovering Hidden Risk: https://www.audacy.com/podcasts/uncovering-hidden-risks-45444/episode-1-artificial-intelligence-hunts-for-insider-risks-347764242Insider Risk Blog Post: https://techcommunity.microsoft.com/t5/security-compliance-identity/don-t-get-caught-off-guard-by-the-hidden-dangers-of-insider/ba-p/2157957Nic Fillingham’s LinkedIn:https://www.linkedin.com/in/nicfill/Natalia Godyla’s LinkedIn:https://www.linkedin.com/in/nataliagodyla/ Transcript[Full transcript can be found athttps://aka.ms/SecurityUnlockedEp23]Nic Fillingham:Hello and welcome to Security Unlocked, a new podcast from Microsoft where we unlock insights from the latest in news and research from across Microsoft security engineering and operations teams. I'm Nic Fillingham. Natalia Godyla:And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft Security. Deep dive into the newest threat intel, research and data science. Nic Fillingham:And profile some of the fascinating people working on artificial intelligence in Microsoft Security. Natalia Godyla:And now, let's unlock the pod. Natalia Godyla:Hello Nic, welcome to today's episode, how's it going with you? Nic Fillingham:Hello Natalia, I'm very well, thank you, I hope you're well, and uh, welcome to listeners, to episode 23, of the Security Unlocked podcast. On the pod today, we have Rob McCann, applied researcher here at Microsoft, working on insider risk management, which is us taking the Security Unlocked podcast into- to new territory. We're in the compliance space, now. Natalia Godyla:We are, and so we're definitely interested in feedback. Drop us a note at securityunlocked@microsoft.com to let us know whether these topics interested you, whether there is another avenue you'd like us to go down, in compliance. Also always accepting memes. Nic Fillingham:Cat memes, sort of more specifically. Natalia Godyla:(laughing) Nic Fillingham:All memes? Or just cat memes? Natalia Godyla:Cat memes, llama memes, al- Nic Fillingham:Alpaca- Natalia Godyla:... paca memes. Nic Fillingham:... memes. Yeah. Alpaca. Yeah, this is a really interesting uh, topic, so insider risk, and insider risk management is the ability for security teams, for IT teams, for HR to use AI and machine learning, and other sort of automation based tools, to identify when an employee, or when someone inside your organization might be accidentally doing something that is going to create risk for the company, or potentially intentionally uh, whether they have, you know, nefarious or sort of malicious intent. Nic Fillingham:So, it really- really great conversation we had with- with Rob about what is insider risk, what are the different types of insider risk, how is uh, AI and ML being used to go tackle it? Natalia Godyla:Yeah, there's an incredible amount of work happening to understand the context, because so many of these circumstances require data from different departments, uh, uniquely different departments, like HR, to try to understand, well is- is somebody about to leave the company, and if so, how is that related to the volume of data that they just downloaded? And with that, on to the pod. Nic Fillingham:On with the pod. Nic Fillingham:Welcome to the Security Unlocked podcast, Rob McCann, thank you so much for your time. Rob McCann:Thank you for having me. Nic Fillingham:Rob, we'd love to start with a quick intro. Who are you, what do you do? What's your day to day look like at Microsoft, what kind of products or technology do you touch? Give us a- give us an intro, please. Rob McCann:Well, I've been at Microsoft for about 15 years, I am a- I've been an applied researcher the entire time. So, what that means is, I get to bounce around various products and solve technical challenges. That's the official thing, what it actually means is, whatever my boss needs done, that's a technical hurdle, uh, they just throw it my way, and I have to try to work on that. So, applied scientist. Nic Fillingham:Applied scientist, versus what's a- what's a different type of scientist, so what- what's the parallel to applied science, in this sense? Rob McCann:So, applied researcher is sort of a dream job. So, when I initially started, they're sort of the academic style researcher, that it's very much uh, your production is to produce papers and new ideas that sort of in a vacuum look good, and get those out to the scientific community. I love doing that kind of stuff. I don't so much like just writing papers. And so, an applied researcher, what we gotta do, is we gotta sort of be this conduit.Rob McCann:We get to solve things that are closer to the product, and sort of deliver those into the product. So we get very real, tangible impact, but then we're also very much a bridge. So, part of our responsibility is to keep, you know, fingers on what's going on in the abstract research world and try to foster, basically, a large innovation pipe. So, I freaking love this job. Uh, it's exactly what I like to do. I like to solve hard technical problems, and then I like to ship stuff. I'm a very um ... I need tangible stuff. So I love it. Nic Fillingham:And what are you working on at the moment, what's the scope of your role, what's your bailiwick? (laughing) Rob McCann:My bailiwick is uh, right now I'm very much focused on IRM, which is insider risk management, and so what we've been doing over the last year or so, insider risk management GA'd in February of 2020, I want to say. So, Ignite Today is a very festive sort of one year anniversary type thing. That with compliance solutions. So, over this last year, what we've done a lot of is sort of uh, build a team of researchers to try to tackle these challenges that are in insider risk, uh, and sort of bring the science to this brand new product. So, a lot of what I'm doing on a daily basis is on one hand, the one hand is, solve some technical things and get it out there, and the other hand is build a team to strengthen the muscle, the research muscle. Natalia Godyla:So, let's talk a little bit more about insider risk management. Can you describe how insider risk differs from external risk, and more specifically, some of the risks associated with internal users? Rob McCann:It's uh, there's some overlap. But it's a lot different than external attack. So, first of all, it's very hard, not saying that external attack is not hard, I- I work with a lot of those people as well. But insiders are already in, right? And they already have permissions to do stuff, and they're already doing things in there. So there's not like, you have a- a ... some perimeter that you can just camp on, and try to get people when they're coming in the front door. Rob McCann:So that makes it hard. Uh, another thing that makes it hard is the variety of risks. So, different customers have different definitions of risk. So, risk might be um, we might want to protect our data, so we don't want data exfiltrated out of the company. We might want trade secrets, so we don't want people to even see stuff that they shouldn't see. We don't want workplace harassment, uh, we don't want sabotage. We don't want people to come in, and implant stuff into our code that's gonna cause problems later. It's a very broad space of potential risks, and so that makes it challenging as well. Rob McCann:And then I would say the third thing that makes it very challenging is, what I said, different customers want- have different definitions of risk. So it's not like ... like, I like the contrast to malware detection. So, we have these external security people that are trying to do all this sophisticated machine learning, to have a classifier that can recognize incoming bad code. Right? And sort of when they get that, like, the whole industry is like, "Yes, we agree, that's bad code, put it in Virus Total, or wherever the world wants to communicate about bad code." And it's sort of all mutually agreed upon, that this thing is bad. Rob McCann:Insider risk is very different. It's um, you know, this customer wants to monitor these things, and they define risk a certain way. Uh, this customer cares about these things, and he want to define risk a certain way. There is a heightened level of customer preferences that have to be brought into the- the intelligence, to- to detect these risks. Natalia Godyla:And what does detecting one of those risks look like? So, fraud, or insider trading, can you walk through what a workflow would look like, to detect and remediate an insider attack? Rob McCann:Yeah, definitely. So- so, first of all, since it's such a broad landscape of potential damage, I guess you would say, first thing the product has to do is collect signals from a lot of different places. We have to collect signals about people logging in. You have to collect signals about people uploading and downloading files from a- from OneDrive, you have to ... you have to see what people are sharing on Teams, what people are ec- you know, emailing externally. If you want the harassment angle, you gotta- you know, you gotta have a harassment detector on communications. Rob McCann:So the first thing is just this huge like, data aggregation problem of this very broad set of signals. So that's one, which in my mind is a- is a very strong advantage of Microsoft to do this, because we have a lot of sources of signals, across all of our products. So, aggregating the data, and then you need to have some detectors that can swim through that, uh, and try to figure out, you know, this thing right here doesn't quite look right. I don't know necessarily that it's bad, but the customer says they care about these kind of things, so I need to surface that to the customer. Rob McCann:So, uh, technics that we use there a lot are anomaly detection. Uh, so a lot of unsupervised type of learning, just to look for strangeness. And then once we surface that to the- the customer, they have to triage it, right? And they have to look at that and make a decision, did I really- do I really want to take action on this thing? Right? And so, along with just the verdict, like, it's probability 98% that this thing is strange, you also have to have all this explanation and context. So you have to say, why do I think this thing is strange? Rob McCann:And then you have to pull in all these things, so like, it's strange because they- they moved a bunch of sensitive data around, that- in ways they usually didn't, but then you also need to bring in other context about the user. This is very user-centric. So you have to say things like, "And by the way, this person is getting ready to leave the company." That's a huge piece of context to help them be able to make a decision on this. And then once the customer decides they want to make a decision, then the product, you know, facilitates uh, different workflows that you might do from that. So, escalating a case to legal, or to HR, there are several remediation actions that the customer can choose from. Nic Fillingham:On this podcast, we've spoken with a bunch of data scientists, Nic Fillingham:... and sort of machine learning folks who have talked about the challenge of building external detections using ML, and from what you've just explained, it sounds like you probably have some, some pretty unique challenges here to give the flexibility to customers, to be able to define what risk needs to them. Does that mean that you have to have a customized model built from scratch for every customer? Or can you have a sort of a global model to help with that anomaly detection that then just sort of gets customized more slightly on top based on, on preferences? I, I guess my question is, how do you utilize a tool like machine learning in a solution like this that does require so much sort of customization and, and modification by the, by the customer? Rob McCann:That's, that's a fantastic question. So, what you tried to do, you scored on that one.Nic Fillingham:(laughs).Rob McCann:You try to do both, right? So, customers don't wanna start from scratch with any solution and build everything from the ground up, but they want customizability. So, what you try to do, I always think of it as smart defaults, right? So, you try to have some basic models that sort of do things that maybe the industry agrees is suspicious type, right? And you expose a few high-level knobs. Like, do you care about printing? Or do you care about copying to USB? Or do you want to focus this on people that are leaving the company? Like some very high level knobs. Rob McCann:But you don't expose the knobs down to the level of the anomaly detection algorithm and how it's defining distance and all the features it's using to define normal behavior, but you have to design your algorithm to be able to respect those higher level choices that the u- that the user made. And then as far as the smart default, what you try to do as you pr- you try to present a product where out of the box, like it's gonna detect some things that most people agree are sort of risky, and you probably wanna take a look at, but you just give the, you offer the ability to customize as, as people wanna tweak it and say, nah, that's too much. I don't like that. Or printing, it's no big deal for us. We do it. We're printing house, right? Nic Fillingham:Does a solution like this, is it geared towards much larger organizations because they would therefore have more signal to allow you to build a high fidelity model and see where there are anomalies. So, for example, could the science of the insider risk management work for a small, you know, multi hundred, couple hundred person organization? Or is it sort of geared to much, much larger entities, sort of more of the size of a, of a Microsoft where there are tens of thousands employees and therefore there's tens of thousands of types of signal and sort of volume of signal.Rob McCann:Well, you've talked to enough scientists. I look at your guys's guest list. I mean, you know, the answer, right, more data is better, right? But it's not limiting. So, of course, if you have tons and tons of employees in a rich sorta like dichotomy of roles in the company, and you have all this structure around a large company, if you have all that, we can leverage it to do very powerful things. But if you just have a few hundred employees, you can still go in there and you can still say, okay, your typical employees, they have this kind of activity. Weird, the one guy out of a 100 that's about ready to leave suddenly did something strange, uh, or you can still do that, right? So, you, you got to make it work for all, all spectrums. But more data is always better, man. Um, more signals, more data, bring it on. Let's go. Give me some computers. Let's get this done. Natalia Godyla:Spoken like a true applied scientist. So, I know that you mentioned that there's a customized components inside of risk management, but when you look across all of the different customers, are you seeing any commonalities? Are there clear indicators of insider threats that most people would recognize across organizations like seeing somebody exfiltrate X volume of data, or a certain combination of indicators happening at once? I'm assuming those are probably feeding your smart defaults?Rob McCann:Correct. So, there's actually a lot of effort to go. So, I s- I said that we're sort of a bridge between external academic type research and product research. So, that's actually a large focus and it happened in external security too. As you get industry to sort of agree like on these threat matrices, and what's the sort of agreed upon stages of attack or risk in this case. So, yeah, there are things that everybody sort of agrees like, uh, this is fishy. Like, let's make this, let's make this priority. So, that, like you said, it feeds into the smart defaults. The same time we're trying to, you know, we don't think we know everything. So, we're working with external experts. I mean, you saw past podcasts, we talked to Carnegie Mellon, uh, we talked to Mitre, we talked to these sort of industry experts to try to make this community framework or, uh, language and the smart defaults. Uh, and then we try to take what we can do on top of that. Nic Fillingham:So, Rob, a couple of times now, you've, you've talked about this scenario where an employee's potentially gearing up to leave the, the company. And in this hypothetical situation, this is an employee that may be looking to, uh, exfiltrate some, some data on their way out or something, something that falls inside the scope of, of identifying and managing, uh, insider risk. I wonder, how do you determine when a user is potentially getting ready to leave the company? Is that, do you need sort of more manual signals from like an HR system because an employee might've been placed on a, on a, on a review, in a review program or review period? Or, uh, are you actually building technology into the solution to try and see behaviors, and then those behaviors in a particular sort of, uh, collection in a particular shape lead you to believe that it could be someone getting ready to leave the company? Or is it both or something else? Rob McCann:So, quick question, Nick, what are you doing after this podcast?Nic Fillingham:Yeah.Rob McCann:Do you want a job? Because it feels like you're reading some of my notes here (laughter). Uh, we, uh-Nic Fillingham:If you can just wait while I download these 50 gigs of files first-Rob McCann:(laughs).Nic Fillingham:... from this SharePoint that, that I don't normally go to, and then I sort of print everything and then I can talk to you about a job. No, I'm being silly. Rob McCann:No, I mean, I mean, you hit the nail on the head there. It's, uh, there are manual signals. This is the same case with say asset labels, like file labels, uh, highly sensitive stuff and not sensitive stuff. So, in both cases, like we want the clear signals. When the customers use our plugins or a compliance solution to tell us that, you know, here's an HR event that's about ready to happen. Like the person's leaving or this file's important. We are definitely gonna take that and we're gonna use it. But that's sort of like the scientists wanna go further. Like what about the stuff they're not labeling? Does that mean they just haven't got around to it? Or does that mean that it's really not important? Or like you just said, like, this guy is starting to email recruiters a lot, this is like, is he getting ready to leave? So, there's definitely behavioral type detection and inference that, uh, we're working on behind the scenes to try to augment what the users are already telling us explicitly. Natalia Godyla:So, what's the reality of insider risk management programs? How mature is this practice? Are folks paying attention to insider risk? Is there a gap here or is there still education that needs to happen? Rob McCann:Yeah. So, there has been people working on this a lot longer than I have, but I do have to say that things are escalating quickly. I mean, especially with modern workforce, right? The perimeter is destroyed and everybody's at home and it's easier to do damage, right? And risk is everywhere, but some, you know, cold, hard numbers, like the number of incidents are going up, b- like, over the last two years. But I think Gardner just come out and said in, in the last two years, the number of incidents have went out by about half. So, the number of incidents are happening more probably, maybe 'cause of the way we work now. The amount of money that people, uh, companies are spending to address this problem is going up. I think Gardner's number was, when, uh, the average went up several million over the last couple of years, um, they just sort of released an insider risk survey and more people are concerned about it. So, all the metrics are pointing up and it just makes sense with the way the world is right now. Nic Fillingham:Where did sort of insider risk start? What's sort of the, the beginning of this solution... what did the sort of incubation technology look like? Where did it start? Uh, are you able to talk to that? Rob McCann:I mean, sure. A little bit. So, this was before me, so a lot of this came out of, uh, DSRE, which is our, our sort of internal security team for, at Microsoft babysitting our own network. So, they had to develop tools to address these very real issues, and the guys that I did a podcast with before Tyler Mirror and, and Robin, they, um, they sorta, you know, brought this out and started making it a proper product to take all these technologies that we were using in-house and try to help turn them into a product to help other people. So, it sort of organically grew out of just necessity, uh, in-house. But as far as like industry, like, uh, Carnegie Mellon, uh, certain National Insider Threat Center and I think they've been, uh, studying this problem for over a decade.Nic Fillingham:And as a solution, as a technical solution, did it start with like, sort of basic heuristics and just looking for like hard coded flags and logs, or did it actually start out as a sort of a data science problem and, you know, the sort of basic models that have gotten more sophisticated over time? Rob McCann:Yeah. So, it did start, start out with some data science at the beginning as well. Uh, so of course he always have the heuristics. We do that in external attack too. Heuristics are very precise, they, uh, allow us to write down things that are very specific. And they're very, very important part of the arsenal. A lot of people diss on heuristics hero sticks, but it's a very im- very important part of that, that thing. But it also has, it started out with some data science in it, you know, the anomaly detection is a big one. Um, and so there were already some models that they brought right from, uh in-house to detect when stuff was suspicious. Natalia Godyla:So, what Natalia Godyla:... what's the future of IRM look like? What are you working on next?Rob McCann:Well, I mean, we could, you could go several ways. You know, there could be broadness of different types of risk. The thing that I enjoy the most is sort of the more sophisticated ways of doing newer algorithms, maybe for existing charters, or maybe broad charters. Rob McCann:Uh, one thing that, I- I'm very interested in lately is the sort of interplay between supervised learning and, and anomaly detection. So you can think of as, uh, semi-supervised. That's a thing that we've actually been playing with at Microsoft for, for a long time. Rob McCann:I've had this awesome, awesome journey here. I've, I've always been on teams that were sorta, like ... It's kinda like I've been an ML evangelist. Like, I always get to the teams right when they're starting to do the really cool tech, and then I get to help usher that in. So, I got to do that in the past with spam filtering, when that was important. Remember when Bill Gates promised that we were gonna solve spam in a, in two years or whatever. Those were some of the first ML models we ever did i- in Microsoft products, and even back then we're playing with this intersection of, you know, things look strange, but I know that certain spam looks like this, so how do you combine that sort of strangeness into sort of a semi-supervised stuff ...Rob McCann:That's the stuff that really floats my boat is ho- how do you, how do you take this existing technology that some people think of as very different ... There's unsupervised, there's supervised, uh, there's anomaly detection. How do you take that kinda stuff and get it to actually talk to each other and do something cooler than you could do on one set or the other? That's where I see the future from a technical standpoint behind the scene for smarter detectors, is how we do that kind of stuff. Rob McCann:Product roadmap, it's related to what we're, we talked about earlier about the industry agreeing on threat major sees and customers telling us what's the most important to them. That, that's stuff's gonna guide, guide the product roadmap. Um, but the technical piece, there's so much interesting work to do.Natalia Godyla:When you're trying to make a hybrid of those different models, the unsupervised and supervised machine learning models, what are you trying to achieve? What are the benefits of each that you're trying to capture by combining them?Rob McCann:Oh, it's the story of semi-supervised, right? I have tons and tons of data that can tell me things about the distribution of activity, I just o-, d-, only have labels on a little bit of it. So, how do I leverage the distributions of activity that's unlabeled with the things that I can learn from my few labeled examples? And how do I get those two things to make a better decision than, than either way on its own? Rob McCann:It's gonna be better than training on just a few things in a supervised fashion, 'cause you don't have a lot of data with labels. So you don't wanna throw away all that distributional information, but if you go over to the distributional information, then you might just detect weirdness. But you never actually get to the target which is risky weirdness, which is two different things.Nic Fillingham:Is the end goal, though, supervised learning, so if you, if you have unsupervised learning with a small set of labels, can you use that small set of labels to create a larger set of labels, and then ultimately get to ... I'm horribly paraphrasing all this here, but, is that sort of the path that you're on?Rob McCann:So, we're gonna try to make the best out of the labels that we can get, right? But, I don't think you ever throw away the unsupervised side. Because, uh, I mean, this c-, this has come up in the external security stuff, as well, is if you're always only learning how to catch the things that you've already labeled, then you're never gonna really s-, be super good at detecting brand new things that you don't have anything like it. Right?Rob McCann:So, you have to have the ... It's sorta like the Explore-exploit Paradigm. You can think of it, at a very high level you can think of supervised as you're exploiting what you already know, and you're finding stuff similar to it. But the explore side is like, "This thing's weird. I don't know what it is, but I wanna show it to a person and see if they can tell me what it is. I wanna see if they like that kinda stuff."Rob McCann:Uh, that's sorta synergy. That's, that's a powerful thing.Nic Fillingham:What's the most sophisticated thing that the IRM solution can do? Like, have you been sort of surprised by the types of, sort of, anomalies that can be both detected and then sort of triaged and then flagged, or even have automated actions taken? Is there, is there a particular example that you think is a paramount sort of example of what, what this tech can do?Rob McCann:Well, it's constantly increasing in complexity. First of all, anybody who's done applied science knows how hard it is to get data together. So when I work with the IRM team, first of all, I'm blown away at the level of the breadth of signals they've managed to put together into a place that we can reason over. That is such a strong thing. So the, their data collection is super strong. And they're always doing more. I mean, these guys are great. If I come up with an idea, and I say, "Hey, if we only had these signals," they'll go make it happen. It is super, super cool.Rob McCann:As far as sophistication, I mean, you know, we start, we start with heuristics, and then you start doing, like, very obvious anomaly detection, like, "Hey, these, this guy just blew us out of the water by copying all these files." I mean, that's sort of the next level. And then the next level is, uh, "Okay, this guy's not so obvious. He tries to fly under the radar and sort of stay low and slow. But can we detect an aggregate? Over time he's doing a lot of damage." So those more subtle long-term risks. That's actually something we're releasing right now.Rob McCann:Another very powerful paradigm that we're releasing right now is, not just individual actions, but very precise sequences of actions. So you could think of it in a external as kill chain. Like, "They did this, and then they did this, and then they did this." That can be much more powerful than, "They did all three of those separately and then added together," if you know what I mean.Rob McCann:So that sort of interesting sequences thing, that's a very powerful thing. And once you sorta got these frameworks up, like, you can get arbitrarily sophisticated under the hood. And so, it's not gonna stop.Nic Fillingham:Rob, you talked about working on spam detection and spam filters as previous sort of projects you were working on. I wonder if you could tell us a little bit about that work, and I wonder if there's any connective tissue between what you did back then and, and IRM.Rob McCann:Yeah, so I've worked on a lot more than spam. So, I got hired to do spam, to do the research around the spam team, but it quickly, uh, it was this newfangled ML stuff that we were doing, and, uh, it started working on lots of different problems, if you can imagine that. And so we started working on spam detection, and, and phish detection. We started working on Microsoft accounts. We would, we would look at how they behave and try to detect when it looks like suddenly they've been compromised, and help people, you know, sort of lock down their accounts and get, and get protection.Rob McCann:All those things it's been cool to watch. We sorta, we sorta had a little incubation-like science team, and we would put these cool techniques on it and it would start working well, and then they've all sort of branched out into their own very mature products over the years. A- and they're all based very heavily on, uh, the sort of techniques that, that have worked along the way.Rob McCann:It's amazing how much reuse there is. I mean, I mean, let's boil down what we do to just finding patterns in data that support a business objective. That's the same game, uh, in a lot of different domains. So, yes, of course, there's a lot of overlap.Nic Fillingham:What was your first role at Microsoft? Have you always been in, in research on applied research?Rob McCann:I have always been a spoiled brat. I mean, I, I just get to go work on hard problems. Uh, I don't know how I've done it, but they just keep letting me do it, and it's fun. Uh, yeah, I've always been an applied researcher.Nic Fillingham:And that, you said you joined about 14 years ago?Rob McCann:Yep. Yep, yep. That was even back before, uh, the sort of cluster machine learning stuff was hot. So we, I mean, we used to, we used to take, uh, lots of sequel servers and crunch data and get our features that way, and then feed it into some, like, single box, uh, learning algorithms on small samples. And, like, I've got to see this progression to, like, distributed learning over large clusters. In-house first, we used to have a system called [Cosmos In-House 00:28:04]. I actually got to write some of the first algorithms that did machine learning on that. It was super, super rewarding. And now we have all this stuff that we release to the public and Azure's this big huge ... It's a very, very cool to have seen happen.Nic Fillingham:Giving the listener maybe a, uh, a reference point for, for your entry into Microsoft-Rob McCann:(laughs)Nic Fillingham:... is there anything you worked on that's either still around, or that people would have known? I think, like, just the internal Cosmos stuff is, is certainly fascinating. I'm just wondering if there's a, if there's a touchstone on the product side.Rob McCann:Spam filtering for Hotmail. That was my first gig.Nic Fillingham:Nice! I, I cut my teeth on Hotmail.Rob McCann:Yeah, yeah-Nic Fillingham:Yeah, I was a Hotmail guy. I was working on the Hotmail team as we transitioned to Outlook.com.Rob McCann:Mm-hmm (affirmative).Nic Fillingham:And I was, uh, down in Palo Alto, I can't even remember. I was somewhere, where- wherever the Silicone Valley campus is-Rob McCann:SVC-Nic Fillingham:We were rolling like a boar-, a boardroom waiting for the new domain to go live, and we got, like, a 15 minute heads-up. So I'm just Nic@Outlook.com. That's, that's my email address, and I got, I got my wife her first name at Outlook.com. Nic Fillingham:Were you there for that, Rob? Do you have a, did you get a super secret email address?Rob McCann:I was not there for the release, but as soon as it was out, I went and grabbed some for my kids. So I w-, I keep my Hotmail one, 'cause I've had it forever, but, uh-Nic Fillingham:Yeah.Rob McCann:... I got all my kids, like, the, the ones they needed. So.Rob McCann:It's amazing how much stuff came out of the, that, that service right there. So I talked about identity management that we do for Microsoft accounts now. I, that stuff came from trying to protect people, their Hotmail accounts. So we would build models to try to determine, like, "Oh, this guy's suddenly emailing a bunch of people that he doesn't usually," anomaly detection, if you can imagine, right? The-Nic Fillingham:Yeah-Rob McCann:... same thing works. Rob McCann:All that stuff, and then it sorta grew in, and then Microsoft had a bigger account, and then that team's kinda like, "Hey, you guys are doing this ML to detect account compromise, can you come, like, do some Rob McCann:... of that over here," and then it grew out to what it is today. A lot of things came from the OML days, it was very fun.Natalia Godyla:Thinking of the different policies organizations have and the growing awareness of those policies, over time, employees are going to shift their tactics. Like you said there are some who are already doing low and slow activities that are evading detection, so, how do you think this is going to impact the way you try to tackle these challenges, or have you already noticed people try to subvert the policies that are in place?Rob McCann:Yeah, so that's the, that's the next frontier, which is w-, you know, why I said we started just getting into, like, the low and slow stuff. It's gonna be like all other security, it's gonna be, "These guys are watching this thing, I gotta try something different."Rob McCann:Actually that's a good motivation for the sort of the high-level approach we're taking, which is tons of signals, so there's not very many activities you could do. You could print, copy to USB, you could upload to something, you could get a third-party app that does the uploading for you. There's not very many avenues that you could do that we're not gonna be able to at least see that happening. Rob McCann:So you couple that with some, that mountain of data with some algorithm that can try to pick out, "This is a strange thing, and this is in the context of somebody leaving." It's gonna be an interesting cat-and-mouse, that's for sure.Natalia Godyla:Do you have any examples of places where you've already had to shift tactics because you're noticing a user try to subvert the existing policies? Or are you still in the exploration phase trying to figure out what really, what this is really going to look like next?Rob McCann:So, right now I don't think we've had ... We haven't got to the phase yet where we're affecting people a lot. Uh, this is very early product, we're a year in. So, I don't see the reactions yet, but I, I guarantee it's gonna happen. And then we're gonna learn from that, and we're gonna say, "Okay, I have the Explore-exploit going. The Explorer just told me that something strange that I've never seen before happened." We're gonna put some people on that that are experts that figure out what that's gonna be. We're gonna figure out how to bring that into the fold of agreed-upon bad stuff, so we're gonna expand this threat matrix, right, as we go along? And we're gonna keep exploring. And that's the same for every single security product.Nic Fillingham:Rob, as someone that's been able to sort of come into different teams and, and different solutions and, and help them, as you say, sort of bring more academic or theoretical research into, into product, what techniques are you keeping your eye on? Like, what's, what's coming in the next two or three years, maybe not necessarily for IRM, maybe just in terms of, as machine learning, as sort of AI techniques are evolving and, and, and sort of getting more and more mature, like, what, where are you excited? What are you, what are you looking at?Rob McCann:So you want the secret sauce, is what you're asking for?Nic Fillingham:That's exactly what I want. I want the secret sauce.Rob McCann:(laughs) Um, well, I mean, there's two schools of thought. There's one school of thought which is, "You better keep your finger on the pulse, because the, the new up-n-comers, the whippersnappers are gonna bring you some really cool, cool stuff." And then there's the other school of thought which is, "Everything they've brought in the last ten years is a slight change of what they, was before, the previous ... It's a cycle, right, as with s-, i- ... Science is refinement of existing ideas.Rob McCann:So, I'm a very muted person that way, in that I don't latch on to the next latest and greatest big thing. Um, but I do love to see progress. I s-, just see it as more of a multi-faceted gradual rise of mankind's pattern-recognition ability, right?Rob McCann:Things that excite me are things that deal with ... Like, big data with big labels? Super, super cool stuff happening there. I mean, like, you know, who doesn't like the word deep learning, or have used it-Nic Fillingham:What's a big label? Is there a small label?Rob McCann:(laughs) No, I mean lots of labeled data. Like, uh-Nic Fillingham:Okay.Rob McCann:... yes.Nic Fillingham:Big data sets, lots of labels.Rob McCann:Yes. That stuff, um, that's exciting. There's a lot of cool stuff we couldn't do two decades ago that are happening right now, and that's very, very powerful. Rob McCann:But a lot of the business problems in security, especially, 'cause we're trying to always get this new thing that the bad guys are doing that we haven't seen before. It's very scarce label-wise. And so the things that excite me are how you inject domain knowledge, right? I talked about, we want customers to be able to sort of control on some knobs that you, like, focus the thing on what they think's important. Rob McCann:But it also happens with security analysts, because, there's a lot of very smart people that I get to work with, and they have very broad domain knowledge about what risks look like, and various forms of security. How do you get these machines to listen to them, more than them just being a label machine? How do you embed that domain knowledge into there?Rob McCann:So there's a lot of cool stuff happening. Uh, in that space, weak learning is one that's very popular. Came out of Stanford, actually. But I'm very la-, I'm very, very excited about what we can do with one-shot, or weak supervision, or very scarce labeled examples. I think that's a very, very powerful paradigm.Nic Fillingham:Doing more with less.Rob McCann:That's right. Rob McCann:And transfer learning, I'm sure you guys have talked to a lot of people about that. That's another one. A lot of things we do in IRM ... Well, in, in lots of security is you try to, like, leverage labeled, uh, supervised classification ... Like, think about HR events. Rob McCann:So, maybe I could, don't have a m-, a bunch of labeled, "These are IRM incidents" that I can train this big supervised classifier on. But what I can do is I can get a bunch more HR events, and I can learn things, like you said, that predict that an HR event is probably happening, right? And I chose that HR event, because that's correlated with the label I care about, right? So, I can use all that supervised machinery to try to predict that proxy thing, and then I can try to use what it learned to get me to what I really want with maybe less labels.Nic Fillingham:Got it. My final IRM question is, from what I know about IRM, it feels like it's about protecting the organization from an employee who may maliciously or accidentally do something they're not meant to do. And we've used the example of an employee getting ready to leave the company. Nic Fillingham:What about, though, IRM as a tool to spot well-meaning, but, but practices that, that o-, expose the company to risk? So instead of, like, looking for the employee that's about to leave and exfil 50 gigs of cat meme data that they shouldn't, what about, like, just using it to identify, "You know what, this team's just sort of got some sloppy practices here that's sort of opening us for risk. We can use the IRM tool to go and find the groups that need the, sort of the extra training, and to, need to sort of bring them up to scratch. And so it's almost more of a, um, just thinking of it more in sort of a positive reinforcement sense, as opposed to sort of an avoiding a negative consequence.Nic Fillingham:Is that a big function of IRM?Rob McCann:Yeah, I mean, I, I'm sorry if I didn't, uh, communicate that well, but, IRM is definitely intentional and unintentional. In s-, in some of the workflows the way you can do when we detect risky activity is just send an email to the, uh, to the employee and say, "Hey, this behavior is risky, change your ways, please," right? Rob McCann:So, you're right, it's, it can be a coaching tool as well, it's not just, "Data's gonna leave," right? Intentionally.Nic Fillingham:Got it. You've been very generous. This has been a great conversation. I wondered, before you leave us, do you have anything you would like to plug? Do you have a blog, do you have a Twitter? Is there a- another podcast? Which one were you on, Rob?Rob McCann:Uncovering Hidden Risk. I would also like to point you guys to, uh, an inside risk blog. I mean, we, we publish a lot on, on what's coming out and where the product is headed, so it's: aka.ms/insiderriskblog. That's a great place to sorta keep abreast on the technologies and, and where we wanna go.Nic Fillingham:That sounds good. Well, Rob McCann, thank you so much for your time. Uh, this has been a great conversation, um, we'll have to have you back on at some point in the future to learn more about weak learning and other th-, other sort of, uh, cool new technique you hinted at.Rob McCann:Yeah. I appreciate it. Thanks for having me.Rob McCann:(music)Natalia Godyla:Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.Nic Fillingham:And don't forget to tweet us @msftsecurity or email us at securityunlocked@microsoft.com with topics you'd like to hear on a future episode.Nic Fillingham:Until then, stay safe.Natalia Godyla:Stay secure.