I didn’t want to bring him this information, because I knew how quickly the conversation would go sideways, but I was required to do so… “We have an incident on our hands and we’re currently responding.” I said calmly. It wasn’t my first rodeo.
“Great! Contain the machines so we can stop the data breach and kick the hackers out. What threat intelligence do we have on this?” he responded a little too casually.
I ignored the countless flaws in his language- he must have been reading again- tossing terms around like data breach, hackers, and threat intel. Why was everything a data breach? We didn’t know anything at this point… and did he actually say the entire word- intelligence- “huh,” I thought to myself. Unfortunately, I couldn’t ignore his comment on containment, as that would be the first thing he would ask for an update on. “Containment will be tricky, it’s not a simple case, as we are only aware of this single domain contr-“, I began before I was quickly interrupted.
His voice elevated to an almost-shout, “It shouldn’t be that hard, you just contain it!” Luckily this was over the phone and he could not see my eyes roll back as I shut them slowly and took a deep breath, shaking my head before I exhaled.
While a bit dramatic, I’ve seen this scenario play out a number of times with numerous individuals, many are leaders and many more have a background in information security. Unfortunately, what I’ve learned over the years is that knowing the lingo, or having been through one or two “incidents” (yes, that is in quotes- there are significantly varying degrees of incidents), doesn’t necessarily qualify an individual to be making decisions or recommendations during an incident.
In this post, I plan to focus on Containment and the various options and decision points that should be considered before moving towards containing an incident.
Scope
There is a lot to be said about understanding the scope of an incident- including how quickly you can understand the scope- and how that ultimately impacts containment. If you ask any tried and true incident responders, you’ll generally get two different answers based on their experience and environment, though the better ones will understand both sides and the rational of when to use one versus the other. The first camp believes that keeping all hosts online and understanding the full extent of the compromise before doing a one-time mass contain is the optimal solution.
Others believe that playing ‘whack-a-mole’ and containing assets the moment you identify them is the optimal solution. Still others believe the answer lies somewhere in the middle. From my experience the answer is, “it depends,” and two examples follow:
I have seen data exfiltrated while consultants have been working on scoping an incident for a couple of weeks. Having to explain to the C-suite that nothing was done (e.g. contained) with what was already known is a losing conversation when data is stolen.
Containing devices the moment you notice they are compromised can be a short path to victory, IF you have the capabilities to out maneuver the attackers by determining the scope of the incident quickly. This typically requires extremely fast collection and analysis capabilities, of which only a very mature set of companies I’ve met have the ability to perform. Otherwise containing device by device will just alert the attackers to your presence and they can (and usually will) begin compromising other assets.
Ultimately this decision is going to rest on your C-suite’s appetite for risk as well as the capabilities and track record of your team.
Device Type
When it comes to the device, there are three main aspects that need to be taken into consideration, as they will affect the outcome and speed of the investigation.
1. Is the machine a Desktop, Laptop, or Server? Servers and Desktops generally don’t grow legs and walk off- and are usually static in nature. Knowing a device won’t move opens up different options such as network related blocks. Laptops on the other hand may require you to consider movement and different connectivity concerns such as being off network, auto-connecting Wi-Fi at home, etc… and generally will require a host-based approach to containment.
ProTip: Never trust a user to simply power down a device and keep it in that state.
2. What operating system is the device running? Windows can lend itself to generally be straightforward to contain, but some containment mechanisms don’t port over to Mac and *nix platforms. That said, devising common and reusable methods to contain devices based on OS, and having those solutions readily available is critical.
3. Back to our story- is it a Domain Controller? Web Server? A business may not have the ability to simply pull the box offline without a massive impact to either users or customers; thusly, if your compromised device plays a critical role, the answer may not be black and white, as a business leader may not understand or even accept the business risk that will be introduced by the solution. In some cases, the IRT may need to get creative in containing the device such as neutering the malware. If you can neuter the malware and maintain eyes on the adversary while working the incident, that may be a better middle-ground option that is a more palatable and effective means of containment.
Reachability
1. Is the asset currently online or offline? Given that responding to alerts is still not an immediate/real-time reality, coupled with most teams needing to take time to track down the actual device, there is a good chance the asset may have been taken offline by a mobile user. In this event, a host-based approach may not be immediately feasible and perhaps considering something such as disabling user credentials (e.g. AD, VPN) to prevent them from reconnecting to the network may make sense.
2. Containing a device is great, but what some responders may soon realize is that they themselves can no longer access a device. Dependent on the alert and your organizations capabilities, you may want to first collect evidence from the device to further understand the incident before knocking it offline. Don’t lock yourself out.
ProTip: Years ago when designing different containment options, we kept in mind that contact with a machine would have an impact. After careful consideration and testing, we came to the conclusion that making minor changes on the end-point, in support of containment, had a negligible, if any, impact on the evidence remaining for the team to analyze what happened.
Speed
Speed takes two forms in regards to containment- how quickly a team can get containment in place, and how quickly containment is needed. To touch on the second portion first, if the compromise is months/years old and there is zero evidence of current activity, do you really need to move to contain the device as quickly as possible? If you believe so, then what additional risks will you possibly introduce to the business, especially if the team moves too quickly and perhaps heavy-handedly to contain a business critical asset? In these situations, I feel it best to stop and scope the incident more broadly and perhaps even devise a custom containment solution for the specific situation.
In regards to how quickly you can execute a containment option, this will generally depend on the organizational makeup and alliances made ahead of time, let alone process documentation. For example, if the security team is requesting a network block, do they have an emergency exception process in place ahead of time with a phone number to call at off hours, 7 days a week? If not, you may find yourself fighting a process and political battle when you least expect it. Documenting all of your containment options ahead of time, building out processes across teams, and practicing those options to understand the lead time needed to get them in place will pay dividends down the road. Nobody ever wishes they didn’t spend their time preparing when an incident hits and they need to invoke their IR plan.
Incident Severity
Last, but not least, is the reality that not all incidents are created equal and neither should all containment options be as well. For example, if you are going up against nuisance malware, there may be options to contain the device without impacting the end user. Conversely, if data exfiltration is currently taking place, you may need to move quickly to contain- perhaps in a more heavy handed fashion- and minimize the damage. Again, planning ahead and having a decision tree handy to assist with how to contain a situation can be critical.
Final Thoughts
Incident containment isn’t simple, and is perhaps the hardest and most important decision that is made during an incident. Without taking time up-front to understand each facet of containment, documenting them, testing them, and also knowing what fallback options you may have available- there is a very likely possibility that you will fumble with such a critical decision during an incident. Additionally, without understanding the business impact containment will have, you may cause even more harm than good when taking into consideration the broader picture. And always remember, “the more you sweat in peace, the less you bleed in war.”