Embedding Incident Response into the DNA of the Organization

Posted 9 years ago by Sean Mason

In high school I used to compete as a swimmer and my favorite event was the 50yd freestyle. I trained an average of 4 hours per day (2hrs in the AM, and 2hrs in the PM) and during practice, we would break down each movement- the start, hand movements, the flip turn, breathing, etc… and practice for hours. When a competition finally came around, we were well prepared and my fastest 50yd free time was 19.71 seconds; I don’t even recall swimming this race because it was over so quick. There was barely any thought to the race and everything was muscle memory- my body was simply doing what it knew how to do from the hours, days, weeks and months of practice- it was embedded in my DNA. Incident Response is no different; to truly be successful, IR needs to become muscle memory and ingrained in the DNA of the organization.

There is a misconception that having an IR plan will suffice and the statistics seem to indicate having a plan is on the rise. While having a plan is great, they are rarely more than just guidelines and are not the robust set of company specific procedures they should be, especially if you don’t have people practicing them day in and day out. In many cases the plan does not even have buy-in from the many departments (e.g. legal, communications, privacy, etc…) needed to make it successful.

To truly be successful, an organization needs to have put in the leg-work required to build out an organizational capability that can be relied upon in a time of crisis. The People, Elasticity, Organizational Make-up, and more will all come into play and having a team that actually understands and can execute the processes involved at each and every step of the IR plan is paramount to being able to respond successfully- and the only way to accomplish that is through repetition and practice. For example, too many times I hear a CISSP say, “Containment” without having any understanding of what is involved in containing a device; what containment mechanism will be used, will it successfully block the C2, how will it impact forensic evidence, the investigation, or perhaps even more importantly, the business.

This brings me to the second half of my post. If you, the reader, are content to outsource your SOC and IR responsibilities, the following isn’t necessarily for you. If you are either looking to build-out or perhaps already have a team in place, the following are my thoughts on the key aspects of what an organization needs to make IR part of the DNA of the information security team.

People
Having seen more than anyone’s share of truly interesting incidents, I get asked the question a lot, “What would you prioritize over anything in the security realm.” With the assumption that [required] compliance is off the table- the answer is people. If you have only one spare dollar, it should go to investing in people. A recent Ponemon survey stated that 73% of organizations have 1 FTE or less dedicated to Intel-Detection-Response activities, and this is a huge disappointment. My recommended minimum would be a team of 3-4 FTE solely dedicated to detection and response activities (and not all Senior folks- see below) which will allow you to learn and build out from there. It is amazing what you can accomplish if you put a couple of smart people together with the sole focus of finding evil. Additionally, don’t forgo training dollars for both yearly classes as well as time during the week for your IRT to self-study and teach each other.

Process
A good incident response plan will include countless processes and their related subprocesses that will address various situations (not all) that may be encountered. Many times I’ve heard the term, “Cowboy Up” in relation to Incident Response and while it may sound cool, the last thing you should be doing is inventing process on the fly. “The more you sweat in peace, the less you will bleed in war” is good advice to follow and refining your processes based off of constant learning is critical to success. For example, per my earlier commentary on Containment and the related chart, the team should know the different containment options, what they may look like, when they can be used, timing required, and what the impact may be. The best way for the team to understand the processes in detail and commit them to muscle memory is to spend time designing them, and then testing them in both test and production environments as often as possible.

Prioritization
“Not all incidents are created equal” and any company of any size has an “incident” on their hands right now. It is only the incidents of true impact that should truly raise alarm bells internally though. Unfortunately, I see too many organizations getting mired in the details of “common malware” or other low hanging fruit, spending precious resources and “crying wolf” when responding to a no-impact infection of malware. While it is definitely useful to practice good network hygiene and clean these issues up, they should not be blown out of proportion. However, at the same time, lower risk incidents should be looked at as an opportunity for the IRT to continually practice their incident response skills (containment, collection, analysis, etc…) and adjust processes, procedures, and posture based on what is learned and slipping through the defenses. This aids in turning IR into an exercise in muscle memory.

Organization
This is one of the toughest issues out there, because as a leader and a technologist, you always want everyone to know everything wing-to-wing, and while this can work great in a small organization the reality is that it doesn’t scale for a number of reasons in larger orgs. Think of the information security generalists you know- they can speak on pretty much every subject, but like the CISSP they are only a mile wide and an inch deep. If you’re operating at scale, you’re going to need to build out teams that specialize in certain aspects of DFIR- be it Intel, Detection, Triage (SOC), Reverse Engineering, and/or Analysis and allow them to own those aspects and be held accountable for the day to day delivery of those functions. While I’d like to think there is an easy solution to let everyone learn everything, such as cross-training or job movement every 6 months, it is unfortunately easier said than done in most large organizations due to HR policies.

I would also urge you to read Jack Crook’s (@jackcr) blog post discussing Silos in an IR organization and while his stance is contrary to mine, he makes great points that should be considered.

Organizational Sustainability
If an organization has a team of all senior folks, you may pat yourself on the back and think you’re set. Realistically though, people do leave companies and also look for other opportunities internally. Trying to find senior level talent in IR- let alone information security- right now is one of the most challenging jobs a leader has. Those 6-12 months wasted searching for a candidate that doesn’t exist, is 6-12 months you could be training a new junior or mid-career individual. Having a good team mix of senior, junior and mid-career talent will enable your organization to be sustainable for the long run, allow career path opportunities, provide opportunities to non-IR folks to join the team and learn, provide fresh ideas, and also cushion the impact of losing talent. A sustainable organization shows evolution to a point where it is part of the DNA of the information security program.

Incidentally, if you are skeptical that junior folks can make an impact in this space, check out two fantastic individuals only a couple years out of college:

Public Service Announcement: Can you imagine if every information security team out there offered one internship and also hired one entry-level person per year? There is negative unemployment in information security for a reason- move the ball forward and change the game for the industry.

Organizational Elasticity
Most IR organizations aren’t staffed to be able to handle one incident, let alone multiple incidents at one time. Per the advice above on having a diverse team, look to leverage your mid-career folks as the leaders when multiple incidents are on-going and have them cut their teeth on the less important and less visible incidents. Additionally, creating non-technical roles such as an “Incident Coordinator” who can essentially Project Manage incidents for you will take a lot of burden off of your more technical responders and allow for more throughput and better task management- would you rather your senior forensic analyst build slides for an Executive meeting or look for more indications of lateral movement? As a benchmark, we had two Incident Coordinators during my tenure at GE.

Cross-Team Collaboration & Transparency
Never operate in a vacuum! Break down those silos, especially on the incident response team if you have multiple teams sharing the work. Leadership is critical here and the last thing a good IR Team needs is leaders who operate with the philosophy that, “This is my domain.” IR is a team sport and you can only be successful both in the short and long term by working with numerous teams. Educate and enlist teams outside of the IR domain and have them help you for the betterment of the company. Make them aware of the attacks that you are seeing and the techniques being used- they may surprise you with clever approaches to combat the threats either immediately or with changing their longer term plans for the environment. Support cross-team projects within the larger information security team to tackle problems to help build that camaraderie at all levels. Ultimately, having the right leadership in place with the right mindset and ability to build external relationships is critical to success and ensuring IR is part of the fabric of the larger security program.

Metrics
Metrics are essentially the measurement of health for the IR capability and should never be overlooked- they should be used to help guide the team in the proper direction. Check out my previous post on IR metrics here.

 

Think big picture; Incident Response isn’t just a plan, it isn’t just a breach notification, it isn’t just intel, and it isn’t just forensics- it is all of these things and more. Anything done right takes a lot of work, and this is no different. Incident Response practiced daily and supported correctly will turn into Continuous Detection & Response- and in turn will become muscle memory- better positioning the company to be successful and quickly mitigate impact and recover in the event of an actual incident. With IR becoming ingrained in the DNA and responses being reduced to sheer muscle memory, there will most likely not even be a need to refer back to the IR Plan during the incident- and that is priceless.