Surprising Lessons from Incident Response Testing

When LMG Security’s consultants work with clients on a security program review, we always ask about whether they test their organization’s incident response preparedness. Having an incident response plan is step one, but regular incident response testing is essential for training staff, maintaining familiarity, identifying gaps, and improving your response readiness.

Our security consultants regularly assist clients by facilitating incident response tabletop exercises, where the consultant presents an incident scenario and encourages the client participants to walk through response actions and decisions to test response preparedness. Even clients that have a solid incident response plan, find gaps during incident response simulation exercises.

Through our experience with these exercises, our team has encountered a number of situations where such tests revealed results that were surprising to our clients, both on the management and process side of response, as well as technical surprises. The frequency with which we see these surprises is a clear indicator of the importance and value of regular testing.

Common Questions & Gaps Uncovered During Incident Response Testing

Here are a few examples of common challenges uncovered during incident response testing:

• Who authorizes decisions? Often, the IT team was not sure if they were authorized to take containment steps that would have an operational impact, such as disabling accounts or taking key systems offline. Further, they did not know who on the IR team could authorize such actions. Any confusion over decision-making authority can create delays and increase the potential spread and overall impact of a security incident, since speed is essential for effective containment.
• When do we escalate? Incident response testing can reveal communication gaps that need to be addressed. For example, the IT team at a small company insisted on investigating suspicious activity quite thoroughly before escalating this issue and letting the management team know something was going on. To their credit, the team wanted to be sure they could provide as much information as possible when informing management. However, the management team was insistent they wanted to be made aware of a potential incident right away, even if details were limited. When the IT Lead asked the CEO: “But what if it turns out to be nothing?” The CEO stood up and replied enthusiastically, “Well that will be wonderful!”
• To pay or not to pay? Ransomware scenarios often lead to interesting discussions. Our consultants often see disagreement among management teams on whether or not to pay the ransom, often due to ethical objections to paying criminals. While debate is healthy and essential, an actual ransomware incident usually has a quick timeline for making decisions. It is not the time for extended discussion of ethical concerns and options. It is much better to have that conversation during a tabletop exercise, when your organization’s data operations are not actually at risk. (Read more on Ransomware payment recommendations.)
• We have backups, right? During one incident response testing exercise, we found one organization had an unusually short back-up retention period – less than a week! We were not the only one who was surprised; the management team was under the impression that their retention periods were longer and followed best practices. Discussion during the exercise identified limited storage capacity as the problem and led to budgeting to support additional capacity.
• Don’t we have that technology? Talking through incident detection and response can help management teams understand their organization’s technical capabilities and limitations. Here are a few examples we uncovered during incident response testing that have surprised management teams:
  • Limited detection capability – We have heard about detection tools that have not been fully implemented or properly tuned to provide needed visibility, such as an IDS/IPS purchased and plugged in, but not yet put inline to actually monitor traffic. Other gaps include lack of alerting, no one assigned to review alerts, and IDS/IPS operating only as IDS, while management thought it was blocking suspicious traffic, not just detecting it.
  • Audit logs – Some systems are not capturing audit logs, the logs are not retained long, or they are not included in monitoring via the SIEM.
  • Skills gaps – Organizations may have one person on the team with needed knowledge on their security tools and how to use them. This can be related to reviewing firewall or IDS/IPS activity, retrieving audit logs, or creating and preserving forensic images. A tabletop exercise can help bring these gaps to light so that cross-training can be planned.
• We have known cybersecurity gaps and vulnerabilities? Incident response testing exercises can also help identify a variety of security control gaps. In many cases, the gaps are known to the IT team, but management was not aware and therefore remediation was not planned and prioritized. Some examples:
  • Unauthorized use of cloud storage
  • Sensitive information stored locally, against policy
  • Lack of technical enforcement of strong password requirements
  • Local administrator rights on some endpoints
  • Delayed patching or obsolete operating systems
• What does our cybersecurity insurance cover? Cybersecurity insurance is a frequent topic during an incident response testing exercise, and we often see challenges in this area. A common scenario is where an organization has cybersecurity insurance, but no one on the response team has good knowledge of what it covers, when it should be activated, or how to activate it. In some cases, the organization does not have clear guidance on who can make decisions around when to activate insurance coverage or who is authorized to submit a claim. These gaps can cause delays or even cause an organization to miss out on response services their insurance can help with, such as forensic investigation services or media response support.

Final Thoughts on Incident Response Testing

Now that you know the most common surprises companies find during incident response testing, here is what you should know about having an incident response testing plan:

• Test your incident response plan regularly (annually is a good rule of thumb) to avoid unpleasant surprises when you experience a real incident. (Read more about Business Continuity and Disaster Recovery (BCDR) planning.)
• Encourage a security-conscious culture by asking all employees to report any known or suspected vulnerabilities or gaps in your organization’s security controls for investigation and remediation. Send periodic reminders to your team to ensure they remember.
• Consider hiring a third party to facilitate your incident response testing exercise. Having an experienced external expert in the room can increase the training benefits and reveal gaps that you did not consider.

If you need help with incident response planning or testing, or if you experience an incident and need incident response support, contact us. Our expert team can help.