How Business Continuity and Disaster Recovery Plans Can Save the Day: Lessons from the CrowdStrike Software Outage

 https://techcrunch.com/2024/07/19/what-we-know-about-crowdstrikes-update-fail-thats-causing-global-outages-and-travel-chaos/

A recent software update from cybersecurity firm CrowdStrike has led to a global outage affecting Windows computers. The glitch in the Falcon Sensor software caused widespread disruptions across businesses, airports, banks, broadcasters, and healthcare systems.

What Happened?

Late Thursday into Friday, reports surfaced of Windows computers crashing and displaying the infamous "blue screen of death." The problem, originating with a faulty CrowdStrike software update, rapidly spread from Australia to other regions, including Asia, Europe, and the United States. Despite initial fears of a cyberattack, CrowdStrike confirmed that the issue was due to a defect in their update, not malicious activity.

Impact and Response

The outage has led to significant disruptions:

• Business Operations Many organizations are struggling with halted operations.
• Transportation Airports and train stations experienced delays and cancellations.
• Healthcare Some healthcare networks were temporarily disrupted.

Microsoft also reported a separate but unrelated outage affecting its Azure cloud services. Microsoft CEO Satya Nadella assured that the company is collaborating with CrowdStrike to provide technical support.

CrowdStrike’s Role

Founded in 2011, CrowdStrike has become a major player in cybersecurity, serving thousands of corporate customers, including many Fortune 500 companies. Their Falcon Sensor software is critical for managing and securing enterprise systems. This incident highlights the far-reaching impact of their software on global operations.

Affected Entities

The CrowdStrike outage impacts anyone using Windows systems with Falcon Sensor installed, including:

• Retail Cash registers and point-of-sale systems.
• Transportation Departure boards and ticketing systems.
• Education: School computers and administrative systems.
• Healthcare Patient records and hospital systems.

The Federal Aviation Administration even imposed a ground stop on U.S. flights due to the outage. While Amtrak services remain unaffected, many other sectors are experiencing significant disruptions.

Government and Agency Response

The U.S. government, including President Biden and various federal agencies, is closely monitoring the situation. Agencies like the Department of Education and Social Security Administration have been impacted, with some closing offices temporarily. The Cybersecurity and Infrastructure Security Agency (CISA) is working with CrowdStrike and other partners to assess the situation and provide assistance.

Fixes and Workarounds

CrowdStrike has issued a patch and provided a workaround for affected systems. Users can manually delete a defective file from their systems to restore functionality, but this process can be cumbersome, especially for organizations with large-scale deployments.

Security Concerns

CISA has warned that malicious actors might exploit the confusion caused by the outage. Social engineering expert Rachel Tobac advises vigilance against phishing attempts and impersonation scams, as criminals may use the outage as an opportunity to target organizations.

Misinformation and Confusion

The scale of the outage and the initial chaos led to widespread misinformation, with some incorrectly attributing the issue to a cyberattack. Social media has seen confusion and false claims about the nature of the incident, underscoring the challenges of managing information during such crises.

Business Continuity Planning (BCP)

BCP focuses on ensuring that critical business functions continue during and after a disruption. In the case of the CrowdStrike outage-

• Identification of Critical Functions BCP helps identify which business functions are crucial and must remain operational despite the incident. For organizations impacted by the outage, this might involve prioritizing operations that are essential for maintaining customer service or compliance.

• Alternative Processes BCP includes strategies for maintaining business operations using alternative processes or workarounds. During the CrowdStrike incident, this could mean switching to backup systems or manual processes to ensure that critical operations like transactions, communications, and customer support continue.

• Resource Allocation The plan helps in allocating resources effectively to manage the disruption. This involves ensuring that the right personnel and tools are available to address the immediate impacts of the outage and support recovery efforts.

• Communication Plans Effective communication is critical during an outage. BCP includes protocols for internal and external communication, ensuring that stakeholders are informed about the status of operations and recovery efforts. This helps manage expectations and maintain trust.

Disaster Recovery Plan (DRP)

DRP focuses on restoring IT systems and data after a disruption. In the context of the CrowdStrike outage-

• Incident Assessment DRP helps in assessing the extent of the damage caused by the outage. This includes identifying affected systems, understanding the nature of the fault, and evaluating the impact on data and operations.

• Recovery Procedures DRP outlines specific steps for recovering IT systems and restoring normal operations. For the CrowdStrike issue, this includes applying patches, removing defective files, and ensuring that affected systems are brought back online with minimal data loss.

• Backup and Restoration DRP ensures that there are backup systems and data that can be used to restore normal operations. In the event of data corruption or loss due to the software defect, having up-to-date backups allows for quicker recovery.

• Testing and Validation DRP involves testing and validating recovery procedures to ensure they work effectively in real scenarios. This includes regularly updating and testing recovery plans to handle similar incidents in the future

How They Work Together

• Integrated Approach BCP and DRP should be integrated, with BCP ensuring business functions continue and DRP focusing on IT system recovery. Together, they provide a comprehensive approach to managing and mitigating the impact of an outage.

• Minimizing Downtime While BCP aims to keep business processes running, DRP focuses on restoring IT systems. Both plans work in tandem to minimize downtime and operational disruptions.

• Communication and Coordination Both plans emphasize the importance of communication and coordination. BCP addresses how to communicate with stakeholders and manage operations, while DRP details the technical steps needed to restore systems.

In summary, a well-developed BCP and DRP can significantly improve an organization’s ability to manage and recover from incidents like the CrowdStrike outage. They provide structured approaches to maintaining operations and restoring systems, helping organizations navigate disruptions effectively and minimize the overall impact.