Global Outage Alert: Windows BSOD Crisis Following CrowdStrike Update – Recovery Steps & Qualys Assurance

On Friday, July 19, 2024, morning, reports surfaced globally of Microsoft Windows operating system users encountering the infamous Blue Screen of Death (BSOD) following the latest update from CrowdStrike. This widespread issue has severely impacted critical services, including telecommunications, banking, airline and railway operations, supermarkets, hospitals, and major news networks.

For those customers concerned about this impact on Qualys, Qualys does not leverage any services from CrowdStrike, and there is no impact on our production environments.

As we have many joint customers with Crowdstrike, below is some information for those customers on how they can address the challenges caused by the issue.

Steps For Regaining Access to Windows PCs, AWS & Azure 

CrowdStrike has outlined a four-step process for regaining access to Windows PCs affected by the update:

1. Boot Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
3. Locate and delete the file matching “C-00000291*.sys”.
4. Boot the host normally.

For cloud environments, customers can revert to a snapshot taken before 4:09 am UTC.

For AWS (Amazon Web Services), follow these steps:

1. Detach the EBS volume from the impacted EC2 instance.
2. Attach the EBS volume to a new EC2 instance.
3. Fix the CrowdStrike driver folder.
4. Detach the EBS volume from the new EC2 instance.
5. Attach the EBS volume back to the impacted EC2 instance.

For Azure, follow these steps:

1. Log in to the Azure console. 
2. Go to Virtual Machines and select the affected VM. 
3. In the upper left of the console, click “Connect”. 
4. Click “More ways to Connect” and then select “Serial Console”. 
5. Once SAC has loaded, type in ‘cmd’ and press Enter. 
6. Type ‘ch -si 1’ and press the space bar. 
7. Enter Administrator credentials. 
8. Type the following commands: 
   • ‘bcdedit /set {current} safeboot minimal’ 
   • ‘bcdedit /set {current} safeboot network’ 
9. Restart the VM. 
10. To confirm the boot state, run the command: ‘wmic COMPUTERSYSTEM GET BootupState’. 

The manual nature of this fix poses a significant challenge for companies, especially those without backups for all VDIs, potentially slowing down the recovery process. Customers will also need a recovery key to access Safe Mode if Bitlocker is enabled on the system disk. 

CrowdStrike Engineering has reversed the changes causing this issue, with the error code displayed on affected systems: “Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19.“

Discover Assets Running CrowdStrike Falcon Using Qualys CyberSecurity Asset Management (CSAM) 

The initial and crucial step in managing this incident involves pinpointing all assets where Falcon sensor is installed. Use CSAM 3.0 with External Attack Surface Management to identify assets with Falcon. 

software:(name:”Falcon” OR name:”CrowdStrike”) 

You can also use asset.lastUpdated: token with the timestamp based on your time zone in addition to software:(name:”Falcon” OR name:”CrowdStrike”) to identify assets with Falcon that have stopped communicating after the incident and may still be down. 

Qualys Assurance – No Impact on Our Services

We want to reassure our customers that Qualys does not leverage any services from CrowdStrike, and there is no impact on our production environments (shared and private platforms), codebase, customer data hosted on the Qualys Cloud Platform, Qualys Agents or Scanners, and Qualys’s corporate infrastructure. All Qualys platforms and agents remain fully functional and operational.

In light of these recent global outages, we want to highlight the safety and reliability of the technologies used in the Qualys Agent for Vulnerability Management, Policy Compliance, Software Composition Analysis, Cloud Agent Passive Sensor, Custom Assessment and Remediation, Patch Management, and Global AssetView. Developed in user mode, our approach does not interfere with the operating system kernel thus minimizing the risk of impacting customer assets, ensuring uninterrupted service.

Qualys enforces stringent policies and procedures against newly introduced detections before publishing to production. Automated checks are in place to prevent any potentially malicious commands since the last build from being published to production environments. Commands like rm, kill, and mv are carefully monitored to ensure safety.

For Qualys’ Endpoint Detection and Response and File Integrity Monitoring solutions, which do involve a kernel driver, all detections, content, signatures, and data collections are non-intrusive and not resource intensive. Our updates (For Agent Binaries, Signatures, Manifests, Backend and other components) are released in a staggered manner, beginning with our internal deployment and subsequently extending to a group of customers to reduce the broad impact of an unfortunate event. Before any code release, each update undergoes comprehensive security and functional testing, including static and dynamic analysis.

Our testing protocols include (not limited to):

1. Conducting static code/binary analysis of new features and code changes.
2. Performing driver load-unload tests on all supported Windows platforms, including x86 and x64 architectures, both on servers and client workstations.
3. Assessing performance impact using custom scripts to test the performance impact on both the system and applications by toggling kernel components on and off.
4. Conducting longevity and performance tests in various modes (burst, high, normal) before signing off agent and driver packages.

Agent Version Control: Ensuring Consistency and Security

Agent Version Control, an innovative feature of the Qualys Cloud Agent, offers enhanced control and flexibility, facilitating efficient management and update of agents and ensuring alignment with the latest security advancements.You can create Agent Version Control Profiles to lock specific platforms (e.g., Windows, Linux) to the organizationally certified agent version. Choose the timing and method of updates to align seamlessly with your operational needs, minimising disruptions to daily activities while maintaining high security standards to maintain a stable, secure and well-regulated IT environment.

At Qualys, we remain committed to providing solutions that empower you to protect what matters most, ensuring business continuity. We strongly recommend that organizations verify they are engaging with authorized vendor representatives through official channels. We understand that many organizations have been impacted and are working hard during this challenging time to restore their normal operations; we are dedicated to actively assisting wherever we can.

If you have any questions about the Qualys Cloud Agent or need any assistance regarding this incident, please contact your TAM or support.

Get the latest updates from CrowdStrike for this outage.