CrowdStrike BSOD outage affecting Windows Computers

There is a significant outage this morning affecting many computers. A recent update from CrowdStrike has resulted in a widespread outage, impacting numerous systems with a Blue Screen of Death. The issue stems from the Falcon sensor update. The outage is causing a Blue Screen of Death, something we have not seen much of in recent years.

There is no news on the CrowdStrike website, but you can find plenty on the web and the CrowdStrike X (Twitter) feed, mostly from irritated customers. Many users in the X feed are complaining that CrowdStrike has not apologized for the incident.

The outage has notably affected sectors such as airlines, media, and banking, leading to significant disruptions including grounded flights. Check your flight status.

CrowdStrike is actively deploying a solution, but some systems are unable to remain online long enough to receive the update. For those experiencing persistent BSODs, CrowdStrike recommends the following steps for a temporary workaround:

1. “Boot Windows into Safe Mode or the Windows Recovery Environment.
2. “Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
3. “Locate the file matching ‘C-0000029*.sys’, and delete it.
4. “Boot the host normally.”

Some step details that may be helpful

• Let the system boot up and crash three times, this will give you a startup menu.
• Click Troubleshoot, Advanced Options, Command Prompt.
• If you are prompted for a BitLocker Recovery Key
  • If BitLocker is managed via Intune, this can be found at https://myaccount.microsoft.com, under “devices”. Make sure to match the Hostname of the device and the Key ID
  • Otherwise, ask your IT administrator for your BitLocker Recovery Key
• In the command prompt window, go to the crowdstrike folder.
  • c:
  • cd \windows\system32\drivers\crowdstrike
• del C-00000291*
• exit
• Click continue to Windows

For detailed guidance on the resolution, refer to CrowdStrike’s official channels. It is advisable for affected users to follow the recommended steps to mitigate the issue until a permanent fix is established.

Update 2024-07019 07:30 – CrowdStrike has an update on their site: https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/

This image was posted by Efficient-Set-3711 on Reddit and may be helpful.

Steps for recovering Azure server affected by CrowdStrike Falcon BSOD

These steps are not fully vetted so use them with care and have an understanding of the details of the azure steps before proceeding.

1. Create a Test VM: In the same subscription and subnet as the VM you need to fix.
2. Export Template: Go to the problematic VM, select Export Template, and export it without parameters.
3. Delete the VM without deleting the resources. Do not delete the disk!
4. Attach the OS Disk from the affected VM to the text VM.
5. Delete the CrowdStrike File:
   • Navigate to the attached drive’s CrowdStrike directory: <Drive Letter>:\Windows\System32\drivers\CrowdStrike
   • Locate and delete the file matching “C-00000291*.sys”.
6. Detach the Disk from the test VM7.
7. Create a New VM using the template downloaded in step 2.

Guidance Azure on the CrowdStrike Falcon BSOD

We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state. We approximate impact started around 19:00 UTC on the 18th of July.

Additional details from CrowdStrike are available here: Statement on Windows Sensor Update – crowdstrike.com

Update as of 10:30 UTC on 19 July 2024:

We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines. Customers can attempt to do so as follows:

• Using the Azure Portal – attempting ‘Restart’ on affected VMs
• Using the Azure CLI or Azure Shell (https://shell.azure.com)

We’ve received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.

Additional options for recovery:

We recommend customers that are able to, to restore from a backup from before 19:00 UTC on the 18th of July.

• Customers leveraging Azure Backup can follow the following instructions:

How to restore Azure VM data in Azure portal

• Alternatively, customers can attempt to repair the OS disk offline by following these instructions: 

Attach an unmanaged disk to a VM for offline repair

• Disks that are encrypted may need these additional instructions:

Unlocking an encrypted disk for offline repair

Once the disk is attached, customers can attempt to delete the following file. 

Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys

The disk can then be attached and re-attached to the original VM.

We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.

Additionally, we’re continuing to investigate additional mitigation options for customers and will share more information as it becomes known.

This message was last updated at 11:36 UTC on 19 July 2024

Amazon AWS guidance on the CrowdStrike Falcon BSOD

https://health.aws.amazon.com/health/status