We will be regularly updating this page with information and advice on the evolving situation.
Last update: 19 July 2024, 14:54 UTC
| Record of changes | |
| 09:40 UTC | Briefing note published |
| 11:10 UTC | New information from CrowdStrike added, alongside guidance for those in cloud/hosted environments. |
| 12:24 UTC | Highlighted the double reboot method; added information obtained from an analysis of the corrupted CrowdStrike driver. |
| 14:54 UTC | Expanded Double Reboot Method to specify trying between 2 – 15 attempts. Added the time of the patch deployment to the timeline. |
What happened
A global IT outage has occurred, affecting airlines, banks, broadcasters, and other sectors. The issue appears to stem from a fault in a recent CrowdStrike update, causing widespread system crashes and "blue screens of death" (BSODs) on computers with CrowdStrike security sensors installed.
Double Reboot Method
If you are experiencing a BSOD, there may be a simple way to bypass this and boot successfully. Try turning your machine on and off again anywhere from 2 to 15 times. Windows has an in-built mechanism for detecting failed boots caused by corrupted drivers and will attempt to load a backup configuration.
Make sure these are full restarts from a powered off state, and not restarts induced by a BSOD boot loop.
If this method does not work, apply the steps below.
There are four likely scenarios of impact, listed below from most severe to less severe.
- Your devices are experiencing BSOD, BitLocker is enabled, and you do not have access to Recovery Keys.
This scenario could result in businesses needing to rebuild large numbers of systems from scratch; however, there is a possibility that a patch from CrowdStrike or Microsoft may alleviate the situation. See Table 1 below, Recovering a device unable to boot. - Your devices are experiencing BSOD, BitLocker is enabled, and you can access Recovery Keys.
You will need to retrieve your BitLocker keys and undertake a logistical exercise to assign these to the affected assets before physically accessing the device and deploying the recommended fix. See Table 1 below, Recovering a device unable to boot. - Your devices are experiencing BSOD and BitLocker is not enabled.
You must physically access the device and deploy the recommended fix. See Table 1 below, Recovering a device unable to boot. - Your device is still online but has CrowdStrike installed.
CrowdStrike have patched the original issue and you should update to avoid further issues. See Table 1 below, If your device is still working.
Event timeline
At approximately 05:20 (UTC), CrowdStrike issues a public advisory acknowledging widespread reports of BSODs occurring in Windows machines, which appeared to be affecting devices with CrowdStrike sensors installed.
Around an hour later, at 06:27 (UTC), CrowdStrike had linked the issue to a recent update covering multiple CrowdStrike products. The update reportedly impacted a driver used by their sensors. Affected operating systems appeared unable to load the modified driver, leading to the BSODs.
At 09:45 (UTC), CrowdStrike published an advisory noting that a patch has been deployed.
This means if you have yet experienced an impact, you should update to avoid further issue
At approximately 12.00 (UTC), we became aware of information conducted by independent experts reviewing the CrowdStrike update. They reported that the issue did not stem from a corrupted driver, but rather from an issue with the Windows Hardware Quality Labs (‘WHQL’) process. Supporting resources used by the driver appear to be unverified, causing the driver to appear as corrupt during boot, leading to the BSOD.
What to do now
We have outlined our guidance below for common situations.
Table 1 – S-RM guidance
| Situation | Guidance |
| If your device is unable to boot. |
Try to restart your machine 2-3 times manually. It is possible that Windows will recognise the corrupted service and restart with an earlier configuration. If this does not work, see situation Recovering a device unable to boot. |
| If your device is still working. |
Your IT team should apply the latest CrowdStrike patch. As of 09:45 (BST), they have advised of a new patch that addresses the original issue. You should update your sensor to avoid further issues. |
| If you have a BSOD machines in a cloud/hosted environment |
For impacted machines in the cloud, you have a few more options to apply the fix.
|
| Recovering a device unable to boot. |
If you have BitLocker Recovery Keys Apply the fix recommended by CrowdStrike:
If you DO NOT have BitLocker Recovery Keys You have two options:
|
Additional considerations
- CrowdStrike have committed to providing resourcing for their customers. Ensure you are raising any impact with their support team.
- This is an evolving situation, and more information may come to light. Expect evolving guidance across the next few hours and days.
- If you anticipate a major restoration workstream, due to wide impact or inaccessible BitLocker Recovery Keys, consider activating your major incident processes and seek any additional resources required to undertake recovery.
- The possibility of a future update from CrowdStrike or Microsoft may provide a more immediate solution. For those that are hard down, we recommend putting aside affected devices and implementing additional means of working for the time being.
