When the world woke up today, most were eagerly anticipating the start of another wonderful summer weekend. What many instead experienced is what one Redditor coined ‘What Y2K wished it was.’ The massive global tech outage was due to a flaw CrowdStrike’s latest release. It has resulted in thousands of canceled flights, major banking disruptions and even crashed Starbucks’ mobile ordering system. While only Windows users were affected (and greeted by the ‘Blue Screen of Death’) over half of American Fortune 500 companies are CrowdStrike customers, so the damage has been far-reaching. Thousands of companies have reported disruptions and by proxy millions of end-users have been impacted.
Formstack is not a CrowdStrike customer, so we were not impacted. That said, many of our customers are. We also have several former CrowdStrike employees on staff, so in the interest of trying to make a bad day better, we’re here to offer some advice on the best path forward and shine a light on some best practices that can prevent similar situations in the future.
If you are one of the folks who has been affected by the CrowdStrike Falcon outage and you’re feeling frustrated, understand that you are not alone. I don’t recommend canceling your contract or trying to uninstall the sensor across your fleet. The positives of endpoint detection response (EDR) far outweigh the financial and operational inconveniences being experienced in today’s incident. (If you are technically inclined, however, CrowdStrike has released some workarounds that you can use to get back up and running here: https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/)
I would encourage folks to look at this as an opportunity to revisit some best practices in deployment models in a network. While we are told to always stay up to date with the latest and greatest software updates, I recommend employing the N-1 deployment approach, where updates are automatically rolled out but remain one version behind the latest deployed version, especially to critical systems. This allows a company to test the functionality of the software against its own software stack before fully deploying. As a former CrowdStrike employee, I know that CrowdStrike Falcon offers N-1 and even N-2 dropdown options for updating policies within their platform.
No one wants to hear the advice ‘hurry up and wait,’ but it may be the best path forward for many affected by today’s outage. The long-term plan of adjusting your deployment model should be considered by everyone, but the last thing anybody should be doing is compounding their problems and making them worse. For travelers trying to get home, or businesses losing money, I know this is painful, but tomorrow the sun will rise again.
