Having attended ISC2 Secure Asia Pacific and hearing a lot of interesting presentations about supply chain and resiliency I found myself circling back to all the opinion pieces I have read about the great BSOcalypse in the past weeks. Plenty of content out there addressing the Why? What? How? I want to take a slightly different approach and discuss why it was inevitable and what CISOs can do to protect their companies.

The Inevitability of BSOcalypse

Back in the early 2000’s everyone was trying to build suites of products, it became clear that this just resulted in suite of products that were sub-optimal wrapped around a good core product. The 2010’s saw a transition to integrated suites of best-in-class products.  That process has been going on for over a decade now and it’s forced a coalescing of vendors in each vertical. If I was to build a new product today, much of the content eg. identity management, payment, infrastructure wouldn’t be written by me and wouldn’t be managed by me. I would just be writing another point solution and weaving it into a web of other point solutions. There has been a massive aggregation of users in these point solutions, with the leaders in each vertical showing tremendous growth each year. 

This brings us to the inevitability. Crowdstrike was the IT industries equivalent of the Evergiven blocking the Suez canal. When things fail today, it’s going to be big; billions of dollars today have been lost by organisations who never purchased Crowdstrike and by business leaders who have never even heard of Crowdstrike. It was enough that one of their vendors in this complicated interconnected web of dependencies just happened to use Crowdstike and ipso facto the business is unable to operate. Outages whether caused by negligence or malicious actors are increasing in impact because we are more interconnected by bottlenecked, aggregated services.

What’s a CISO to do?

              With the ever-increasing complexity of technology stacks, it’s really no wonder that 62% of CISOs are experiencing burn out*. Attack surfaces are increasing, compliance and legislative burdens on IT are relentless. After two decades of building and selling technology products I can offer a few titbits of advice to the consumers of enterprise technology.

1 – Rolling releases/Canary releases

The vast majority of my IT career has been spent working in Asia, on the occasions that customers found themselves in the early stages of a failed rolling release they were always aghast that they had been randomly chosen by our deployment tools as the “guinea pigs”. I’d inevitably find myself sitting across a table from the angry faces of their IT leadership teams defending the practice. Asia has always been a pretty conservative in adopting new IT practices, it’s pretty clear today that customer should not only accept rolling releases, but they should also demand companies do them.

2 – Mandatory Updates

              Vendors have always wanted to enforce mandatory updates; Clients are happy to never update because it’s already working. They let their environments get so outdated that upgrading becomes a security issue. There is a happy middle ground which involves customers having some capability to delay an upgrade but being forced to upgrade after some time.

              As an Aussie the timing of the CrowdStrike incident really doesn’t sit well with me, deploying content into a product running on the kernel at 2pm on a Friday afternoon (Australia Eastern Standard Time). Given the option as a customer I would never let a critical patch run on my premise Thursday through Sunday unless some sort of special approval process was triggered for a critically urgent issue.

3 – Trust and compliance

              We depend on various standards (ISO/SOC/etc) to keep vendors honest, they’re supposed to be like a secret handshake, I know the handshake, do you know the handshake? You do? Great let’s do business. Clearly it’s not enough, plenty of failures have come from organizations that have impeccable security reputations, organizations which spent billions on auditing and accreditations.

              Maybe we need to take some of the security specialists manning the security operation centers chasing script kiddies and reallocate them to thoroughly pentesting these applications. I really do wonder if a good pentesting team could have forced Crowdstrike Falcon to fail. Or did everyone just give Crowdstrike the benefit of the doubt? I’m sure the answer will come out in the coming months, as cyberforensic teams try to replicate the events of 19^th July. If it turns out that it’s trivially easy to make it fail, the implication would be staggering.

4 – Multi cloud, multi device

              It’s never been easier to deploy across multiple cloud environments. As part of my work as tech-advisor at salesinnovation.io I’ve been working with a local APAC vendor Kapstan.io in Singapore. They make it trivially easy to do multi-cloud deployments using Containerization and Kubernetes. Harshil the CEO of Kapstan and I will be delivering a webinar in the next two weeks and I highly recommend you tune in to what he has to say on the topic.

              Similarly, it’s never been easier or cheaper for IoT devise to have hardware/OS redundancy. A $200 raspberry pi can easily outstrip the capabilities of a PC from a few years ago. All those airport devices that failed in the last two weeks cost tens of thousands in maintenance each per year. So why don’t they have redundant hardware and operating systems? Having spent time in maritime software/hardware, it’s a topic that is discussed daily, so what’s going on at airports?

5 – Multi-vendor

              Why were customers using Crowdstrike locked in to one vendor across all their environments? The consensus I’ve heard from the CISO community is it’s too hard to go down the path of multiple vendors.

I do appreciate that it’s hard, each vendor has processes teams need to learn, configurations that need to be done etc. But this isn’t just any software, it’s kernel level EDR. No one is saying you need multi-vendor for your word processor, but maybe for products running as part of your device boot up it might be a good idea.

6 – Thin client apps

              Many of the systems affected by the breach were end-user consoles, in an era where browsers can do truly amazing things, how many of these thick client applications need a thick client. Some of the most resilient companies I’ve talked to that were Crowdstrike customers and yet weathered the storm perfectly fine were companies that had super thin end user machines (basically just a browser). So all they need to do was to nuke the affected endpoints and rebuild in a couple of hours.

7 – Coding assistants and security coaching

              I was having a spirited discussion with a customer around coding assistants which led to me firing up VSCode and throwing Tabnine a security question about the code in question.

Sure, enough the issue I suspected was picked up, item number 5) failure to validate the input adequately. Now we can argue back and forth about the efficacy of coding assistant for improving coding, but if a coding assistant can pick up that your input validation is rubbish, then you probably don’t need a rocket scientist to identify this.

              I don’t know if it would help Crowdstrike with their kernel level code, but it’s certainly good enough to pick up the issue in this project. (check out my webinar on secure use of coding assistants)

Conclusions

It’s been a tiring two weeks. I’m sure plenty of my cybersecurity colleagues share the same sentiment.

It might be time for all of us to pause and reassess where we are spending our cyber security efforts and maybe demand a little more from our vendors.

*Gartner