This is, in a lot of ways, impressive. This is CrowdStrike going full “Hold my beer!” about people talking about what bad production deploy fuckups they made.
You know you’ve done something special when you take down somebody else’s production system.
*production systems around whole world
Few people can put that into their CVs, that a real achievement!
I’m volunteering to hold their beer.
Everyone remember to sue the services not able to provide their respective service. Teach them to take better care of their IT landscape.
Typically auto-applying updates to your security software is considered a good IT practice.
Ideally you’d like, stagger the updates and cancel the rollout when things stopped coming back online, but who actually does it completely correctly?
Applying updates is considered good practice. Auto-applying is the best you can do with the money provided. My critique here is the amount of money provided.
Also, you cannot pull a Boeing and let people die just because you cannot 100% avoid accidents. There are steps in between these two states.
you cannot pull a Boeing and let people die
You say that, but have you considered the savings?
People are temporary. Money is forever.
I have. They are not mine. The dead people could be.
Edit: I understand you were being sarcastic. This is a topic where I chose to ignore that.
That’s totally fair. :)
I work at a different company in the same security space as cloudstrike, and we spend a lot of time considering stuff like “if this goes sideways, we need to make sure the hospitals can still get patient information”.
I’m a little more generous giving the downstream entities slack for trusting that their expensive upstream security vendor isn’t shipping them something entirely fucking broken.
Like, I can’t even imagine the procedureal fuck up that results in a bsod getting shipped like that. Even if you have auto updates enabled for our stuff, we’re still slow rolling it and making sure we see things being normal before we make it available to more customers. That’s after our testing and internal deployments.I can’t put too much blame on our customers for trusting us when we spend a huge amount of energy convincing them we can be trusted to literally protect all their infrastructure and data.
I can put the blame to your customers. If I make a contract with a bank they are responsible for my money. I don’t care about their choice of infrastructure. They are responsible for this. They have to be sued for this. Same for hospitals. Same for everyone else. Why should they be exempt from punishment for not providing the one service they were trusted to provide? Am I expected to feel for them because they made the “sensible choice” of employing the cheapest tools?
This was a business decision to trust someone external. It should not be tolerated that they point their fingers elsewhere.
Funny how CrowdStrike already sounds like some malware’s name.
It literally sounds like a DDoS!
Botnet if you will
Not too surprising if the people making malware, and the people making the security software are basically the same people, just with slightly different business models.
Classic protection racket. “Those are some nice files you’ve got there. It’d be a shame if anything happened to them…”
It sounds like the name of a political protest group.
The real malware is the security software we made along the way.
We’ve known that since Norton and McAfee.
At least McAfee’s antics were entertaining
The guy left his company long before the shenanigan cascade
After working at a company that had Crowdstrike installed on all machines, it is most certainly malware.
deleted by creator
You’re hired!
Idk boss, people weren’t too happy the last time we tried that.
The answer is obviously to require all users to change their passwords and make them stronger. 26 minimum characters; two capitals, two numbers, two special characters, cannot include ‘_’, ‘b’ or the number ‘8’, and most include Pi to the 6th place.
Sorry, I don’t understand. Do you mean there have to be 6 digits of Pi in there, or the sixth character must be π? I’m down either way.
We won’t tell you, and the rule gets re-rolled every 14 seconds. It may stay the same or it may change.
Also, there are requirements we check for that we don’t tell you about! 🤭
What’s the saying about dying a hero or becoming the villain?
Maybe this is a case of hindsight being 20/20 but wouldn’t they have caught this if they tried pushing the file to a test machine first?
It’s not hindsight, it’s common sense. It’s gross negligence on CS’s part 100%
Well, it is hindsight 20/20… But also, it’s a lesson many people have already learned. There’s a reason people use canary deployments lol. Learning from other people’s failures is important. So I agree, they should’ve seen the possibility.
I saw one rumor where they uploaded a gibberish file for some reason. In another, there was a Windows update that shipped just before they uploaded their well-tested update. The first is easy to avoid with a checksum. The second…I’m not sure…maybe only allow the installation if the windows update versions match (checksum again) :D
Windows has beta channels for their updates
It’s a sequence of problems that lead to this:
- The kernel driver should have parsed the update, or at a minimum it should have validated a signature, before trying to load it.
- There should not have been a mechanism to bypass Microsoft’s certification.
- Microsoft should never have certified and signed a kernel driver that loads code without any kind signature verification, probably not at all.
Many people say Microsoft are not at fault here, but I believe they share the blame, they are responsible when they actually certify the kernel drivers that get shipped to customers.
Now threat actors know what EDR they are running and can craft malware to sneak past it. yay(!)
Smart threat actors use the EDR for distribution. Seems to be working very well for whoever owned Solar Winds.
SHOULD’VE USED OPENBSD LMAO
Yes but the difference is one of them is also going to help you fix it.
deleted by creator
Who says it was accidental?
Netflix knew they were going to move from DVD rentals to streaming over the Internet. It is right in their name.
CrowdStrike knew they were eventually going to _________. It is right in their name.
