OT: OT-“Look, maybe I didn’t say every single little tiny syllable, no. But basically I said them, yeah.”

MarkusKetterer

Shoulda got one game in
IMG_1369.png


What the actual f***, Minnesota?
 
  • Wow
Reactions: BUCKSHOT

Bobby Bottle Service

Win for Rick
Dec 15, 2005
5,376
3,545
Toronto
I almost guarantee you Crowdstrike laid off some of their Software Quality Assurance Engineers and thought AI could do their automated tests.

Sounds like a simple linter test should have caught this though, a malformed .sys file.

Is there a reason every org is pushing the latest crowdstrike update without testing though? Is that simply how crowdstrike operates?
CrowdStrike uses a kernel driver that is signed but pulls all .SYS files in a local directory. The malformed one had a null pointer to a memory address that doesn't exist which caused systems to bluescreen. This is the selling feature for an EDR product, to which CrowdStrike *was* considered best in class. The kernel level access allows them to monitor memory for malicious behavior (contrast an Anti-Virus which looks for known attacks) and speeds up the response time.

This basic error should have been caught in QAT or even IST because it would have blue-screened those systems immediately. There are going to be a whole bunch of people getting walked. Wouldn't be surprised if the BOD walks the CEO to demonstrate they are committed to change.

That said, I was supposed to work last weekend so I got an unexpected reprieve.
 

Jim Bob

RIP RJ
Feb 27, 2002
58,882
39,948
Rochester, NY
CrowdStrike uses a kernel driver that is signed but pulls all .SYS files in a local directory. The malformed one had a null pointer to a memory address that doesn't exist which caused systems to bluescreen. This is the selling feature for an EDR product, to which CrowdStrike *was* considered best in class. The kernel level access allows them to monitor memory for malicious behavior (contrast an Anti-Virus which looks for known attacks) and speeds up the response time.

This basic error should have been caught in QAT or even IST because it would have blue-screened those systems immediately. There are going to be a whole bunch of people getting walked. Wouldn't be surprised if the BOD walks the CEO to demonstrate they are committed to change.

That said, I was supposed to work last weekend so I got an unexpected reprieve.
If they fire a whole lot of people due to this it will be a wasted educational experience that they paid greatly for.

Humans in general are not great at learning from the mistakes of others. They are better at learning from their own mistakes. Although, lessons repeated are quite common, as well.
 

TehDoak

Chili that wants to be here
Sponsor
Feb 28, 2002
32,250
9,606
Will fix everything
If you work remotely, I highly suggest investing in a monitor that suits your work style/setup.

I had previously purchased a monitor with a built in KVM in 2020. It was a 2018 model that was released in 2017, and did a very poor job with the USB-C and power delivery, specifically with macbooks. It would wake the macbook from sleep periodically and switch over to the USB-C input, even if you were actively using another one. It was fine IF you just used the USB-C part. IT was starting to have other issues though, like USB-C just cutting out completely while using it and macbooks having a hard time picking up the monitor without plugging and unplugging multiple times.

Right before my contract got ended prematurely, I ordered a new monitor that specifically supported the Thunderbolt 4 spec rather than just USB-C with power delivery. Combined with higher DPI, better brightness, and higher refresh rate, it's much better. It's a 2025 model that was released this year, so it has all the latest and greatest. It's even has a mode to adjust the brightness with a light sensor for less eye strain.
 

Chainshot

Give 'em Enough Rope
Sponsor
Feb 28, 2002
156,416
110,832
Tarnation
If they fire a whole lot of people due to this it will be a wasted educational experience that they paid greatly for.

Humans in general are not great at learning from the mistakes of others. They are better at learning from their own mistakes. Although, lessons repeated are quite common, as well.

They will make a selected few heads roll and change policy to make sure it won't happen again. I would expect the manager of the team that did this, their boss, and then the head of that division plus likely the CEO, though the CEO gets the parachute and told to go away with millions of dollars.
 

Gras

Registered User
Mar 21, 2014
6,666
4,072
Phoenix
If you work remotely, I highly suggest investing in a monitor that suits your work style/setup.

I had previously purchased a monitor with a built in KVM in 2020. It was a 2018 model that was released in 2017, and did a very poor job with the USB-C and power delivery, specifically with macbooks. It would wake the macbook from sleep periodically and switch over to the USB-C input, even if you were actively using another one. It was fine IF you just used the USB-C part. IT was starting to have other issues though, like USB-C just cutting out completely while using it and macbooks having a hard time picking up the monitor without plugging and unplugging multiple times.

Right before my contract got ended prematurely, I ordered a new monitor that specifically supported the Thunderbolt 4 spec rather than just USB-C with power delivery. Combined with higher DPI, better brightness, and higher refresh rate, it's much better. It's a 2025 model that was released this year, so it has all the latest and greatest. It's even has a mode to adjust the brightness with a light sensor for less eye strain.
3 Monitors is the minimum starting point.
 

TehDoak

Chili that wants to be here
Sponsor
Feb 28, 2002
32,250
9,606
Will fix everything
The fact that delta is STILL on the struggle bus with the Crowdstrike issue 3+ days after the bug hit is a real failure of their IT infrastructure (so badly that the secretary of transportation called them out publicly)

Considering they just spent 5B in stock buybacks and their IT can't handle this firedrill, some serious heads have to roll. The initial surge is 100% on Crowdstrike...but not being able to handle a crucial business critical outage like this is on their IT department. Logistics don't even matter at this point, 96+ hours later. The fix was immediate, it was about mobilizing resources to get it done.

3 Monitors is the minimum starting point.

I ran 2 monitors previously, and 3 at times.

1 ultrawide is the way to go these days, especially coupled with a KVM. I have to turn my head to see the other side of the screen I have. I think multiple of these would give me serious neck issues.
 

Gras

Registered User
Mar 21, 2014
6,666
4,072
Phoenix
The fact that delta is STILL on the struggle bus with the Crowdstrike issue 3+ days after the bug hit is a real failure of their IT infrastructure (so badly that the secretary of transportation called them out publicly)

Considering they just spent 5B in stock buybacks and their IT can't handle this firedrill, some serious heads have to roll. The initial surge is 100% on Crowdstrike...but not being able to handle a crucial business critical outage like this is on their IT department. Logistics don't even matter at this point, 96+ hours later. The fix was immediate, it was about mobilizing resources to get it done.



I ran 2 monitors previously, and 3 at times.

1 ultrawide is the way to go these days, especially coupled with a KVM. I have to turn my head to see the other side of the screen I have. I think multiple of these would give me serious neck issues.
I run my side monitors in portrait, having that verticality when opening full screen documents is a godsend and doesnt interfere with what I am working on the main monitor.
 
  • Like
Reactions: brian_griffin

Chainshot

Give 'em Enough Rope
Sponsor
Feb 28, 2002
156,416
110,832
Tarnation
Years ago we had an intern who was messing about with a couple of video cards. He managed to set up five and then eight off a single unit. We referred to his cube as "the megaplex".
 

brian_griffin

"Eric Cartman?"
May 10, 2007
16,786
8,044
In the Panderverse
I assume that's the sphere in Vegas? If not, help a brother out with the answer...
Ethiopian cuisine is definitely very underrated. Used to love hitting up an Ethiopian restaurant whenever I was in Bahrain
Yes, but is it filling?
View attachment 896296

What the actual f***, Minnesota?
Check out the Minnesota state fair episodes on any Food network show - DDD / Fieri, etc.
You can get anything deep fried there.
C'mon everything is better deep fried
Unless it's in old oil, then it's shit.
CrowdStrike uses a kernel driver that is signed but pulls all .SYS files in a local directory. The malformed one had a null pointer to a memory address that doesn't exist which caused systems to bluescreen. This is the selling feature for an EDR product, to which CrowdStrike *was* considered best in class. The kernel level access allows them to monitor memory for malicious behavior (contrast an Anti-Virus which looks for known attacks) and speeds up the response time.

This basic error should have been caught in QAT or even IST because it would have blue-screened those systems immediately. There are going to be a whole bunch of people getting walked. Wouldn't be surprised if the BOD walks the CEO to demonstrate they are committed to change.

That said, I was supposed to work last weekend so I got an unexpected reprieve.
I don't understand the above, but I get the gist. CrowdStrike is essentially preventative vs. AV's which are curative (or, perhaps more like Tamiflu where you still get it but it's symptoms are mild and you can manage through it without being down for the count for several days).
 

Bobby Bottle Service

Win for Rick
Dec 15, 2005
5,376
3,545
Toronto
If they fire a whole lot of people due to this it will be a wasted educational experience that they paid greatly for.

Humans in general are not great at learning from the mistakes of others. They are better at learning from their own mistakes. Although, lessons repeated are quite common, as well.
These are basic change management protocols. The person who accidentally use the null pointer which was essentially a typo, because it referenced a memory location that didn't exist only has a small portion of the blame.

There are other teams that are supposed to validate that the updated file works correctly before being promoted to production. Those teams have responsibilities and that would not count as a learning experience.

Probably not a good analogy, but if you have a security guard who forgets to lock doors, that's not a learning experience. That's a firing.
 

Jim Bob

RIP RJ
Feb 27, 2002
58,882
39,948
Rochester, NY
These are basic change management protocols. The person who accidentally use the null pointer which was essentially a typo, because it referenced a memory location that didn't exist only has a small portion of the blame.

There are other teams that are supposed to validate that the updated file works correctly before being promoted to production. Those teams have responsibilities and that would not count as a learning experience.

Probably not a good analogy, but if you have a security guard who forgets to lock doors, that's not a learning experience. That's a firing.
I work in software R&D and have spent a significant portion of my almost three decades at my employer in software testing.

There is zero reason for a failure such as this to get released. The most basic of smoke tests should have uncovered this bug from all reports. In my experience, when you have something like this happen, it is due to schedule pressure leading to tests not getting run like they are supposed to.

Firing people is unlikely to lead to Crowdstrike being less likely to cut corners due to schedule pressure unless they change the culture and prioritize quality testing and release practices.

Like we have seen with the Sabres for the past 13 years, different does not guarantee better.
 

oldgoalie

Goaltending matters.
Jan 7, 2004
13,367
6,260
VA
The fact that delta is STILL on the struggle bus with the Crowdstrike issue 3+ days after the bug hit is a real failure of their IT infrastructure (so badly that the secretary of transportation called them out publicly)

Considering they just spent 5B in stock buybacks and their IT can't handle this firedrill, some serious heads have to roll. The initial surge is 100% on Crowdstrike...but not being able to handle a crucial business critical outage like this is on their IT department. Logistics don't even matter at this point, 96+ hours later. The fix was immediate, it was about mobilizing resources to get it done.



I ran 2 monitors previously, and 3 at times.

1 ultrawide is the way to go these days, especially coupled with a KVM. I have to turn my head to see the other side of the screen I have. I think multiple of these would give me serious neck issues.
I just replaced my 10+ year old iMac 27" with a new Apple Studio Monitor and a Mac Mini M2. The monitor is spectacular; not wide screen, as I don't need it for coding or anything like that. But the connectivity, sound and camera are awesome.
My old iMac is just about Apple "obsolete", so I hopped on the M2 chips to accommodate the expected AI additions that should take advantage of the M series chips.

In over 10 years, I never had an issue with that iMac, while I've replaced my wife's Windows machines 3 times.
 
  • Like
Reactions: Jim Bob

Ad

Upcoming events

Ad

Ad