Years ago I worked for a major blue chip tech company that started in a garage 80 years ago.
I was a tier 1 customer engineer. When tech support and onsite techs fail, they call us. Remote, onsite, whatever is needed to resolve. A major supermarket chain had an ongoing issue with backup tape drives on their POS servers. they have one at every supermarket, a pretty robust server running SCO UNIX. The tape drive fails every 6 weeks, almost like clockwork. Hundreds of servers across the country, an expensive tape drive and onsite service every 6 weeks for each and every one. 3 years in to an 8 year contract…. They were losing money hand over fist and no one could figure it out.
I jumped in a company van and went to inspect every server in a 50 mile radius of my office. Looking for environment issues that might matter. I schedule down time for each site so I can reboot, reseat, and otherwise dig around in it. So these are all night calls after closing.
They were ALL filthy, stored in janitor closets or tucked under a warehouse shelf. But this particular server is sealed and filtered for use in fairly dirty environments, and the insides are clean. The customer is replacing intake filters like they should at every site.
It was also noisy at all these sites.
Then while inspecting one and pondering the issue in a particularly quiet location I hear something. A click from the tape drive. Normal, tape being initialized- a backup must be happening soon.
But no back up. Weird. Noted. Keep inspecting. About 5 minutes later- click again. 5 minutes later- click again. And again.
Well, this one is a bad SCSI terminator. SCSI keeps seeking new devices on an open chain. Easy to fix. I go get the part and replace SCSI, then keep inspecting.
Click. …. Click … click
What the hell? Another cable. Same.
I finish, button it up and go to the next site. Noisy, but I pay close attention. Click … click ….
happening here too.
7 more sites scheduled that night and the next. All doing it. I grab a terminated SCSI cable for a different system. replace it at the last site after verifying it clicks. No click. all is quiet.
We had a batch of bad terminators, the SCSI was causing the drive to initialize every 5 minutes when not in use till it dies. All of the dedicated replacement cables for these servers came from the same manufacturing batch that made them. ~$3M a year contract in jeopardy and around $1.6M a year in service and parts being spent. I solved it. We sent a different SCSI for replacement to each site and the issue went away.
The company had a policy. Save the company more than $100k, you get a bonus equal to 10% of the amount saved. I put in for it, with all the numbers, and a conservative estimate of $3M in parts over the remaining 5 years.
I got back a letter of appreciation and a memo that cancels the program the day before I submitted the forms. The memo is dated that day, but the text says as of “x” date the program will be discontinued.
Rather than pay the bonus of ~300k over 5 years, they retroactively ended the program. Then laid off my entire division and replaced them with a call center overseas that we had to train before we left. The 'techs' there had no tech experience whatsoever, and just wanted some scripts to follow. Uninterested in actually learning the systems.
That was the day I became antiwork.