Slashdot Mirror
Ask Slashdot: Biggest IT Management Mistakes?
snydeq writes: Sure, nobody's perfect. But for those in charge of enterprise technology, the fallout from a strategic gaffe, bad hire, or weak spine can be disastrous, writes Dan Tynan, in an article on the biggest management mistakes in IT. "Some of the most common IT gaffes include becoming trapped in a relationship with a vendor you can't shake loose, hiring or promoting the wrong people, and hiding problems from top management until it's too late to recover." What are some other career- and company-destroyers you've witnessed in your years in IT?
Something that I've encountered is a sales contract signed with no provisions for upgrades. The end result is that we sold a product to a government entity... and then were forced to maintain it on old hardware, while the systems around it were changed and upgraded. There was nothing in the contract that forced the customer to upgrade, so they didn't, and we had to hire ever-more-expensive engineers to maintain this legacy system.
The big one I've seen with my current employer is that they've failed to expand their IT staff as the organization as a whole has expanded. The predictable result is that nothing but the most urgent requests gets handled promptly, and minor problems fester indefinitely.
There's no point in questioning authority if you aren't going to listen to the answers.
I recently worked for a startup with a real big asshole brogrammer who never was in the office and always was misunderstanding what was going on. Eventually I caught on to these charades. He was also lying to investors about our startup actually containing AI when it couldn't have been farther from the truth. To cover my own ass I sent a text message to the CEO warning him that we were committing fraud. After that the mark was on my head. Our "brogrammer" calls me into a room and says that I'm toxic. I counter by asking him his working hours and if he understands what fraud is. Over the next few weeks regardless of what happened I was ripped into a room and told how toxic I was.
Of course I was let go shortly after because the dumb CEO who always called me "his brother" was well aware of the sham and apparently didn't care. The crazy sociopath actually thought that I would still be friends with him after being fired. He also thinks it's some point I will come back to work for him, likely whenever that brogrammer finally leaves. I wonder if he knows that I'm planning on telling the FTC and his investors about this "AI" company. It's a bunch of regex matching natural language to appear as if you are speaking to a digital assistant. They're actually telling customers that this is a real AI system.
I would call this a serious mistake because their entire future is essentially in my hands. Since this Psychopathic CEO thinks I'm his friend and going to keep the lid on this, he's just plodding along blowing his money on other endeavors. I'll just let him build a little bit more of a paper trail for me before I strike. That's what you get for listening to the brogrammer.
I've seen them all, but "buying products or services from Oracle" ranks pretty high up. Or more generally, putting faith in a vendor because of a glossy ad in "CIO Magazine" or somebody in management getting kickbacks. Nontechnical managers are incapable of making these decisions, but want to feel like they're in control, so they try anyway.
... it was actually just RDP to a remote server.
At the first management/vendor meeting, I got to ask the first question: "How will response time compare to what we have now, with our servers in-house?"
"Oh, it will be much faster!"
It got worse from there.
I asked the owner if he knew that light slowed down in a medium and he said he did.
We already used RDP to the desktop and he KNEW about the latency.
Management ignored the red flags I threw on the play and put everything on the cloud against my recommendation.
We were a law firm and a time came when an elderly couple traveled from far away to sign some wills and the goddam cloud was down.
The family law practitioner blew her fucking top and confronted me and told me to implement Plan B.
I told her, "Ma'am, Plan B is Plan A."
It cost a fortune, but they paid termination fees and put everything back the way I had it before they went nuts.
It little behooves the best of us to comment on the rest of us.
We were on Linux for our file server for a decade and ended up needing to switch to Windows. We were tired of not having a good consultant and Windows consultants were easy to find.
The mistake incidentally was not in switching to Windows per se; it can easily do the added tasks we need of it and Linux could not. The mistake was in thinking the issue was in finding good Linux consultants-- the issue was simply in finding good consultants period.
A month or so after I was "promoted" from lowly developer to "Systems Infrastructure Manager" during a whole-scale move from an old green screen AIX based system to a brand new in house custom rewrite in modern tech, we had some of the new replacement hardware onsite and being built up (although the replacement applications werent ready to go, but thats not important to this story).
One friday, the UPS support contractor came in to do his servicing of the UPS - that went well, he finished up and switched it back from "bypass" to "protected". That triggered a surge on the electrical supply to both server rooms, which took the AIX box off line. Due to the nature of the green screen application, there was no way for it to be high availability - the data couldnt be replicated in real time, it didnt even talk to anything other than its own binary database files...
A few hours later, the corrupted AIX box was restored and ready to go - by this time, the company (a busy call centre) had been on manual processes for the entire afternoon. On the advice of the UPS contractor, who said the surge was probably the result of too much load on the UPS at the time, we decided to do a full shut down of the entire system, switch the UPS back over into "protected" and bring everything back up - so we waited until 6pm and did just that...
At 6pm, I threw the switch - and promptly looked over my shoulder at the comms racks behind me in the server room. The comms racks were billowing smoke. The comms equipment was burning. Before I could react, I heard loads of loud pops and bangs - both inside the server room and outside it.
Another surge. This one did real damage - a dozen network switches dead, over 40 PSUs in the servers dead, one server dead outright, and loads of call centre desktops went (loudly) pop.
Panic time. UPS contractor called back in - they gave the UPS a clean bill of health and promptly left, disavowing any responsibility.
The board of directors shat themselves - at that point we didnt know the ultimate damage count, but suffice to say the company was dead in the water to any observer.
Cue a desperate night of testing servers, pulling dead PSUs and swapping redundant PSUs between servers so that each server had at least one good PSU. Comms equipment was harder to solve, having to get some expensive switches from our local shop to tide us over. Desktops were bought from the local consumer PC store to give us enough desktops to run the company.
Ultimately, we were back up and running for 8am Saturday - it wasn't pretty, but it was up and running. 3 of us in the IT tech team worked through the night scraping the bare minimum together.
My predecessors DR plan was fleshed out to the point of "we have a DR site" (a commercial site a town over that we had a contract to use - no equipment there, no plans for how to fail over to it etc etc).
So, on to the management failure....
It just so happens that one of my things "to do" on the following Monday was to submit my DR plan for the "new world infrastructure" to the board, who were having their quarterly board meeting the following week (10 days after the company almost died). It was a modest one, but required some equipment outlay to make any DR event as smooth as possible - kept the same contract with the off site unit etc etc.
They turned it down, said it wasn't needed.
I quit the following week.
There's a widespread belief in IT that test code and UI are easier than other coding tasks.
It's completely false. Both are harder than other coding tasks. Your senior devs tell you to assign these projects to junior devs because they don't want to do them. They don't want to do them because they're much harder than other coding tasks.
It's win-win for the senior devs: they get easier work, and when the junior devs struggle, it makes the senior devs look even better. "oh, man, they can't even write a test suite. Well, I guess I should get the big bonus this year."
Put your best devs on test and UI. Put your junior devs on the simple stuff: backend work.
Make sure you actually can restore. Do it regularly. Restore to different hardware. If using tape, restore using different tape drives. Make testing restores a routine thing. When I was a boss, we did daily backups onto tape and same day read the tapes at the offsite recovery site (about 30 miles away).
During my career, I've seen two restore failures where they'd been backing up for years but the backups were no good and they had no idea.
"Almost every wise saying has an opposite one, no less wise, to balance it." - George Santayana
Windows as a primary OS is somewhat vague. Server or Desktop?
For 4 out of 5, that would be "neither".
Maybe 15% of Google uses their version of Ubuntu for desktop.
I have been to the Googleplex many times. I have seen plenty of Linux desktops, and plenty of people using Macbooks. I have never, not once, seen anyone working on a Windows desktop or laptop for anything other than testing.
And some of the things I hate most about Acrobats are these unnecessary additions to PDF files. It should be a read-only format, impossible to turn into a vector for malware. But no, Adobe screwed that pooch. I cannot open a PDF now, with Acrobat or Preview, without it thinking I have just modified the document and so it will ask me when I close it if I want to save my changes. I don't want encryption, if I wanted encryption I would encrypt the file. I hate vendors that send me "secure" documents and then require me to check in every 6 months to get new certificates so that I can continue to read it. I just want to read the document and I don't need the heavy weight Acrobat with it's strange UI doing this.
Being widespread doesn't make something good, people copy each other even when doing something stupid... Also all the examples you give are large companies which is a very important thing to consider...
SAP is a large, expensive system with many hidden costs in addition to the purchase price. You will likely have to buy lots of dependencies, lots of highend hardware, hire many expensive and highly trained staff to manage it and develop custom additions to handle your own business needs.
If you have the budget to do this, then it can work well... But many smaller companies go in blindly because they want to copy what these larger companies are doing... They get unrealistic quotes from greedy third party consultancies, or only see the ticket price and don't consider the true cost. They buy the software, but don't buy enough hardware to run it adequately, or don't hire sufficiently competent staff to manage it.
Many sales people will blatantly lie to you in order to sign you up for a large purchase, and then completely fail to deliver leaving you locked in with a huge bill and a big mess to clean up.
The end result is colossal failure and a big mess, or a system that limps along and still ends up costing a fortune.
A lot of people have a stupid mindset that "company X is huge and successful, if we copy them we will be successful too".
Copying a company 100x your size is not a good business plan, if you're a small company you should act like one and play to your strengths. You don't have the economies of scale or huge budget enjoyed by large companies, but you have agility that large companies lack.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Software changes... Applications come in and out of fashion, even new versions of the same software change radically between versions.
When we were in school, we were taught wordperfect for dos because "thats what businesses use", by the time we left school there were very few (if any) businesses left using wordperfect or dos. They were using msoffice 95 on windows 95 or nt4, which is still radically different to the versions in use today.
The differences between 2 versions of msoffice or 2 versions of windows can actually be more significant than moving to linux or libreoffice, and the prevalence of people accessing the internet using smartphones and tablets has shown that people don't actually need (specific versions of) windows to do so.
As soon as you become familiar with something, that software will become obsolete and people will be using something else. Teaching specific software is stupid, you need to teach users how to accomplish their goals with a variety of different programs, and how to identify the functionality they require in any software capable of doing it.
People are not incapable of adapting to changes, they just complain about it because they don't like change. Usually they aren't given any choice, and just end up getting on with it.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
There was a story a few years ago about how google banned the use of windows desktops on security grounds. If you needed a windows desktop as a specific requirement of your job (testing, dev etc) you had to be able to justify it.
But it does make quite a statement, the more technically oriented a company is the less likely they are to be using windows.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
A friend makes bank now after his CEO got pissed at sales and marketing doing this shit. His sole job is to go to sales meetings and keep sales honest. He knows the business well, and studies up on the customer to understand who they are and what they seem to need. Then he shoots down sales during the meeting when they start promising too much or trying to sell something that the customer doesn't really seem to want or need.
The marketing folks hate him, but the CEO, developers, and the customers love him.
Velociraptor = Distiraptor / Timeraptor
A lot of people have a stupid mindset that "company X is huge and successful, if we copy them we will be successful too".
Copying a company 100x your size is not a good business plan, if you're a small company you should act like one and play to your strengths. You don't have the economies of scale or huge budget enjoyed by large companies, but you have agility that large companies lack.
Now try telling that to the legion of devs who blindly use products like Cassandra, Hadoop or Storm "because we need to be able to scale". Then they jump through hoops trying to get it to do what they need (and not always successfully).
Just junk food for thought...
The most expensive, wasteful, credibility damaging, productivity reducing, and sheer chaos producing IT management mistake in my experience was the decision by a certain freight company to outsource IT. The two-letter outsourcing company, who will remain nameless, in sales presentations (which I attended) painted a rosy picture of a "right shore" solution with capable vendor-trained personnel in several call centers across the world, so that no matter what the local time, the call center would be on local daytime, which would help them draw the best talent for the job, etc etc. They were offering best-in-class for a fraction of the cost of in-house IT.
Upper management, who honestly thought that the entire job of IT was to push a button whenever a light came on, bought it hook line and cancellation penalty. As is often the case, they shook their collective fingers at us and told us "you'd better document your job thoroughly before you leave". Devops, my department, maintained a fairly extensive knowledge base, so it was only a matter of printing out all of our written procedures and handing them over (with two hands, because it was a lot of paper)to management of the outsourcing vendor. And, they lost them. So we printed them out again and handed them over. And they lost the second batch too. I'm convinced that they never intended to keep them. (More on that below.)
Our last day was Friday, which was also the cutover day. I had transferred to a department that was being retained, so got to stick around and see the carnage. It was fascinating in exactly the same way a high speed head-on collision between two passenger trains is fascinating. You're retching, but you can't look away.
This was back when Blackberry was still a thing, and all the execs carried one. BES went down Saturday and remained down for two weeks. This was the sign to upper management that things were perhaps not going as swimmingly as advertised.
The helpdesk was a shambles. You couldn't understand them, they didn't know what to do or whom to contact, and major incidents would just disappear in the system and never get addressed. Employees would come to those of us who survived the layoff and BEG us not to make them call the helpdesk.
The outsourcing vendor shook their fingers at us and said that those damned former employees had not documented their jobs well enough. I had a (third) copy of our procedures, and the names of the managers I'd handed them to both other times, and argued that we had in fact made a good faith effort, just ask these people. Only to find out that those managers no longer worked for the company. Curious.
The vendor said they could recover from our former IT employees' incompetence, but it'd Cost More Money. And that was the other shoe dropping.
Some former IT personnel were rebadged, so occasionally stuff still got done. They worked long hours in very stressful situations. Most of them moved on as soon as the economy started to improve.
Promises of a "right shore" solution were absolute fabrications. The entirety of IT, except for those few overworked rebadged employees, was a single call center in lower central India, manned by people apparently plucked off the street, sitting at card tables with IP phones.
The company tried to improve response by sending a number of people over to India to train the personnel there, but ran into an interesting phenomena -- as soon as employees of the outsource service got training, they WENT ON TO BETTER PAYING JOBS. This training effort served to flush out the people with any experience, causing an influx of new people who couldn't find an "enter" key with a gun to their head.
A major plumb for people who got a little experience was getting off night shift, which was our day shift. So as soon as we'd built a relationship with an admin and taught him to do valuable things, he'd brag about how he's finally getting off night shift, and we'd never hear from him again.
Speaking of which, I don't think th
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.