Slashdot Mirror
Bone-Headed IT Mistakes
snydeq writes "PCs preconfigured with stone-age malware, backups without recovery, Social Security numbers stored in plain view of high school students — Andy Brandt gives InfoWorld's Stupid Users series a new IT admin twist. Call it fratricide if you will, but getting paid to know better is no guarantee against IT idiocy, as these stories attest."
http://www.infoworld.com/archives/emailPrint.jsp?R=printThis&A=/article/08/06/16/25FE-stupid-users-part-3-admins_1.html Printer friendly version, rather then 7 pages.
Deleting hundreds of thousands of White House emails, and not having a backup?
The RISKS Digest never gets old.
-mkb
http://www.thedailywtf.com/
pretty much a new bone head story every day
http://thedailywtf.com/. Even if some of the stories are probably made up.
C - the footgun of programming languages
Information Security isn't going to get better without a major shift in how people work. As a society, we need to examine who really needs what data and then truly limit everyone to what they need. Until we can define these roles/access levels in black and white terms and permanently adhere to the controls put in place, there will always be IT blunders.
The problem is that these changes are rarely permanent, but more of a pendulum that swings back and forth as events like this occur. If Bob is taking home Social Security numbers on his laptop and someone steals it, controls may be put in place to prevent people from saving files to their laptops (and Bob is let go). Six months later, Suzie complains that she needs to be able to copy a proposal she's working on so that she can work on her flight to Japan. An exception is made. This typically snowballs until we're back to where Joe can copy the accounting records with SSNs.
Ease of access and efficiency nearly always trump security when these breaches aren't fresh in everyone's minds.
When a company simply accepts what the sales drone says about a given product as a fact.
(/local/home/curiosity)-#who -u|grep thecat|cut -c 44-49|xargs kill -9
more privileges than you need mistake! This one plagues IT guys day in and out.
Whoops, I mis-clicked and deleted a domain. Sorry Doc, I accidentally selected all your patients then declared them to have a clean bill of health. Oops I deleted a block of user accounts.
And a few I really did do....
Double "oh sh!t":
I just accidentally removed all my own rights... (I'll never forget the time I made that mistake... )
Setting a block of users to the wrong group, giving them Admin rights.
Clicking on a link that my trusted IT friend sent me...
How much is your data worth? Back it up now.
2. Continuing education for your IT people.
3. Just because someone looks old, doesn't make them a competent 'seasoned' IT guy.
4. Respect your IT pro's opinions.
We all have a plethora of stories of users, but even more of fellow co-workers in over their heads causing massive damage. Sometimes it goes unseen, other times it can desecrate a business. Make sure your IT people are educated, have a passion for what they do. Not just a paycheck monkey draining your resources.
A good test here, if your IT head is an ex-HR manager, mailroom clerk, secretary, or other far removed profession and have yet to get any certifications or degrees to prove their competence after 10 years then you probably are in trouble. Not in every case, but enough to make you worry.
Im not saying that a cert or degree proves that you are competent, but it at least shows that you try to be.
CS: It is all sink or swim...oh and did I mention there are sharks in that water?
I was new to the whole *nix thing but had been let loose as root on all the boxes at work. Someone suggested I could/should create a script to customise my environment so that I could run it when I logged on. Problem was I named the script "df" (my initials) and then promptly decided that it needed to go in to the /usr/bin/ directory. Yeah - now you know why I posted anonymously. :-D
See your mistake was believing that you actually had a "trusted IT friend".
You mad
At my middle school, there was a policy to give every student an ID card. That's fine. They decided that the best number to use for their ID is their Social Security Card. That's bad. They printed out a sheet every day listing the absent students for the day, with their names and their school id's. That is worse. Teachers threw these into their trashcans when they were done. Yes, the train wreck isn't over yet. The spreadsheet containing all of these numbers was on a public share. It was also accessible from the school website.
Or how about 3 years later, in my high school. All of the teachers user names and default passwords were on a spreadsheet on a network share. A publicly accessible network share. If a teacher didn't change their default password (a 4 digit number), A student would have full reign over their data.
Worse off, the grade book program was accessible from any networked machine (thanks Novell)
Thank god this was nearly a decade ago... So, one could pick a random terminal in the school and make subtle changes to their own (or perhaps someone elses) grades.
I used to think "I wish that I was alive during the 80's so that I could have been part of the cracking scene there". In hindsight, I could have done such bad things during the 90's, when I grew up.
Either that, or we should be discussing the boneheaded shiat done by lusers that IT guys have to clean up after. But that's probably already been done before around here, ad nauseum,...
Hold on a minute here.
The IT guy blames his boss for installing the Alexa toolbar, which lead to the deletion of all dynamic content on the company's web site.
No it didn't.
Yes, the Alexa toolbar isn't something anybody needs to run, and yes, Alexa should respect robots.txt, but whoever set up their web site is clearly incompetent:
1) Never rely on robots.txt for security.
2) The article says the Alexa spider captured usernames and passwords? What the hell were usernames and passwords doing unprotected on the web site?
3) The Alexa spider clicked all the Delete links. Never ever use links to delete things! Always use a submit button with POST, not GET. Generally, most spiders won't submit POST forms.
Security through obscurity is even less effective when the obscurity is poor.
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
Not as major is the Infoworld examples, but I still to this day sometimes forget to set-up a virtual interface when configuring a cisco router. This little command me more often than I care to admit:
telnet 192.168.1.1
cisco-router$ en
cisco-router$ config t
cisco-router(config)# int g0/1
cisco-router(config-if)# ip address 10.1.1.1 mask 255.255.255.0
Connection Closed
Gaaaaaaaaaaaaaaaaaaaaaaaah!
My school once had a folder called "Vice-Principal" in the network folders, what did it contain? Why, the C: drive of the vice-principal's computer of course, they didn't let you access "Program Files\" or "Windows\" of course, but what WAS accessible, was a Microsoft Access database containing every student in the school, their PPN number (equivalent of Social Security in Ireland I think), their home phone number, medical conditions, exam results etc. Of course this year they got new computers and completely re-setup the network, this time it seems substantially more secure.
Database take a dump? No backup of the transaction log? Fear not! With just two easy steps, your life will be back on track:
1. Update Resume`
2. Leave Town!
The game.
I used to work with a guy who did the "useless backup" thing. He set up an automated backup system that encrypted the files to tape. It ran fine for a long while. But when we had a server failure and needed to recover from the backup tapes, he couldn't remember what the decryption password was. All he could do was sit there saying "I remember that it was a good one." I just wanted to smack him...
This guy's the limit!
Because I wasn't his boss at the time (I became it later). At the time I asked both him and our boss if we had a decent recovery plan in place. I was assured by both that there was. That's really all I could do. If you want to think otherwise, by all means, do. But don't tell me that I'm "being an arrogant jerk and revelling in the mistakes of others." I was one of the ones who got my ass reamed over that mistake even though I had nothing to do with it.
This guy's the limit!
A company decides to run an internal check to see how many people will respond to a phishing scam. They send out an email to a group looking like the intranet page, "reminding" everyone to submit their username and password for the upcoming upgrade this weeken.
The email is actually an HTML form, but users being users, some of course hit reply instead of filling out the form and hitting submit. Worse yet, some hit "Reply All". Worse yet, some had HTML turned off, so the password wasn't even hidden in HTML source, it was in plain text for all on the list to see.
Yes, testing internally to see how many people are susceptible to phishing attacks is a good thing. However, send it via bcc, so group replies won't have passwords spreading around the company like a bad joke.
Next up, inform some people you are running your test. We have two different security groups, corporate, and the one I'm in. We didn't know about it, and all but shut down corporate security's access to the network. We traced the originating IP to their network, as well as the form submission IP. Since they weren't answering their phones, we didn't have much choice.
I found out because a supposedly "technical" engineer called me saying he had responded to it, and realized some people were replying and he could see other people's passwords. He didn't think there was anything wrong with submitting it, because it looked so real it couldn't be fake.
He stepped over the line the moment he gave the information to another classmate. He HAD to know there was something wrong with that. I can understand perhaps not telling the school staff about it, due to the "shoot the messenger" phenomenon, but anybody with a shred of morality would have destroyed the information, not given it to another KID.
I agree that jail time would have been a pretty harsh penalty, considering the real parties at fault were not facing anything even close to that.
So if you're so clever, how come you didn't warn the guy that might happen at the time?
Maybe because wandering around the office continually reminding professionals how to do their own jobs (assuming they are competent), makes you an arrogant asshole?
"Hey Ted, I know we hired you because you're all pro and stuff, but don't forget [some mind-numbingly obvious thing]. Seriously, I'm just trying to help, not implying that you're dumb as a rock."
This one really wasn't the IT staff's fault, so this is slightly off topic, but this is my all time favorite Daily WTF story.
http://thedailywtf.com/Articles/Im-Sure-You-Can-Deal.aspx
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
I could not access my mbox, the file was gone. Soon a co-worker stopped by... same... mbox gone. 2+2 together a quick visit to IT. "Hello, did you do anything to the company mbox's?", IT: "Oh yes, I observed they took up a lot of space on the disk so I *removed* them all"!
H.
I used to work in Unix Support for a large multi-national. Had loads of customers ring in with cock ups over the years. Some of them were silly, like a developer with root access typing rogue spaces where they shouldn't be. e.g: "chmod -R me / foobar". Conversations always started like "OMG I own the whole system, HELP!". Others were more obtuse, like a world renowned news reporting organisation who allowed one of their developers to install a very important database in his own account. System management got outsourced to Singapore, he then left the company, so Singapore deleted his account. We were left trying to reconstruct was was left from a dd image copy of the disk.
:-)
Another one I remember (about 20 years ago) was where one customer had systems that would crash at about 10am every monday morning. After a very long trouble shooting experience (i.e. months) the cause was found to be a delivery lorry that arrived every monday morning. He would back up to the loading bay, where some rubber bumpers (fenders) had been installed. He had the habit of stopping the lorry when he banged into the bumpers. Unfortunately this sent a shock wave through the building sufficient to cause some of the disks in the computer room throw a hissy fit and park their heads in the middle of whatever I/O they were doing.
In the early 90's I found myself having to pick up SCO Unix support for my sin's. Thankfully it only lasted 4 years. Two specific customer incidents I remember from that time. One was a call from a hospital who's system seemed in a right state. The guy was panicing, so I cut short my usual trouble shooting routine, got in the car and drove down there. Took one look at the system, typed ^D and then left after it'd finished booting to multi-user. Taught me a lesson; embarrassed the hell out of the customer and I never heard from him again.
The second was more interesting. I had a customer in the MoD at HMS Dolphin in Gosport. A number of their systems would crash simultaneously at certain times during the week. There was no real pattern to when, but when one of them went, they all did. I couldn't find the problem. No common denominators. Power monitors didn't show anything. Nothing. That was until one day the customer was staring out the window when the systems crashed. He remembered seeing one of the warships leaving the harbor and sailing right past his window. He also remembered seeing the ship starting its RADAR as it went past; and as the beam swept the computer room, all the systems crashed. The fix: a snotty email dictating that captains don't start their radar until they've cleared the harbor and made it out to sea.
I could go on typing for another hour straight with stories like this that either I've seen, or have happened to friends/colleagues
I once got called to help another technician with a system restore. Over the weekend a server had crashed and we had to rebuild it. First thing we do is to re-install the server. This took a few hours. Then we had to restore the data. No problem. We pull in the tapes but for some reason, it cannot find any files. The tech says that he's sure the backups were successful. Even the previous days and weeks had the same problem.
Figuring we had a busted tape drive, we drive 60 miles to pick up a tape drive from another location. Plug it up and bleah, same results. I ask for the backup log. Sure enough, everything is successful. Only problem is that nothing is configured to be backed up. So every hour, every day, every week, every month the job would complete successfully. Successfully backed up nothing.
The worst I've ever done personally was to install a CIFS module on AIX. This inadvertently updated a TCPIP package. This package had an obscure bug that was only triggered with long running sessions. It tooks hours to determine that the failure wasn't related to another patch that had gone in, and wasn't related to a very similar issue related to the connector...
It's ironic that just this morning I received an email from a user with just this line:
"are we able to get email right now?"
I resisted the urge to reply back with "no."
Turn it off, turn it on. Nothing was written to running-config.
In this case that's probably going to work.
When the router is at a power station in Guangzhou and you'd have to wait until 3AM and call in one of the people in the company who speaks the language so you can call the local office during their business hours and get them to power-cycle it for you... you're a lot happier that you remembered "reload in 15".
One of my co-workers once decided to install a beta version of Windows NT on the company's Novell file server, which EVERYBODY used for EVERYTHING. He did this in the evening when noone would notice and then he left for two weeks' vacation!!! I have never in my entire life met a more arrogant SOB. The entire company was down for over a day as we restored the server from a backup.
The boss refused to fire him (out of a cannon), so we filled the entire volume of his office with computer boxes. We went up and over the drop ceiling to deposit the last few boxes so he could not even open the door. When he returned from vacation, it took him a whole day to figure out how to get the boxes out.
Never assume that just because Backup Exec (or other backup utility) has backed up your data, that you don't indeed have problematic tapes and/or other hardware issues.
Test your god damn tapes people! When a company loses two years worth of data because backups were *never* verified to be working correctly, they're fucked. Needless to say, you'll be out of a job too.
Again. Restore from tape and verify!
Note: this just happened to a company I know. They called me asking for help because their last few IT contractors never verified backups are taking place properly. I really feel sorry for this company, and I've only met the owner once. Sad...
Life is not for the lazy.
I work for a major multinational corporation, but in one of the smaller (and rapidly shrinking) offices. When I started, there were 400 mainly caucasian male engineers in my office. Today, there are barely 150 male caucasians, and about 200 slightly darker caucasians from a certain nation south of China.
Anyhow, with the decimation of our "permanent" workforce and the movement of most of our labs to other, "low cost" centers, the time came to move out of our 300,000 square foot lab and factory facility into a smaller, 100,000 square-foot office-only space. This included moving the data-center.
When doing the budget for the move, the question came up as to how much power would be required in the datacenter in the new building. Of course, the answer was: "as much as we have now," two complete 30-amp 3-phase 208V circuits (180 amps at 208V total - about 40kW). Of course, with that much power being dissipated as heat in the data center, enough cooling would be required to keep the place from being an inferno.
Anyway, wiring two phases was going to cost a lot more money at union labor rates, and when the cost of the move start to overrun the budget, a certain PHB, trying to retain his bonus, decided to arbitrarily start cutting the budget for the move. ALso, the contractors installing the HVAC had already ignored the cooling requirements for the room, and said it would cost extra for them to fix their mistake. Well, let's just say that this certain, anonymous PHB decided that there would be no money in the budget for the extra three-phase circuits or to re-do the cooling.. Also, the cost to fix it then would be X, but the cost of fixing it later was going to be 5X at LEAST.
The result: A 600 square foot data center with about 25kW worth of equipment, 6 standard 15-amp office circuits, and 1.5 tons of cooling capacity. But. since the move was occurring at the end of a fiscal quarter, the PHB decided to spend 5X next quarter instead of 1X this quarter in order to make his bonus numbers.
The fallout: 4 complete 3-phase circuits instead of 2, 2 for the data center, and 2 for the leased portable air conditioners they had to roll in there as a "temporary" (we all know what that means) measure. Also, OSHA issues because since the air conditioners are only supposed to be temporary, they still create an auditory hazard due to their noise level - and you bet your behind that someone reports them to OSHA on a quarterly basis.
Fallout for the PHB? Absolutely NONE, of course.
*sigh*
I work for a Very Large Power Company, mostly hydro-based generation. We've been running our Generation Control System on *nix for about as long as anybody can remember. It's robust, secure and dependable.
However, we're beginning to see issues, especially with subsystems on old(er) proprietary hardware (cough*Alphas*cough) and replacement components are either scarce and expensive, or just plain unobtainable.
So we've recently completed the first phase(s) of a major GCS-upgrade project and the decisions have been rubber-stamped by the Government. (We are what's known as a "State-Owned Enterprise.) The new GCS system will be running on a Microsoft Windows Server platform.
Why?
Because the two contractor chicks who presented the choices to a Government-run committee, whose members have no desire to be held responsible or accountable in any way, shape or form, heavily promoted Microsoft Windows Server, via a bunch of garish PowerPoint presentations and Word documents.
Why?
Because, as one of the contractor chicks candidly admitted not long after, "[I] only know Windows."
So, a national infrastructure control system, one which epitomises the very notion of "Mission Critical", is to be based upon what is quite probably the absolute worst choice of NOS imaginable.
The (unaffiliated) national power distribution company migrated from *nix to MWS a few years ago, for what were essentially the same reasons. Their admins are not envied. Much of their time is spent coaxing the backup-backup-backup-backup servers back up.
One immediate result of the recent decision is that three of this company's best-and-brightest IT people resigned and "moved on". The departure of several more is imminent. I can't call them rats, but they are certainly escaping a ship that's heading straight for the iceberg, full steam ahead.
It's highly likely that this country's governing party will change at the forthcoming national election, although it will change nothing else. If anything, the soon-to-be-incoming party is likely to be even more MS-friendly than the current one, so I don't foresee any likelihood of sanity prevailing anywhere near the top in the near future.
Instead, what's likely to happen is that once the system begins falling apart - as it surely will - MWS will be quietly shelved by lower echelon IT management (avoiding any embarrassment to anybody in an expensive suit) and a *nix-based one will be restored. Estimates of when that will occur range from "Within a year" to "It has to happen eventually."
I use Win XP Pro at home. It's fine for general purpose family use. But MS Windows does not belong on a server: Or, at least, not on any which are expected to remain functional most of the time.
True story and, yes MS fanboys, I know you'll be modding this down to "-1: Troll" and "Flamebait". I can cope with it, thanks. I have bigger worries right now.
Here's my take. For data of any significant value, you should test the backup and recovery procedures every so often to make sure they work. This could include figuring out what happens when critical members are out (hit by the bus) or most of the department is gone (food poisoning at the office party). So as part of Ted's job, he (and a few coworkers) should on occasion run through a test recovery so you know nothing mind-numbingly obvious has been forgotten and to verify that that recovery can still go on even if Ted is hit by the bus. Plus in addition to gaining experience and verifying that the process works, they can generate documentation to help with a real data recovery.
Back when I used to work for a major aerospace manufacturer, we had an interesting incident:
We had a production control system hosted on a series of HP-UX servers. The IT department had just reorganized, placing a new (inexperienced) manager in charge of our systems. One day, all the servers went off line. As the factory ground to a halt, I managed to log in to one via telnet. It seemed to be up, but many functions were failing. I traced the problem to: no /tmp directories remained on any of the systems. I contacted the on-duty admin. with his tidbit of information. She informed be that, "as ordered by management, all /tmp directories were to be removed." Apparently, the new boss had read somewhere, that /tmp was for storage of "junk". He deemed the storage of "junk" to be an inappropriate use of company resources and, to prevent it, all /tmp directories were to be removed.
Have gnu, will travel.
I understand what you are saying, but this twitter guy is really starting to get annoying. So I think the anti-twitters are doing a service to us all.
I disagree. I do not even notice twitter's posts. There's a lot of bullshit posted to slashdot, and I guess over the years I've just learned to filter it out without even thinking about it.
Anti-twitters, however, seem unignoreable. They post not about the article nor about anything related to the article, they point their fingers and stomp their feet and whinge and carry on like a gradeschool tattle-tale. Why is it I notice them but not twitter? I can think of two reasons: first, twitter's particular brand of bullshit fits in and is easily dismissable. second: the anti-twitter posts are jarring and do nothing but promote themselves. I don't even think twitter's posts do that; they just spread BS.
Perhaps slashdot needs another filter category: twitter wankfest. That's really what it is: who can spot the twitter post fast enough and piss and moan about it the loudest. I'd happily filter it all out in an instant, and as I said I am starting to filter out the anti-twitter self-righteous asshats as I encounter them. Twitter's no friend of mine, but at least he isn't interrupting the thread.