Slashdot Mirror
Time Bomb May Have Destroyed 800 Norfolk City PCs' Data
krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"
It's Naw-Fuck.
And it's nowhere near as embarrassing as how we pronounce Buena Vista.
Is it just my observation, or are there way too many stupid people in the world?
... this is the internet... Isn't the apostrophe in the title supposed to be further to the left? :|
I had to read it twice to confirm it was used correctly.
I live in VA Beach, which is the next city down the road (I live a few blocks from exist 20 264, and down-town Norfolk is exit 13ish), and I work in a security-related position, so we tend to keep up on news like this, but this is the first I'm hearing of it, though it looks to have gone down last week (apparently the boot.ini files were modified between 16:30 and 17:30 on 9 February, and only the computers which rebooted during that time period were affected).
It doesn't sound like the attack was particularly complex or anything, so maybe that's why it isn't exactly "newsworthy" (I also don't watch local TV news, so I don't know if they mentioned it), but still, sucks for them. I hope they have good backup policies.
I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches...
Hardly. It's just something that messed with the Win32 folder. This could be fixed by a few temps over the weekend if the city government was half-competent.
I am the richest astronaut ever to win the superbowl.
> We don't know how it got into our system... We speculate...
As long as we're speculating, may I nominate last week's "Operation Cyber Storm" (http://www.dhs.gov/xnews/releases/press_release_0853.shtm).
if they were running backups, they wouldn't be scratching their heads and behaving completely ignorant of what exactly it was or when it was put in. They obviously lost everything, which I'm sorry but I find some darwinism/justice in that. If you don't even have a backup to look at to see what it was sitting on the hard drive waiting to blow up, you're just beyond help. Maybe better luck next time.
But too many out there simply must learn their lessons the hard way. That will never change.
I work for the Department of Redundancy Department.
At first glance that blows my mind. That's absolutely huge. Then I check my linux box and /usr/lib64 is 1.7 GB.
As to why they couldn't just boot to linux or a recovery CD and salvage the data....
"Eve of Destruction", it's not just for old hippies anymore...
So the data is wiped because the System32 folder is fucked up? Uh-huh... guess they have to throw out all those computers and order new ones. Looks like the data's gone forever.
mmmm...forbidden donut
We've instituted offsite backups, both over the tubes and physically taking images of our servers (all virtualized of course) offsite to a bank safety deposit box. If, for whatever reason, the whole damned building explodes tomorrow, we've got the data sitting on servers in two other geographically distant locations. But if we can't get to those, we have the VM images, so as long as we can get our hands on a server capable of running Linux KVM, we could be up and running in short order (I estimate 3-4 hours, including host OS installation).
The days when a physical or digital attack can fuck the whole organization are gone. There are enough traditional and newer backup schemes out there that even long downtimes aren't necessary.
The world's burning. Moped Jesus spotted on I50. Details at 11.
I think Time Bomb is the best of all the No-Heroics superheros. http://www.youtube.com/watch?v=JLaXUTdybjc ....
Oh wait, you were talking about that
Flexible bare-metal recovery for Linux/UNIX
You must have a pretty small site if all of your data is contained within the .vmdk files and you can restore an entire datacenter (from bare metal) in 3-4 hours (including OS install time).
It's relatively small, but we're actually backing images up to hard drives, not to tape or over the wire. The files themselves are both backed up to tape, and use DFS and some other mechanisms (like robocopy replication) to our remote servers. In a worst case scenario, I could pretty much drive the 100 miles, grab the remote domain controller and file servers from one of our satellite sites and drop them in the main office. The guys out there might not be happy that they were accessing everything through terminal services, but oh well.
The world's burning. Moped Jesus spotted on I50. Details at 11.
IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.
Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.
I have seen time bombs left behind by two types of people when being called in as a consultant to deal with the aftermath:
1: The disgruntled employee. He leaves a hidden file that if not touched in 2-3 weeks will start wreaking havoc. I've even seen modified binaries of tar and such that encrypt the files, so even backups are trashed.
2: Someone wanting to frame another person. I've seen this done by clients of other consultants who do not want to pay the consulting fee. So they put a logic bomb in. The admin that left gets blamed and faces jail time. In this scenario, it is a word against word issue almost always, and juries tend to believe business owners far more than the admin who got railroaded.
You just restore the image from a ghost backup without worrying about the data because the data is stored (by policy) on the servers. What? A user ignored that policy? Tough luck for him.
Best Slashdot Co
You got it. it's also a great example of how incompetent most City's IT staff are, Hey municipalities... you get what you pay for. How's those $25,000 a year IT staff working out for ya?
Do not look at laser with remaining good eye.
You cant take any details from any news articles at face value.
Deleted
How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day. If you have an entire office full, ready connected up to the network, you just have to pop in a CD (if you even need one) start the PC and move on. A couple of dozen people could do that lot in a weekends worth of overtime.
Most of the time I spend on rolling out a new PC is delivery, connection and admin. Where's the problem here?
I'll see your Constitution and raise you a Queen.
Twenty bucks says that they never figure out what happened.
(I estimate 3-4 hours, including host OS installation).
I've done this in some small VMWare setups: using snapshot feature on FS (LVM works) plus a few very large external drives (those USB to SATA cradles work great), automate a backup of the snapshots of the OS and VM partitions once every X days take the drive offsite and use another one. With 3 drives, you can rotate them and always keep one offsite. What you now have is essentially a fully working drive you can insert into another server and just turn on, no OS install, no fiddling with VMWare install and versions, recovery time is down to essentially the time it takes to get the drive(if you have to use offsite drive) and get new hardware. Best thing is that the costs are that of a few USB drives and a bit of scripting...
-Em
RelevantElephants: A Somatic WebComic...
From working in the backup industry for years, I'm sure they have backups, the problem is that they never tried to verify or restore them. but is there really isn't any data there, compression is great when you just "tar cv * > /dev/null" ...
Heck one time I had a guy who was getting Parity Errors decide that the best way to solve them was to just shut off Parity Checking... Ignorance is bliss I suppose.
Seriously I can't count the number of times I tried to help someone restore their backups after a critical loss that turned out to never have actually verified that they worked in the first place. Just as bad as when I worked in a photo shop and someone said they couldn't get their film out... put the camera in the light locked compartment, stuck my hands in, just to find that he had taken 36 'priceless vacation pictures' on the back of the camera body instead of film.
-=JML=-
If lil' ol' me can spend a few hundred dollars on enough hard drives stuffed into external enclosures the have two complete backups of all ~1.5TB of data in my system, surely a municipal government can spend a few thousand dollars to do it too.
What the hell, who runs systems that important without backups? Management teams named Shirley?
Living With a Nerd
"destroyed data on nearly 800 computers citywide".
By corrupting the Windows System32 folder install they lost their own files? Did the malware delete some key file that prevents Window's from hosing the disk and crushing the MFT and/or MBR? I doubt it. The OS installs may be unrecoverable, but the article / spokes people seem to jump the gun by stating such generalizations like "destroyed data" and "essentially destroyed these machines". I imagine that actual "data" of importance is still recoverable via external means, and that a quick reformat will make the machine quite OK again.
Maybe this is good incentive for them to install Linux, now that they have a ~800 machine testbed to work with.
'We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.' RPF
The basic idea behind storing snapshots is simply to allow faster recovery of operations even in the case of absolute disaster. We still have nightly differential backups, a weekly full backup, plus Server 2003 DFS and some scripted replication (via robocopy) of file servers. Nothing replaces a good backup scheme, a major pain in the ass to develop, and sometimes a pain to maintain. When we formulated the project, the basic notion was "If a fire/meteor/other disaster takes out one of our offices, how can we reduce downtime and data loss to the barest minimum". Rather than relying on a single backup strategy (ie. tape or distributed FS), we adopted a scheme of using multiple strategies. Daily and weekly backups are still important for accidental deletions and corruptions. Quarterly and annual backups are still important for archival purposes, and this is still the area where tape is king. But trying to restore something like a Server 2003 domain controller or Exchange server purely from backup has always been for me a nightmarish prospect, consuming considerable amounts of time. The idea behind virtual guest snapshots on a weekly basis is that I can get these servers up and running ASAP and use weekly and daily backups to refresh everything to get data up to date.
If the tape fails, well the worst is that I lose at most four business days of data, but hopefully not even that with DFS and other replication strategies. But let's take a worst case scenario, that somehow someone breaks into the network, destroys all the data on all domain controllers, the Exchange server and the file servers at all sites (something I don't find terribly likely). I still have the full backups of all files plus the Exchange and AD domain controller images sitting offsite on an external hard drive in a bank vault. I might lose about five days worth the work at the outside, which would be bad, no doubt about it, but certainly not the catastrophe of losing all my data, but that's only in a worst-case scenario.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Just blame Terry Childs. It was a backdoor into a citywide system. Clearly he's responsible. Doubtlessly the D.A. is already concocting a theory involving him having visited the city during a conference, and installing a modem into the network. By phoning a specific number and entering a sequence of numbers from his prison phone, he's brought the network to its knees.
"you know why? Because we got the bomb, thats why" -Dennis Leary
Maybe with tapes this is a reasonable expectation.
However, users and IT folk alike copy files to and from CD, to and from the internet, across networks, from drive to drive, from USB to hard drive and back and they don't run into parity errors.
So it's not unreasonable to assume that software and hardware designed to be backup tools wouldn't fail as often as they do.
When my drives fail, it's almost always VERY OBVIOUS, not some subtle creeping error.
I think most of the time the problem is not data corruption, but lack of planning if the data will be in a usable form or not.
I have Ghost backups for my home PC, and I backup my data using external drives. But I have never gone through the process of learning and doing the recovery on the boot partition because that backup is a last ditch thing. When my drive fails, I will either spend the time to do that, or just say "bah, time for a new computer anyway" and go that route.
When you sort the bits first compressed backups are really small.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
* Check every few seconds to see if network goes down
* Write a bogus entry in the log files that points to some oddball behavior, like a disk-read error or something
* If network is down freeze screen so it looks like computer just locked up
* Ignore all input
* Wipe key parts of disk so forensic recovery is impossible or at least very difficult
* Wipe key parts of memory so forensic recovery is impossible or at least very difficult
* Wipe key parts of cache so forensic recovery is impossible or at least very difficult
* Force or fake a BSOD screen so a casual user will think his computer crashed and blame any resulting data loss on the crash
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Anyone else find it odd that the first thing the IT techs there "quickly isolated and rebuilt the offending print server."?
Sounds to me like they know who it may be and were covering up for a friend.
This sounds like a job for the Innocence Project.
It also sounds like something police and courts need to be made more aware of.
This sounds like the tech industry's equivalent of a divorcing parent accusing the other of child abuse - where the type of abuse doesn't leave scars and the child is too young to give credible evidence, it's 50/50 or less whether justice will prevail.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
That's a hard lesson to learn.
Now I understand why the desktops weren't backed up properly - their drive content was considered disposable. This is not unreasonable given the above statement.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
A similar thing happened where I work (uni campus), although due to config errors, not a timebomb.
400 machines got imaged and we're scrambling to collect drives, install new ones, reimage and then run recovery on the old orig drives.
Microsoft really needs to add the ability to set user profiles on a different partition, as you can w/ UNIX.
No sig for you!!
I think that Arizona, with its odd mix of Indian, Spanish, English and who-knows-what takes the cake with odd spellings and pronunciations.
Ft. Huachuca (Wa-chu-ka)
Mogollon Rim (Mo-gee-yawn)
Tempe (Tem-pee)
Canyon de Chelly (dee-shay)
On the other hand, I spent some time in Pueblo, Colorado where about 1/4 of those born there pronounced it Pee-eb-lo.
Great civilizations have lived and died on false theories. Don't mess up mine with a few facts.
FTA: "... the city found that the system serving as the distribution point ... was a print server. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server."
Ok, if I have a single workstation with "AntiVirus 2009", I will probably nuke it without a second thought. If one of my servers has been commandeered to serve as the command and control channel for a worm that just ate 800 of my PCs, I SURE AS HELL AM GOING TO GET A dd OR OTHER FORENSICALLY SOUND IMAGE OF THE MACHINE BEFORE I WIPE IT!!!!!!!!! For crying out loud, they contacted the FBI, but they just destroyed what could have been the single most important piece of evidence! Do they have a Best Buy in Norfolk? For $100 they could have brought the machine up on a clean hard disk and set the existing one aside for forensic examination without wasting the time of taking an image of the drive.
Also, they have no idea how the attack occurred, but they are sure it didn't come from the internet. Any evidence to back that up? It's one thing to say it probably didn't come from the internet because our logs show no traffic to support that possibility. It's ridiculous to make that same statement based on a gut feel.
If this article is accurate, these guys are playing amateur hour IT security. Their first action should have been to contact a qualified incident handler.
put the camera in the light locked compartment, stuck my hands in, just to find that he had taken 36 'priceless vacation pictures' on the back of the camera body instead of film.
Now, that's what I called a Kodak Moment (TM).
Growing up in Ohio, some of the pronunciations for local places are horrible.
The first are mostly just anglicizations. Not awful, but sometimes quaint, odd, and hickish. There are a lot more that I'm forgetting.
Lima - "LYE-muh".
Ravenna - "Ruh-VEN-nuh"
Medina - "Meh-DYE-nuh"
Berlin - "BER-lin' "
Milan - "MYE-lin'
Vienna - "VYE-en-nah"
Bellefontaine - "Bell Fountin' " Ack.
Then they just get really bad and annoying.
Nevada - "Nuh-VAY-duh". Really. And most locals pronounce the state Nuh-vah-da or Nuh-vad-ah, so what gives?
Mantua - "MAN-uh-way." The Italians are laughing and Shakespeare must be turning in his grave.
Versailles - "Vur-SAILS" Ugh.
A preposition is a terrible thing to end a sentence with.
You must have a pretty small site if all of your data is contained within the .vmdk files and you can restore an entire datacenter (from bare metal) in 3-4 hours (including OS install time).
If you use any of the various wizards that create an install script based on your actual VM host config, you can usually re-install a host in less than 10 minutes.
Then, if you have a good backup of the actual running config of the host (i.e., the VM database, the virtual disk files, etc.), it's just a matter of getting the data to where it belongs.
For most, the biggest issue would definitely be acquistion of the hardware (the hosts, all the network hardware, SAN, etc.), which would generally take a lot longer than the re-install time.
The only Netware that is not a Netmare.
I fully believe a 12+ year uptime.
Bet it's still running strong.
2 was good once it was setup. Genning sys was a netmare however.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
Just to clarify, I was talking about host's file system snapshots (think LVM), and not VMWares's guest snapshots. FS snapshots will let you get a consistent backup of host OS and all the VMs. (if you backup a running VM without a FS snapshot you likely end up with a useless corrupted file)
This way you can grab the off-site backup drive, install it into fresh hardware, turn it on and have a fully functional system in matter of seconds.
And no, it does not replace file-level backups - its just for emergency recovery. (Of course theoretically you can start the backup on a non-networked hardware and get the files you need, but there are better solutions.
-Em
RelevantElephants: A Somatic WebComic...
It would be interesting to see the evolution of the department's budget in the last few years.
It wouldn't surprise me that they got ride if their best assets and replaced them with cheaper guys with less than the required experience just to "reduce operating cost". They may have saved a few dollars during a year or two doing that but the shit had to spread one day...
When I set up my backup system, one of my concerns was that my loved ones might need to restore my backups, should I get hit by a truck someday.
I realized that if my filesystem is a rat's nest when I back it up, then my backups will be rat's nests as well, as would any restored data. So I have spent several months scrupulously organizing all of my filesystems on all of my computers.
That simplifies my backups, and would make it much easier to find what one needed when doing a restore.
My external drives both have ext3 filesystems. But they each have a small FAT32 partition containing installers for Mac OS X and Windows ext3 filesystem drivers. There are three such drivers available for Windows; I included all three.
My backups are not quite yet where they need to be, but they're getting close. I have automated continuous backup of my Subversion repository for instance. Whenever I do a checkin of my own personal code, a post-commit hook backs up my entire repository with the hot-backup.py script, then within three hours a cron job replicates the repository backup to my external drive.
I also need to write up detailed instructions that my survivors could follow, then mail them off to all of my relatives.
Recovering my data would be complex enough that these instructions would have to come right out and tell my mother to get my sister to do it for her. Happily my sister is hip to The Penguin.
Request your free CD of my piano music.
Re-worked summary of TFA:
- All that has been damaged is the System32 folder of user machines.
- 'Destroyed' I imagine is an IT staff trying to dumb down his language to his perception of the level of the reporter's IT knowledge
- Their IT may have done quite well, the only 'damage' is to PCs that were shut down in the 1 hour window between the attack starting and IT containing it
- Employees were supposed to save to the network. The only issue stated is that some staff were breaking the rules and saved things to their own PC.
All they need to do with the affected machines is to boot from a Windows or Linux CD, copy the files to memory stick and throw their standard "new install" image on. No data loss. No network down time. All they're looking at is some hassle for the ~ 18% of users affected and a very busy IT department. Provided the affected users have other machines to work on (or however not losing much productivity) they're not far off having the best scenario any It department can realistically hope for (well, I'd like to say it's reasonable to hope for not having pissed off employees). Sure, no doubt a dozen IT managers can post their "perfect" system, and another dozen IT managers can show how they could destroy it.
But whoever hated them enough to install the timebomb would obviously have sabotaged the backups. Maybe that was what the delay was all about.
http://michaelsmith.id.au
Workstation backups? I didn't know that was something that anyone did.
Don't take life so seriously. No one makes it out alive.
If all this did was modify or delete the system32 directory, then the data is still going to be on the drives and should be easily recoverable - so I wouldn't refer to those machines as "essentially destroyed," - I wouldn't even refer to the data as "destroyed."
All it means is the machine wont boot normally. I know for most users that renders the machine temporarily useless, but even a low level IT tech should be able to recover data or get the system booting again, there are about 5 different ways to do it.
That doesn't change the fact that someone unleashed this on a civic network, but it bothers me when electronic attacks are described in a way that makes them sound much worse than they actually are - because we already have the government looking to use "cyber security" as the next big issue which they'll surely try to tackle via censorship, privacy violations, internet filtering, and wrongheaded laws.
According to the local 6 pm news, all fixed and back online, data intact. Evidently, the affected machines were on a shared network, NOT just the City's.
Some one said it came from the print sever so any will to bet in came form some hole in HP or some other vendor software?
Came through the rift in Cardiff and drifted all the way to Norfolk.....
Paula Bean was working there? May be?
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
There was the guy who locked out administrator privileges in the San Francisco computer system. He recently went on trial in December, but I have not heard a verdict.
It's just something that messed with the Win32 folder.
So far as they know, at this point.
I agree that the rest of it could be fixed - although it's likely to be easier to wipe and re-image the systems (if they bothered to build them from images in the first place, that is) than to try and restore the installation.
But the original problem could lie in the user data files or somewhere else as well, once they restore those and someone clicks on an infected file they are doing it all over again.
SB
It's old. The more humans I meet, the more I like my cats. At least they are honest.
Not necessarily (although I agree with you).
That's why competent IT staff do spot checks on the integrity of their backups.
(Or if you're paranoid enough and have the time, or it's indicated by the mission critical aspects of your data, you verify ALL of them)
SB
It's old. The more humans I meet, the more I like my cats. At least they are honest.
It's not, except for the insane or people who aren't able or willing to use a reasonable imaging and app distribution system.
It appears that people who didn't RTFA or who work at tiny tiny sites are criticizing these guys without knowing what the hell they're talking about.
No one does workstation backups because it's costly, risky, inefficient, and generally doesn't work. The only way to make it work is to say "put all the documents you need to backup here" and here is better off being a network drive anyway.
VMware Data Recovery is a piece of shit that rarely works the way you want it. Try reading the forums sometime to see how much grief it gives others.
LET THIS BE A LESSON TO ALL YOU SYSTEM ADMINISTRATORS!
whom I have heard saying (repeatedly) - "it is not a critical server, it is only a print server... we can wait to patch it later."
From just the article, I have a pretty good guess as to what or how it happened... or how I could replicate such an event with two commands, and little or no evidence left behind.
A disgruntled citizen comes in to use a public access terminal placed there for citizens to look up public records, and PRINT THEM OUT. This public terminal is locked down - sure, it is also on its own private VLAN, lest anyone plug into the network with their own laptop... heck, lets go one further and say they even bound the mac address to the switch port to make sure that any other network device plugged in wouldn't work (unless they spoofed the mac address).
So, our Disgruntled Citizen Hacker (DCH) takes a bootable USB thumb drive/boot CD and inserts it into the computer and reboots it to Backtrack4 or some other utility - or they simply plug into the network using their own laptop...
Once booted from his device, DCH launches an ancient exploit against the print server that "doesn't contain any sensitive data" according to the SYSADMIN "and can be rebuilt within hours if it ever got infected." - except that DCH isn't all about stealing data, he's all about getting revenge against the cop that gave him that speeding ticket - and HE'S GONNA SHOW YOU!
Once his script kiddie exploit has him sitting at the c:\ prompt, he does a "NET VIEW" and sees that the print server is on the domain, and can see the entire network from its secondary interface that connects it to the internal network. This system administrator has even copied the SYSINTERNALS suite of tools to the hard drive (he even added them to the PATH! -OR- he copies the SYSINTERNALS suite from his boot device) and with one command, DCH gets to work. "PSEXEC \\* DEL c:\boot.ini" and hits enter, the command starts cycling through all the computers on the network -but he screwed up... it is taking much too long to connect to each computer - only to screw up the boot.ini file? Naw, thats too easy to recover from.
CTRL+C
-DCH's Adrenaline is now pumping-
PSEXEC -d \\* DEL *.* /F /Q /S
This time, it runs in disconnected mode.
"Ah yes, much faster." DCH says to himself - except he screwed up again, he forgot to put the "C:\" in front of the *.*, so it is (Q)uietly, yet (F)orcefully deleting all the files listed under the %SystemRoot%\System32 folder and (S)ub-folders (including those files marked as read only), instead of the entire C: drive. Major adrenaline sets in - he's not gonna cancel it this time. He's already committed, it's too late now. That and he's lost his nerve and is visibly shaking as he's feeling the rush.
He retrieves his boot device, reboots the computer, and quietly walks away, trying oh-so-hard to not raise any suspicions as he quietly walks back to his car. "Take THAT..Your Honor." he mumbles to himself as he jams the key into his Honda Civic, it fires up with a roar as the ported exhaust reverberates throughout the parking garage. He revs the engine and squeals the tires as he leaves the ramp - radio blaring.
One hour and 800 computers later the print server is taken offline -and promptly rebuilt- exactly according to the disaster recovery plan. Doesn't matter - even if they did forensically analyze it, the only evidence they'll find is a single error (among thousands of errors) in the event log that was caused by the exploit, of itself signifying nothing conclusive. The admins never did set up event log correlation, so once the server was rebuilt, all bet were off. So, our DCH walks away, scot free.
But wait! Did he really?
Check the courthouse cameras. On Tuesday, Feb. 9, sitting down at 4:07pm you'll see the DCH take his seat at the public terminal. He looks around and cannot believe that the stupid IT depar
Good security is based upon reality and common sense. Common sense is a function of having common knowledge.
The flaw is that when a government entity pays for higher quality, people then screech about government waste and inefficiency. "I can't believe they're paying $50,000 a year for IT jobs that are really only worth $25,000 a year."
I agree, however sometimes even having backups means diddly squat, especially if the admin was never smart enough to test the backup system's integrity. I worked in one place where we had backups galore, until one day we needed to restore a db, and when we asked for it, it took 2 hours to rummage through to find one backup predating even the installation of our db, we were screwed. The disk failure, made it impossible to recover the mdf and ldf files off the disk, and without having checked that each night the backup image made was ok to use for a restore, we never knew that the backups were not good ones until it was too late.
Sadly, the admin lost his job over this one, and I think left IT altogether.
The teams name was NcGill.
They called themselves Lill.
But everyone knew them as Nancy.
No brain, no pain.
Sadly, the admin lost his job over this one
I don't feel the least bit sorry for him. If the company relied on him being the expert and taking care of backups and he completely failed at that aspect of his job, that's the price for catastrophic fail in any job position. Something like that doesn't even rate as an accident. Accidents can happen. That's just plain negligence, and considering the severity, gross negligence. Out you go, mind the door...
I work for the Department of Redundancy Department.
I agree, but I hate to see people lose their jobs, especially during these times.
You hate to see an incompetent IT person whose negligence causes his company to lose tens of thousands of dollars getting replaced with someone that knows what they're doing and can actually be relied on?
How do you feel about the company that loses all that due to the incompetence of an employee they were trusting with their future?
I don't feel the least bit bad for that person. From a purely selfish perspective I could even be a little happy about it, I can do better than that, and I can use new job opportunities, and that'd be a win-win situation for that company and me. The only loser is, well, the loser. The only real tragedy in that whole story is what had to happen to the company before he got replaced.
I work for the Department of Redundancy Department.
>You hate to see an incompetent IT person whose negligence...
That's not what I said, don't twist what I am saying, I said I hate to see ANYBODY lose their job during this time where we are still somewhat in an economic crisis, I agree that many
incompetent workers should get replaced, but they still have families, and they still
need jobs, and anyone losing a job during these hard times are always worse off then
when everybody is happy and making money.