Slashdot Mirror
Emory University SCCM Server Accidentally Reformats All Computers Campus-wide
acidradio writes: "Somehow the SCCM application and image deployment server at Emory University in Atlanta accidentally started to repartition, reformat then install a new image of Windows 7 onto all university-managed computers. By the time this was discovered the SCCM server had managed to repartition and reformat itself. This was likely an accident. But what if it weren't? Could this have shed light on a possibly huge vulnerability in large enterprise organizations that rely heavily on automated software deployment packages like SCCM?"
SCCM is pretty good. It makes my desktop techs jobs significantly easier to deploy assets company wide. In this case, it sounds like someone pressed some buttons without being 100% clear as to what was going on. Unfortunate someone will not be working in IT ever again.
Knowing that people have been running various kinds of centralized update services, perhaps across multiple OSes, and spanning several years now, listening to a story about an update server literally going rogue and nuking everything attached to it, and then for the coup de grace, basically committing suicide at the end by reformatting itself, does not sound like an accident.
If it truly was, I'd hate to see what the hell purposeful intent looks like.
Bad news most likely on this front. I have worked University IT, and I can guarantee they are going to have problems.
For one, no matter how many layers of backups you have, when you are working with a bunch of 90 year old academics, they will always find a way to miss every single one.
And more grievous, Universities tend to have important data that absolutely cannot be backed up in any normal way. Data that is legally obligated to stay on one specific computer in one specific room and never leave; under penalty of legal action.
Troll is not a replacement for I disagree.
It sounds like the commenter above was teachable - he no doubt learned his lesson.
It also sounds like the company's owner knew he could learn this lesson. That's the mark of a great manager.
Whether the Emory staffer responsible for this mistake is teachable or not, I hope his boss can tell the difference. Some folks aren't teachable, some are. If the Emory boss is worth his paycheck, he should be able to tell.
"Somehow" makes it sound mysterious and inexplicable. I'd be willing to bet that the truth is far less sensational. I could see a student tech assistant doing something like this on a dare, or a low-skilled admin just clicking OK one too many times, without actually reading the warning message.
We use SCCM extensively at my office, and yes, it's entirely possible to tell it to reimage every single computer. You just need to target the deployment at "All Systems" and make it mandatory. My guess is that some admin picked the wrong collection, which is fairly easy to do in SCCM 2007 (2012 has Collection folders, which helps with that), and there's no warning messages -- just a summary of "this deployment is going to these devices, click Finish to do it." Of course, most other mass management tools assume that the admins know what they're doing, so they don't have much in the way of guard rails either.
One of the more obnoxious elements of SCCM is that there's no real way to recall a command you send out; clients pick up policy at periodic intervals, and without manual intervention, they'll just grab the policy and do what it says even if you kill the server in question. You can block deployments by taking down distribution points (if the clients can't grab content, they won't run the deployment), but you still have to be fairly quick about it to stop it.
What we do to prevent these sorts of disasters is implement process around the use of the ConfigMgr console and ensure only the people who know how to use it actually use it. To prevent an OS reimaging incident, our OS deployments go through a static set of collections by process and are always optional (requiring a manual touch, either at PXE boot or in the UI) except for a specific set of collections that are segregated in their own folder and have names and descriptions with scary words that make it clear what's going to happen. For instance, in our "Clean Reimage" folder, we have a collection that says, "Windows 7 Reimage (Clean, PXE, Forced)" with a description to the effect of, "*** A computer placed in this collection will be REIMAGED and LOSE ALL LOCAL DATA. Local state is NOT preserved or transferred. ***" If we were a larger IT organization, we'd probably use SCCM's role-based security to limit access to clean reimages to a specific group of people.
The Freelance Wizard
That doesn't matter so much because things are changing at such a glacial speed. It may as well be 1999 for the small amount of 64 bit, multithreaded stuff that uses network capability well which is out there. If you defrosted a Sun sparc user from back then and put them on a Win8 machine they would be disappointed.
No, capability isn't enough. The student's personal computer still needs to be configured to PXE boot before hitting other boot sources. Even that wouldn't be enough. Something has to trigger a reboot. So, if the machine's boot order has PXE before hard drive, and has Wake on LAN configured, AND is powered off as opposed to merely sleeping or hibernating, then it *MIGHT* be affected. However Wake on LAN requires that the MAC address of the target computer be known by the issuer of the Wake on LAN command, the SCCM server in this case. The odds of all these prerequisites being in place for a student's personal computer is remote in the extreme.
When our name is on the back of your car, we're behind you all the way!