Slashdot Mirror
Emory University SCCM Server Accidentally Reformats All Computers Campus-wide
acidradio writes: "Somehow the SCCM application and image deployment server at Emory University in Atlanta accidentally started to repartition, reformat then install a new image of Windows 7 onto all university-managed computers. By the time this was discovered the SCCM server had managed to repartition and reformat itself. This was likely an accident. But what if it weren't? Could this have shed light on a possibly huge vulnerability in large enterprise organizations that rely heavily on automated software deployment packages like SCCM?"
Sounds like a good way to get rid of Malware
Time to test those backups!
Kind of sounds like a snake eating its tail....
The configuration deployment server apparently upgraded itself into a configuration deplorement server.
Ezekiel 23:20
The problem with centralized control is that the center can give any commands it wants...
A thousand pounds of wood moving at 300 feet per minute. Don't get in the way.
SCCM is pretty good. It makes my desktop techs jobs significantly easier to deploy assets company wide. In this case, it sounds like someone pressed some buttons without being 100% clear as to what was going on. Unfortunate someone will not be working in IT ever again.
I think the big surprise here is that this doesn't happen more often.
Consider how many corporations, universities, and such have huge PC deployments with automated updates. I've seen updates that drop all the PCs off the network, but I've never seen one where everything is wiped.
I'm also surprised that I haven't heard of malware that accidentally wiped a network of 100K or more machines when someone sent the wrong command.
Or maybe the news here is that it was in a more open environment where people hear about it. If a publicly traded company wiped a thousand PCs at its headquarters, you bet they would try to keep it quiet.
I bet the IT department is changing each other's diapers now! And updating their resumés....
Mostly random stuff.
Considering how easy it would be for a less-than-savvy IT person to accidentally encourage this situation, I doubt this is a sensational case of some huge vulnerability.
As someone who regularly provides consultation to IT staff, I know full well that there's plenty of 'administrators' that wade into waters they don't understand. We often encounter the aging IT staff member that's forced to interact with software they don't quite understand or we have the younger IT staff that impulsively click on what they don't understand, both occasionally leading a company to some manner of pandemonium level disaster. Or you simply have a dysfunctional IT department that doesn't communicate and, "oh, I'll just move this server into this container right here..." Just another day in IT.
Knowing that people have been running various kinds of centralized update services, perhaps across multiple OSes, and spanning several years now, listening to a story about an update server literally going rogue and nuking everything attached to it, and then for the coup de grace, basically committing suicide at the end by reformatting itself, does not sound like an accident.
If it truly was, I'd hate to see what the hell purposeful intent looks like.
This is University. Based on my experience with University IT, there will be loads and loads of important data that you are legally obligated to NOT do that to. It cannot leave one specific room, in any form.
Normally, the computers are still contacted to the network and the Internet, but everyone using them must know NOT to copy any of these files off of C.
Troll is not a replacement for I disagree.
It reformatted the drives and put Windows on them. Eeewww! That's gross!
Bad news most likely on this front. I have worked University IT, and I can guarantee they are going to have problems.
For one, no matter how many layers of backups you have, when you are working with a bunch of 90 year old academics, they will always find a way to miss every single one.
And more grievous, Universities tend to have important data that absolutely cannot be backed up in any normal way. Data that is legally obligated to stay on one specific computer in one specific room and never leave; under penalty of legal action.
Troll is not a replacement for I disagree.
So there are laws which dictate which hard drives and/or appliances store data relative to the OS? They can still be in the same room if that's the concern but if there are laws that actually say "x, y, and z must be stored on the same partition as the operating system" then I say they get what they deserve and perhaps those laws need to be re-examined.
When I worked in the IT in my work study program, shared computers would be re-imaged often. We would usally re-image 300-400 computers at time. Often it was just some professor wanting certain program, it was just easier and safer just to wipe the machines. Malware was big problem at the time, sasser hit all of are computers, that sucked.
It sounds like the commenter above was teachable - he no doubt learned his lesson.
It also sounds like the company's owner knew he could learn this lesson. That's the mark of a great manager.
Whether the Emory staffer responsible for this mistake is teachable or not, I hope his boss can tell the difference. Some folks aren't teachable, some are. If the Emory boss is worth his paycheck, he should be able to tell.
mac systems may not even boot with the old Partition tables that are needed for older NON EFI systems that windows runs on.
also the Mac os Recovery Partition may even be wiped out.
"Somehow" makes it sound mysterious and inexplicable. I'd be willing to bet that the truth is far less sensational. I could see a student tech assistant doing something like this on a dare, or a low-skilled admin just clicking OK one too many times, without actually reading the warning message.
Not only have they had to re-image all their PCs, you've now slashdotted their web server!
So it goes, so it has been, so it will be.
Ecclesiastes 1:9 and from Battlestar Galactica the new one. "All this has happened before and will happen again."
apple uses EFI / UEFI.
older Pc's don't have it and in some cases with pc that have it have it turned off so they can boot XP, Windows 7 (in some cases), disk encryption, and other stuff.
32bit os's yes some places put 32 bit loads on systems with 4gb or more ram in them.
Well in some cases that's for the best. 4GiB or even 8GiB is the standard amount of memory of many computers, not that one would want any thing less anyway (it wouldn't lower the costs noticeable).
And there are reasons to run 32 bit OS installations on 64 bit systems, I've have to use a separate 32 bit Windows installation in a VM to run some important programs.
That would be silly. ;)
It would be as silly as... wiping all the computers of an entire university
Privacy is terrorism.
Don't underestimate IPMI and its equivalent (what was Intel's name for their proprietary alternative? ME ?)
With this kind of technologies, you have a small mini-embed system in the motherboard, which talk over a TCP/IP network and provides all functionnality (including wake-on-lan, including shutdown, including reboots (AND specifying what resource to reboot to - like starting PXE or emulated-over-network USB driver), including VNC remoting, and Serial-over-IP (for some server remoting).
Think of all the niceties you have inside a VirtualBOX or VMWare emulator, but on actual real hardware.
It's very popular in enterprise settings so sysadmins can update a whole division's computers, instead to having to walk to each one of hundreds of machine).
So it available under some form on most workstations, servers, and enterprise laptops.
If you need to upgrade machines, the usual procedure is:
1. you politely ask the user to not start any over-night computation and if possible leave their machine off.
2. when the evening comes you log into the lights-out management of each of the target the workstation over *IP* (no MAC-address shenningans, straith IP address) (you either have simple accounts with password, or you can have it integrated to some larger systems)
3. because none of the user would have actually left the workstation off as requested, you first try to nicely shutdown the machines (you send an ACPI soft-off signal, and hope that the OS will react and nicely save and shutdown).
4. after a timeout you force shutdown the remaining machines (you send a Reset or Power OFF signal)
5. you boot all the machine and ask them to load your payload instead of the disk (it can be either classical PXE, or you can remotely emulate a USB drive) so all machine boot the management software even that bizare guy who insist on having CD as his primary boot device on his workstation.
6. you remotely launch the necessary administration. Either it runs unattended, or you can remotely control if needed (depending on the tool used, you might use a proprietary administration tool, or SSH, or Serial-over-Internet. If it's an asinine tool, you might need to VNC)
7. you reboot the machine and let them follow normal boot order.
The system has lots of advantage:
- it can be scripted with command line tools.
- you can even change BIOS settings, etc.
- it's handled by an embed system (usually part of the chipset) so this is completely independent of what the main CPU is doing (the workstation might be running or might be powered off).
The system has a few drawbacks:
- massive security problems: the user-friendly web-interface of lots of "lights-out" implementation is buggy and an exploit-nightmare (most securing recommandation start by "turn the web console off").
- security problem: you need to setup proper accounts and passwords (to avoid a script kiddie wrecking havock after having guessed a few standard passwords. Minimum is setting acceptable local accounts)
So, going back to the discussion:
- if the workstations have Lights-out remote administration (virtually all enterprise hardware is sold with)
- if the remote administration is properly setup...
You have full control possibilities on the whole fleet of workstations, without any of the hassle of dealing with low-level functionnalities like PXE, WOL, etc.
If an administrator has done a bad job at automating it all, you might end up with the whole enterprise being remotely reformated.
At least the students machine are probaly safe:
- the consumer grade laptops don't usually feature "lights-out" management.
- the student who has bought an enterprise-class laptop probably hasn't it activated (and if the student is savvy enough to turn it on for own use, the student has sure enough setup decent accounts).
Given the security problems mentionned above, that means NSA and China have probably full administrative access to 1/3 of all enterprise workstations running everywhere.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]