Emory University SCCM Server Accidentally Reformats All Computers Campus-wide

acidradio writes: "Somehow the SCCM application and image deployment server at Emory University in Atlanta accidentally started to repartition, reformat then install a new image of Windows 7 onto all university-managed computers. By the time this was discovered the SCCM server had managed to repartition and reformat itself. This was likely an accident. But what if it weren't? Could this have shed light on a possibly huge vulnerability in large enterprise organizations that rely heavily on automated software deployment packages like SCCM?"

Comments

Re:An...accident..?

[bruce_the_loon]

This isn't the update server section of System Center (WSUS), it's the machine deployment system (Configuration Manager), and it can quite easily do this if left as-is out of the box with multiple technicians on it. And it can be done accidentally.

Here's the scenario as it likely happened.

• Technician finished a master PC install task sequence and tested with one PC. Now he is ready to deploy to his computer lab.
• There are two options for failure here. SCCM allows for collections of machines to be built for all purposes (data gathering, deployments etc), so he probably puts a quick group together and gets that step wrong and the collection includes all computers in the AD tree. One of our technicians did this after two years of using SCCM regularly.
• Or he goes hunting for an existing collection and ends up selecting the default All Systems collection which includes everything. If there are a lot of collections or his is named too similarly.
• After another hundred odd clicks, he hits deploy and SCCM sends a message to the client service on all computers in the selected collection to run the new deployment task sequence. Including the SCCM server because it also has a client and is in All Systems collection or gathered in an incorrectly specified collection.
• Each PC then downloads the image, reboots and wipes itself with the image. The server, also in the collection, will do the same at some point.

We've had two near-misses with misconfigured collections and one hit with a different problem* which cannot have happened in this case. SCCM isn't the most intuitive user interface and if you're being pressured by users or trying to get out of the door for the weekend, you can stuff it up easily.

Our solution was to restrict access to the built-in collections and to build collections per computer lab which are presented as read-only to the technicians. And then gave them a day of lectures. It sort of works.

* The other problem was caused by image dumping with Ghost of an image that was sysprepped, but had the SCCM client still installed on the image. Because of that, several dozen PCs had clients with the same client ID, like the Windows GUID, but separate and not cleared by a sysprep. The technician later built a SCCM image and deployed it correctly to one PC in a personal collection. Unfortunately SCCM populated the deployment list based on the client ID of the PC in the list and hit quite a few overnight. Luckily a lot of the machines in the batch were off overnight. I don't think this is the case because it hit the server too and that would have received a new client install during the SCCM installation.