Slashdot Mirror


Emory University SCCM Server Accidentally Reformats All Computers Campus-wide

acidradio writes: "Somehow the SCCM application and image deployment server at Emory University in Atlanta accidentally started to repartition, reformat then install a new image of Windows 7 onto all university-managed computers. By the time this was discovered the SCCM server had managed to repartition and reformat itself. This was likely an accident. But what if it weren't? Could this have shed light on a possibly huge vulnerability in large enterprise organizations that rely heavily on automated software deployment packages like SCCM?"

31 of 564 comments (clear)

  1. Cool by rossdee · · Score: 5, Funny

    Sounds like a good way to get rid of Malware

    1. Re:Cool by Anonymous Coward · · Score: 5, Interesting

      Unfortunately, SCCM also supports Linux and Mac OSX clients. I wonder whether it tried to install Windows 7 on them also? Users would be really pissed to discover their Mac/Linux box was now lurching under Windows...

    2. Re: Cool by Anonymous Coward · · Score: 5, Funny

      I worked at Emory for years and I have no doubts this was sheer incompetence not sabotage.

    3. Re: Cool by Noah+Haders · · Score: 4, Funny

      I like to think it was the SCCM server itself that said fuck you all I've had enough. I'm pushing the red button and we're all going down.

    4. Re:Cool by camperdave · · Score: 5, Insightful

      No, capability isn't enough. The student's personal computer still needs to be configured to PXE boot before hitting other boot sources. Even that wouldn't be enough. Something has to trigger a reboot. So, if the machine's boot order has PXE before hard drive, and has Wake on LAN configured, AND is powered off as opposed to merely sleeping or hibernating, then it *MIGHT* be affected. However Wake on LAN requires that the MAC address of the target computer be known by the issuer of the Wake on LAN command, the SCCM server in this case. The odds of all these prerequisites being in place for a student's personal computer is remote in the extreme.

      --
      When our name is on the back of your car, we're behind you all the way!
    5. Re:Cool by mikael · · Score: 4, Interesting

      That's what some universities actually do. They have a custom built dual-boot OS partition image (Linux + Windows) will all the standard applications that have been licensed and required for lab use (Mathematica, Microsoft Word, Firefox, Opera). This image gets stomped onto the drive of every idle system every night. So even if some spyware installs itself overnight, it gets overwritten.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    6. Re:Cool by RabidReindeer · · Score: 3, Funny

      What no tape backup?

      Look at all the money we saved!

  2. SCCM server reformats itself? by Rick+Zeman · · Score: 5, Funny

    Kind of sounds like a snake eating its tail....

  3. Configuration deplorement by K.+S.+Kyosuke · · Score: 4, Funny

    The configuration deployment server apparently upgraded itself into a configuration deplorement server.

    --
    Ezekiel 23:20
  4. Sounds like IT incompetence by areusche · · Score: 4, Insightful

    SCCM is pretty good. It makes my desktop techs jobs significantly easier to deploy assets company wide. In this case, it sounds like someone pressed some buttons without being 100% clear as to what was going on. Unfortunate someone will not be working in IT ever again.

    1. Re:Sounds like IT incompetence by BitZtream · · Score: 5, Insightful

      Assuming it was just a mistake and not malicious ...

      Probably not. This shit happens, and that person who did it will never do something like this again. Have you ever made a massive, expensive mistake?

      I have, I was 19 years old and cost my company nearly a million dollars due to a silly misconfiguration. After I discovered it, corrected the error and notified my boss, I spent most of the night throwing up. The next morning, after everyone in the company (only 15 people or so) knew what happened, and I walked through the halls on the way to the meeting with the owner and my boss, I thought I'd pass out. As I walked into the Owner's office I didn't even bother to sit down, expecting a fairly short conversation. I was asked to sit down while my boss had this very stern look on his face. So I did, cost them that much money, I can do what they ask.

      The owner than proceeded to tell me the story of how, when working for a certain Germany car company doing CICS programming, he made a mistake that screwed up a production line and cost the company several million dollars. He knew exactly how I felt, and he knew that it would never happen again because I had already punished myself more than he possibly could.

      If they fire the person who did this, they just wasted the whole event. The person learned their lesson and will be extremely cautious in the future. Firing them now just means someone else will get to reap the benefits of this experience, and thats pretty stupid.

      People make mistakes, and in this case the software is at least partially responsible. The SCCM server should have aborted during the preflight checks when it realized it was going to take itself out in the process. The best thing this IT department can do is for the manager/director to keep the specific employe's name under wraps, stop shit from flowing down hill from above and move on. Nothing will benefit anyone if all of Emory treats the person responsible as if he deserves to pay for all the time lost in repairing the damage, he simply can't.

      The hard lesson has been learned by everyone, nothing else will make anyone any better off.

      --
      Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
    2. Re:Sounds like IT incompetence by Richy_T · · Score: 3, Insightful

      Or he could just be an incompetent shit.

      Don't get me wrong, I've made mistakes myself, perhaps not quite to the same level. Hopefully he is someone who can take a lesson but there are many who can't.

    3. Re:Sounds like IT incompetence by bitt3n · · Score: 5, Insightful

      Have you ever made a massive, expensive mistake?

      Glances woefully down at wedding ring...

      The person learned their lesson and will be extremely cautious in the future.

      Thinks back on previous three weddings...

    4. Re:Sounds like IT incompetence by rastos1 · · Score: 4, Funny

      Also known as:

      Harrisberger's Fourth Law of the Lab:
      Experience is directly proportional to the amount of equipment ruined.

    5. Re:Sounds like IT incompetence by jd2112 · · Score: 5, Interesting

      SCCM is pretty good. It makes my desktop techs jobs significantly easier to deploy assets company wide. In this case, it sounds like someone pressed some buttons without being 100% clear as to what was going on. Unfortunate someone will not be working in IT ever again.

      Or perhaps someone decided that having a testing environment for deployment packages was an unnecessary expense combined with personnel who aren't properly trained. Just think how much money they saved by eliminating training and a test environment!

      --
      Any insufficiently advanced magic is indistinguishable from technology.
    6. Re:Sounds like IT incompetence by bitt3n · · Score: 5, Funny

      Fourth unhappy one? you're not making mistakes, you have a problem

      Might as well face it, I'm addicted to love

    7. Re:Sounds like IT incompetence by JanneM · · Score: 3, Insightful

      People make mistakes. Everybody makes them, everybody does it all the time, and they do it even when they should know better, when the consequences are high, and when they've received training specifically aimed at avoiding those particular mistakes.

      Aviation, process and other industries know this by now, after many, many hard-earned lessons. They know you have to design your interfaces under the assumption that people will screw up, push the wrong button, or misread the situation. The general software industry, on the other hand, seems amazingly resilient against accepting this simple fact.

      --
      Trust the Computer. The Computer is your friend.
    8. Re:Sounds like IT incompetence by labnet · · Score: 3, Funny

      Fourth unhappy one? you're not making mistakes, you have a problem

      Might as well face it, I'm addicted to love

      What's love got to do with it?

      It's a second hand emotion.

      --
      46137
  5. Surprisingly Infrequent by crow · · Score: 4, Interesting

    I think the big surprise here is that this doesn't happen more often.

    Consider how many corporations, universities, and such have huge PC deployments with automated updates. I've seen updates that drop all the PCs off the network, but I've never seen one where everything is wiped.

    I'm also surprised that I haven't heard of malware that accidentally wiped a network of 100K or more machines when someone sent the wrong command.

    Or maybe the news here is that it was in a more open environment where people hear about it. If a publicly traded company wiped a thousand PCs at its headquarters, you bet they would try to keep it quiet.

    1. Re:Surprisingly Infrequent by FreelanceWizard · · Score: 3, Insightful

      We use SCCM extensively at my office, and yes, it's entirely possible to tell it to reimage every single computer. You just need to target the deployment at "All Systems" and make it mandatory. My guess is that some admin picked the wrong collection, which is fairly easy to do in SCCM 2007 (2012 has Collection folders, which helps with that), and there's no warning messages -- just a summary of "this deployment is going to these devices, click Finish to do it." Of course, most other mass management tools assume that the admins know what they're doing, so they don't have much in the way of guard rails either.

      One of the more obnoxious elements of SCCM is that there's no real way to recall a command you send out; clients pick up policy at periodic intervals, and without manual intervention, they'll just grab the policy and do what it says even if you kill the server in question. You can block deployments by taking down distribution points (if the clients can't grab content, they won't run the deployment), but you still have to be fairly quick about it to stop it.

      What we do to prevent these sorts of disasters is implement process around the use of the ConfigMgr console and ensure only the people who know how to use it actually use it. To prevent an OS reimaging incident, our OS deployments go through a static set of collections by process and are always optional (requiring a manual touch, either at PXE boot or in the UI) except for a specific set of collections that are segregated in their own folder and have names and descriptions with scary words that make it clear what's going to happen. For instance, in our "Clean Reimage" folder, we have a collection that says, "Windows 7 Reimage (Clean, PXE, Forced)" with a description to the effect of, "*** A computer placed in this collection will be REIMAGED and LOSE ALL LOCAL DATA. Local state is NOT preserved or transferred. ***" If we were a larger IT organization, we'd probably use SCCM's role-based security to limit access to clean reimages to a specific group of people.

      --
      The Freelance Wizard
  6. An...accident..? by geekmux · · Score: 5, Insightful

    Knowing that people have been running various kinds of centralized update services, perhaps across multiple OSes, and spanning several years now, listening to a story about an update server literally going rogue and nuking everything attached to it, and then for the coup de grace, basically committing suicide at the end by reformatting itself, does not sound like an accident.

    If it truly was, I'd hate to see what the hell purposeful intent looks like.

    1. Re:An...accident..? by Anonymous Coward · · Score: 5, Funny

      Might be interesting to see how the Emory Board files this away.

    2. Re:An...accident..? by bruce_the_loon · · Score: 3, Informative

      This isn't the update server section of System Center (WSUS), it's the machine deployment system (Configuration Manager), and it can quite easily do this if left as-is out of the box with multiple technicians on it. And it can be done accidentally.

      Here's the scenario as it likely happened.

      • Technician finished a master PC install task sequence and tested with one PC. Now he is ready to deploy to his computer lab.
      • There are two options for failure here. SCCM allows for collections of machines to be built for all purposes (data gathering, deployments etc), so he probably puts a quick group together and gets that step wrong and the collection includes all computers in the AD tree. One of our technicians did this after two years of using SCCM regularly.
      • Or he goes hunting for an existing collection and ends up selecting the default All Systems collection which includes everything. If there are a lot of collections or his is named too similarly.
      • After another hundred odd clicks, he hits deploy and SCCM sends a message to the client service on all computers in the selected collection to run the new deployment task sequence. Including the SCCM server because it also has a client and is in All Systems collection or gathered in an incorrectly specified collection.
      • Each PC then downloads the image, reboots and wipes itself with the image. The server, also in the collection, will do the same at some point.

      We've had two near-misses with misconfigured collections and one hit with a different problem* which cannot have happened in this case. SCCM isn't the most intuitive user interface and if you're being pressured by users or trying to get out of the door for the weekend, you can stuff it up easily.

      Our solution was to restrict access to the built-in collections and to build collections per computer lab which are presented as read-only to the technicians. And then gave them a day of lectures. It sort of works.

      * The other problem was caused by image dumping with Ghost of an image that was sysprepped, but had the SCCM client still installed on the image. Because of that, several dozen PCs had clients with the same client ID, like the Windows GUID, but separate and not cleared by a sysprep. The technician later built a SCCM image and deployed it correctly to one PC in a personal collection. Unfortunately SCCM populated the deployment list based on the client ID of the PC in the list and hit quite a few overnight. Luckily a lot of the machines in the batch were off overnight. I don't think this is the case because it hit the server too and that would have received a new client install during the SCCM installation.

      --
      Trying to become famous by taking photos. Visit my homepage please.
  7. Re:Oh man by Anonymous Coward · · Score: 5, Funny

    In a résumé, "Watched in horror as images were accidentally deployed" becomes "Supervised the deployment of images on university-managed computers".

  8. Wrong OS by Air-conditioned+cowh · · Score: 5, Funny

    It reformatted the drives and put Windows on them. Eeewww! That's gross!

  9. Backups by wisnoskij · · Score: 3, Insightful

    Bad news most likely on this front. I have worked University IT, and I can guarantee they are going to have problems.

    For one, no matter how many layers of backups you have, when you are working with a bunch of 90 year old academics, they will always find a way to miss every single one.

    And more grievous, Universities tend to have important data that absolutely cannot be backed up in any normal way. Data that is legally obligated to stay on one specific computer in one specific room and never leave; under penalty of legal action.

    --
    Troll is not a replacement for I disagree.
  10. Only a good manager could tell the difference by Anonymous Coward · · Score: 5, Insightful

    It sounds like the commenter above was teachable - he no doubt learned his lesson.
    It also sounds like the company's owner knew he could learn this lesson. That's the mark of a great manager.

    Whether the Emory staffer responsible for this mistake is teachable or not, I hope his boss can tell the difference. Some folks aren't teachable, some are. If the Emory boss is worth his paycheck, he should be able to tell.

    1. Re:Only a good manager could tell the difference by Sun · · Score: 3, Insightful

      I used to work for a company called "Gteko". Don't bother looking them up - they were acquired several years ago. They sold bundled software (OEM) to a handful of companies, all of them huge. One of those was AOL. This is over a decade ago.

      The incident in question took place after I left, so I don't know the specifics. The bottom line is, they screwed up a server deployment that affected the AOL front page for all AOL customers. After that was finally fixed, the company's CEO, expecting pretty much to be shown the door, walked into a meeting with several AOL high execs.

      The meeting started with the following sentence:
      "Let's see how we can make sure this never happens again"

      Even when it's something less "close" to you than an employee, it is sometimes worth it to not terminate someone who made a mistake, even a serious one.

      My current employer, Akamai, has a motto effectively saying: It's okay to screw up, so long as that screwup results in a procedure that will prevent anyone from making the same mistake again.

      Shachar

  11. "Somehow"??? by Tony+Isaac · · Score: 3, Insightful

    "Somehow" makes it sound mysterious and inexplicable. I'd be willing to bet that the truth is far less sensational. I could see a student tech assistant doing something like this on a dare, or a low-skilled admin just clicking OK one too many times, without actually reading the warning message.

  12. Re:Not so sensational... by dbIII · · Score: 3, Insightful

    As much as older IT folks don't want to admit it, they don't learn as quickly as they did when they were younger

    That doesn't matter so much because things are changing at such a glacial speed. It may as well be 1999 for the small amount of 64 bit, multithreaded stuff that uses network capability well which is out there. If you defrosted a Sun sparc user from back then and put them on a Win8 machine they would be disappointed.

  13. Re:backups by Gothmolly · · Score: 3, Funny

    That's what RAID-5 is for, jeez.

    --
    I want to delete my account but Slashdot doesn't allow it.