Slashdot Mirror

← Back to Stories (view on slashdot.org)

UK Hosting Provider 123-Reg Accidentally Deletes Customer Sites (theregister.co.uk)

Posted by msmash on from the blame-the-rogue-certificate dept.

Richard Chirgwin, writing for The Register: UK hosting and domains provider 123-Reg has been struck by a weekend issue that knocked an unspecified number of VPS customers offline. The company posted a status message saying that the unspecified issues arose on April 16. 123-reg customer, software company INNmaster, contacted The Register directing our attention to its post on the topic, claiming a rogue script had deleted customer sites (such as this one). Slashdot has independently received several tips for this incident. Here's a copy of the email that Richard Winslow, Director of 123-Reg sent to the affected customers.

64 comments

  1. 123-Reg by Anonymous Coward · · Score: 1

    Sounds like some sort of new radioactive isotope

    1. Re:123-Reg by Anonymous Coward · · Score: 0

      Maybe that's why they felt the urge to repeat the same phrase about leaving stones unturned twice within two paragraphs.

  2. It's the singularity by 14erCleaner · · Score: 1
    This is the second time this week a "rogue script" has deleted a web hosting provider. I sense a pattern.

    Oh, and the obligatory XKCD

    --
    Have you read my blog lately?
    1. Re:It's the singularity by 14erCleaner · · Score: 3, Funny

      It's agents in the present will probably mod you down for saying that.

      --
      Have you read my blog lately?
    2. Re:It's the singularity by Anonymous Coward · · Score: 0

      This is why you don't run Steam on your company's customer web host.

      CAPTCHA: botches

    3. Re:It's the singularity by Anonymous Coward · · Score: 0

      Sure enough, my boss just got a call from Herman telling him to tell me to get back to work.

    4. Re:It's the singularity by 14erCleaner · · Score: 1

      You also got a mysterious "-1 overrated" downmod.

      --
      Have you read my blog lately?
  3. So that Serverfault question wasn't a hoax by hardill · · Score: 5, Funny

    Looks like that Serverfault question about running rm -rf / was real then

    1. Re: So that Serverfault question wasn't a hoax by Anonymous Coward · · Score: 0

      Funny, I thought the exact same thing about a question titled "HARD DR RECOVERY PLS HELP" question I saw this morning.

    2. Re:So that Serverfault question wasn't a hoax by AmiMoJo · · Score: 1

      Indeed. It looks like they had a script that was supposed to delete old, unused virtual machines. Due to a database error, similar to the Serverfault hoax, the script get a null value which made it think many active machines were in fact dead and so it deleted them.

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    3. Re:So that Serverfault question wasn't a hoax by Anonymous Coward · · Score: 0

      I have worked with several virtual platforms and they all had various protections.

      Don't most throw errors "This machine is in use" "This storage is attached to a running VM" and so on.

      Who runs a script with all the requisite --force --ignore-all-common-sense flags that deletes running VMs.

    4. Re:So that Serverfault question wasn't a hoax by Anonymous Coward · · Score: 1

      Sysadmins running with the --complete-idiot option.

    5. Re:So that Serverfault question wasn't a hoax by Hylandr · · Score: 1

      Or the poor admin that thought he was in the dev environment.

      --
      ~ People that think they are better than anyone else for any reason are the cause of all the strife in the world.
    6. Re:So that Serverfault question wasn't a hoax by Anonymous Coward · · Score: 0

      Linux will happily let you delete files that are in use. It'll remove the file name, but the file's data will continue to exist until all handles to it are closed. This potentially means that they could have accidentally deleted everything months ago and not noticed until they for some reason needed to reboot the server hosting the virtual machines.

    7. Re:So that Serverfault question wasn't a hoax by Krojack · · Score: 1

      Referring to this one?

      http://brobible.com/life/artic...

  4. More space by Anonymous Coward · · Score: 0

    Congratulations. Your account has been upgraded with more free disk space.

    1. Re:More space by Archtech · · Score: 2, Funny

      Congratulations. Your account has been upgraded with more free disk space.

      Said the BOFH... 8-)

      --
      I am sure that there are many other solipsists out there.
  5. A hot I worked for did this once by thaylin · · Score: 1

    In that case a "rogue script" meant that they were trying to prove that anyone could be an administrator so gave a sales person root access who promptly ran rm -rf /*.*.

    On top of that their were no daily backups as promised for that server, because they would not invest money in it.

    ait.com

    Ahh those were some of the worst days of my life, but learned a lot fixing other people's mistakes.

    --
    When you cant win, ad hominem.
    1. Re:A hot I worked for did this once by TheRaven64 · · Score: 3, Insightful

      so gave a sales person root access who promptly ran rm -rf /*.*.

      Running rm -rf /*.*. (or even rm -rf /*.*) is unlikely to cause much damage on a *NIX system as it's very unusual to have any files with dots in their names in the root directory and even more unlikely to have any directories that have dots in their names.

      --
      I am TheRaven on Soylent News
    2. Re:A hot I worked for did this once by Anonymous Coward · · Score: 3, Funny

      > Running rm -rf /*.*. (or even rm -rf /*.*) is unlikely to cause much damage on a *NIX system [...]

      Alas, times have changed. Nowadays every idiot and his dog, coming over from DOS (perhaps via OSX [1]) has the irresistible urge to name his shell scripts "foo.sh". Those are the very same people who think "all files" is spelt "*.*", so the only consolation for all that ugliness is that they bring their own antimatter along. May they annihilate themselves in a flash of light!

      [1] Yeah, yeah. OSX is Unix beneath the covers, they say. But the culture, the spirit, the soul? Rather an Unix zombie, no soul and eating braaaains.

    3. Re:A hot I worked for did this once by thaylin · · Score: 1

      I am sorry, I was trying to be a bit funny and serious at the same timet. We named our customer data partitions /vs{#}. What she did was do a rm -rf /vs* deleting all the customers data, the OS was perfectly fine.

      I probably should have stuck to the serious part of it.

      --
      When you cant win, ad hominem.
    4. Re:A hot I worked for did this once by Anonymous Coward · · Score: 1

      On solaris * resolves to something or nothing, so /*.* resolves to /. which is the same as /
      /*.*. resolves to /.. which also is / (/ is special like that)
      Even on current Linux systems, /.* resolves to / so be careful deleting hidden directories en masse!

    5. Re:A hot I worked for did this once by OzPeter · · Score: 1

      sales person root access who promptly ran rm -rf /*.*.

      Back a long long time ago in place far far away I was working in Unix on a VAX. It was late, I was tired. I wanted to delete a bunch of files. Some of the files I wanted to delete matched the pattern fred.* while others matched the pattern *.barney. So in my muddled head I thought that I could save a typing one command by combining the two patterns and doing rm *.* Of course I realized my mistake the moment I hit return. Fortunately it was in a local directory and I could easily recreate what I had lost.

      --
      I am Slashdot. Are you Slashdot as well?
    6. Re:A hot I worked for did this once by Anonymous Coward · · Score: 0

      Running rm -rf /*.*. (or even rm -rf /*.*) is unlikely to cause much damage on a *NIX system as it's very unusual to have any files with dots in their names in the root directory and even more unlikely to have any directories that have dots in their names.

      Indeed... checked on 2 machines I have access.
      On a Mint desktop:

      $ ls /*.*
      /initrd.img /local.cfg

      And on a Gentoo server:

      $ ls /*.*

      ls: cannot access /*.*: No such file or directory

      So, yeah... not much harm done.

    7. Re:A hot I worked for did this once by rickb928 · · Score: 1

      Salesperson didn't know about & and the power of, well, you know...

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    8. Re:A hot I worked for did this once by trevc · · Score: 1

      Riveting

    9. Re:A hot I worked for did this once by geantvert · · Score: 1

      The parent post was not about using unnecessary extensions but about having files or directories with an extension at the ROOT of the filesystem.
      None of /usr /etc /tmp .. match the pattern /*.* so rm -rf /*.* should be mostly harmless on most systems.

      For instance, on my current box, the only thing that would be removed is the symlink /initrd.imh -> boot/initrd.img-...
      That would probably prevent me to boot but that is easy to repair.

      Regarding the use of the .sh extension for shell script, I kind of disagree with you.
      Extensions are of course not strictly needed since the OS and most applications (not all) do not use them to figure out the filetype but extensions are still a convenient way to identify the nature of a file without having to open it (yeah! yeah! I know the 'file' command).

      I confess! I have been using Linux/UNIX almost exclusively for the last 20 years and I still add .sh or .pl to some of my shell and perl scripts even though I know that neither the OS nor my editor will need the extension. That is not as if I had to type the extension each time I want to run the script. Completion is my friend.

    10. Re:A hot I worked for did this once by geantvert · · Score: 1

      /local.cfg ... not sure what that can be and if that is really supposed to be there.

      If you are not sure then remove it :-)

    11. Re:A hot I worked for did this once by phishybongwaters · · Score: 1

      I have never added extensions to any of my scripts for any other reason than I'd likely forget what the hell it was and sometimes I don't feel like looking or grepping the first line to get the declaration.

    12. Re:A hot I worked for did this once by Anonymous Coward · · Score: 0

      Except for . and .. which refer to the current directory (yes, .. is parent, but parent of / is still /), and thus recursively remove everything.

    13. Re:A hot I worked for did this once by TheRaven64 · · Score: 1

      Neither will be matched by '*.*'. Hidden files, . and .. would be matched by '.*', but a '*.*' glob pattern requires characters before the dot. A '.*' pattern is particularly dangerous for this reason - it looks like it will only match hidden files, but it will also match the current directory and the parent directory.

      --
      I am TheRaven on Soylent News
    14. Re:A hot I worked for did this once by SvnLyrBrto · · Score: 1

      Hey. Don't drag Mac users into this one. Historically, we were used to using type and creator codes in the file's resource fork, naming our files whatever we damned well pleased; and mocked the windows people for their 8.3 filenames every bit as much as the Unix guys did. So even pre-OS X, no *.* people here.

      --
      Imagine all the people...
    15. Re: A hot I worked for did this once by Anonymous Coward · · Score: 0

      *.* ? Seriously? How about just * ? Go back to DOS.

    16. Re:A hot I worked for did this once by Junta · · Score: 1

      /*.* for me resolves for literal '/*.*'. Generally shells try to glob expand, and pass as literal on failure. So if you have no files in / with '.' in them, it will try to remove, literraly, a single file/dir named '*.*'.

      --
      XML is like violence. If it doesn't solve the problem, use more.
    17. Re:A hot I worked for did this once by _merlin · · Score: 1

      Bourne shells pass as literal on failure to match glob. This has always irritated me as commands like "find . -name *.c" do radically different things depending on whether there are any .c files in the current directory. C shells don't pass on a literal glob if it doesn't match - they return an error without running the command.

  6. crybaby by Anonymous Coward · · Score: 0

    INNMasters never contacted The Register, but they did make a lot of fuss on Twitter about it. Someone else's bad PR is good PR if you get involved in some way.

  7. Incompetence abounds by MachineShedFred · · Score: 1

    So either of the following happened:

    1. There is a massively incompetent administrator who scripted a blind delete without testing
    2. There is a massively incompetent administrator who left security open enough that someone else could delete multiple customers stuff

    I guess they just don't build them like they used to, because 'cloud'.

    --
    Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    1. Re:Incompetence abounds by Anonymous Coward · · Score: 0

      In the Cloud everything is somebody else's problem.

    2. Re: Incompetence abounds by Anonymous Coward · · Score: 0

      With the cloud came devs doing ops, often being confused as DevOps. This in turn triggered ops to become devs. The result: strange and unusual practices.

  8. Hey Rocky! Watch me pull a rabbit out of my hat by fustakrakich · · Score: 3, Interesting

    Again?

    --
    “He’s not deformed, he’s just drunk!”
    1. Re:Hey Rocky! Watch me pull a rabbit out of my hat by Anonymous Coward · · Score: 0

      2 years difference, UK vs Italy. What's your point besides not reading ?

    2. Re:Hey Rocky! Watch me pull a rabbit out of my hat by Gravis+Zero · · Score: 1

      that was a stupid hoax. this is just stupid.

      --
      Anons need not reply. Questions end with a question mark.
  9. I feel sorry... by Anonymous Coward · · Score: 0

    ... for these cats. It does kinda sound like they're pulling it together.

    We all makes mistakes (i.e. Fuck up) once in a while... The severity is only the humorous factor.

  10. yip! yip! yip! by Anonymous Coward · · Score: 0

    another stupid story

  11. Let me guess. by Anonymous Coward · · Score: 1

    Perchance, was it "just one line of code"?

    "Webservers hate this one weird trick!"

    "Find out the ancient neckbeard secret servers don't want you to know!"

  12. Hey I want some tooooo by Anonymous Coward · · Score: 0

    The other guy got a big boost in business by saying it happened to him, lets do the same. People are gullible, they won't think another company would possibly do the same hoax so soon.

    1. Re:Hey I want some tooooo by phishybongwaters · · Score: 1

      I'm still baffled as to how a hoax showing your utter incompetence could lead to increased business. Is this the same thing as people getting reservations at places they saw on Kitchen Nightmares?

  13. Resolution [Re:A hot I worked for did this once] by XXongo · · Score: 2

    On solaris * resolves to something or nothing, so /*.* resolves to /.

    Wait, you said that the command eliminates slashdot?

    I'm trying to decide if that's good, or bad...

  14. Don't accuse me of poetry by TechyImmigrant · · Score: 2

    Cloud so convenient
    Service provider not bright
    All the files are gone

    --
    I should use this sig to advertise my book ISBN-13 : 978-1501515132.
    1. Re:Don't accuse me of poetry by msmash · · Score: 1

      best thing I have read in weeks.

    2. Re:Don't accuse me of poetry by Anonymous Coward · · Score: 0

      Burma shave?

    3. Re:Don't accuse me of poetry by WallyL · · Score: 1

      The only thing you read is slashdot?!

  15. Re:Don't accuse me of poetry says ISIS follower by Anonymous Coward · · Score: 0

    Were not expect much from islamic muslimic terroristicate.

  16. ./pu.sh by tepples · · Score: 1

    Nowadays every idiot and his dog, coming over from DOS (perhaps via OSX [1]) has the irresistible urge to name his shell scripts "foo.sh".

    Especially when a script to back up the day's work to a remote server is called pu.sh.

  17. 30+ posts and no... by s0litaire · · Score: 0

    â¦comment of :

    "â¦and nothing of substance was lost."

    ^__~

    --
    Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
  18. Which is worse by trevc · · Score: 1

    A hosting provider deleting all your stuff by accident or a hosting provide deleting all your stuff on purpose?

  19. Two words pointedly missing from that email by gsslay · · Score: 1

    1/ Sorry
    2/ Apologise

  20. Another publicity stunt by nikkipolya · · Score: 1

    Looks like this is another publicity stunt, just like the hoax where a man deleted all his files with a rm -rf .

  21. LINK IS A GOOGLE DOC WHICH CAN TRACK YOU by michaelcole · · Score: 1

    https://www.quora.com/Is-there... Link is a google doc. Which appears to be able to track visitors logged in with a google account. Sorry about your emails.

  22. Oh yeah! by petes_PoV · · Score: 1

    Senior admin to millenial newbie: "What ever you do, don't run the delete_all_evidence.sh script on the production systems. We only use that when the Feds raid us. It completely scrubs the boxes".

    newbie: "Oh .... yeah. So it does. Where do I click undelete"

    --
    politicians are like babies' nappies: they should both be changed regularly and for the same reasons
  23. The .sh extension by Anonymous Coward · · Score: 0

    OK, I was kinda tongue-in-cheek. But still: for an executable script, the extension is silly: if you change the implementation of the script from -- say -- shell to Python, do you correct all call sites? The caller shouldn't know whether the executable is a shell or an ELF or a Perl script. That's what binfmt_misc is for, after all. That's why you have an abstract interface, with argc, argv and the environment.

    If the implementation matters (say if it's a shell script to be sourced from another shell script), it's a completely different cup of tea.

    1. Re:The .sh extension by Anonymous Coward · · Score: 0

      I think it's at least partly a matter of there being a general convention for source.

      People don't generally write, say, C code with no extension and then compile that into a program with no extension - The program gets written using .h and .c extensions, and then compiled into a program with no extension. For things like .sh, the source and the "binary" are the same thing, so you end up having programs with what is essentially a 'source' extension. (.sh, .pl, etc) (something like a makefile with an 'install' target that knows 'just write this to the install location with no extension' would make this more analogous to the 'source code -> shippable binary' case of something with a compile pass).

      I had a case recently where I was replacing a shell script with something else, and I just changed the shell script (which people were used to invoking, and other scripts/cron jobs/etc called) so that all it did was call the replacement. I didn't have to change all call sites immediately - and the contents of the .sh file still matched the language people would expect based on the extension.

  24. That's nothing! by Anonymous Coward · · Score: 0

    That's nothing!

    I accidentally the entire Internet.