Follow-up on EVE's Boot.ini Issue

Krinsath writes "CCP, publishers of Eve Online, have posted a Dev Blog detailing the circumstances leading up to the deletion of XP's boot.ini file, which was earlier discussed on Slashdot. The blog post has intimate details about how the mistake occurred (a new installer from their normal one), how they responded and what CCP has learned from it. While fairly dry, it is to the company's credit that they're being open about one of the more serious bugs to crop up in gaming's recent history."

Comments

That's actually a really straightforward response.

[Silverlancer]

Now if only more businesses acted this way.

Re:That's actually a really straightforward respon

[Harmonious Botch]

But they should delete greater percentages of XP...

How is that even possible

[AndrewBuck]

From the article...

"Why doesn't Windows protect its system startup files? That's a good question, one that I have asked myself in these last few days and wish I knew the answer. But of course I'm not going to blame Microsoft for our mistake. Windows doesn't protect those files and therefore software developers must take care not to touch them. We should have been more careful."

That is a good question. I am not an EVE player myself so I don't know if this update had to be run with admin privileges but it doesn't appear to be that way from the question and reply. If you are not running as admin then how is it even possible to remove a system file that is necessary to boot the system. Unlike the EVE representative making this statement I am going to blame Microsoft, it should not be the developers responsibility to make sure they don't break the OS, it is the OS developers responsibility to make sure that it cannot be broken without admin/system/root access.

-Buck

Re:How is that even possible

[Osty]

That is a good question. I am not an EVE player myself so I don't know if this update had to be run with admin privileges but it doesn't appear to be that way from the question and reply. If you are not running as admin then how is it even possible to remove a system file that is necessary to boot the system. Unlike the EVE representative making this statement I am going to blame Microsoft, it should not be the developers responsibility to make sure they don't break the OS, it is the OS developers responsibility to make sure that it cannot be broken without admin/system/root access.

Two things to note:

1. This was an XP problem. Technically it could've happened on Vista, but I haven't seen anything that said it did. As such, this falls into the same category of problems that Microsoft attempted to fix in Vista with UAC -- nearly everybody ran XP as admin, and many apps expected you to be running as admin.
2. This was a problem with an installer/uninstaller. Since nearly everything on Windows installs into %programfiles% and that's a shared location, installers need admin access (installers that ask if you want to install for "Just this user" or "Everyone" are not going to install in %userprofile% if you choose "Just this user". They're just looking to see if the Start Menu shortcuts should go into "%appdata%\Microsoft\Windows\Start Menu" or "%allusersprofile%\Microsoft\Windows\Start Menu"). Vista will elevate your privleges when you try to run an installer (you'll get a UAC prompt), after which a misbehaving installer could screw up boot.ini. Regardless of operating systems, you almost always install applications as administrator. Yes, you can install apps in $HOME on *nix systems, but 9 times out of 10 you'll use sudo on the installer (sudo apt-get install foo). Therefore this is technically a bug that could happen on any OS. It's not difficult to imagine an application install that deletes your kernel image, for example.

The real WTF here is that they have an important game file named "boot.ini". That's an exceedingly poor choice of filename. Think of it like having a game file called "autoexec.bat" or "vmlinuz" that actually has nothing to do with the DOS boot process or the Linux kernel. The only defense they give for that is "legacy".

Re:How is that even possible

[Bellum Aeternus]

I did patch EVE as a non-privileged user on my XP Pro system and the problem didn't happen to me. It does seem to be that since most people basically have to run as an administrative account to make XP "work", CCP was able to damage the OS as they did.

This is a multi-part failure. One part Microsoft for making an OS that almost requires standard users to run a privileged account all the time to make basic applications work. One part CCP for developing software that damaged the underlying OS.

My only hope is that either Microsoft begins developing software that not only protects users from the outside (they're still yet to do even this) as well as from themselves. They should take a page from Apple's book on how to do this; or Linux adoption picks up and we start seeing more of the applications I (and the bulk of business and home users) need being developed for it.

Re:How is that even possible

[mav[LAG]]

Technically it could've happened on Vista, but I haven't seen anything that said it did.

Well, that would require a group of people who have Vista installed.

Re:How is that even possible

[RulerOf]

Didn't Quake have an autoexec.bat file as a startup script? Quake 3, and I assume for 2 and 1, contained a file called "autoexec.cfg." I always thought it was aptly named, being a DOS veteran myself, because it contains game configs like default keybindings (e.g. bind w +move) and such that actually allow you to control the game in the first place, and it's always called during game startup. Very similar in function to the file that it is named after.

Re:How is that even possible

[Deviate_X]

"how is it even possible to remove a system file that is necessary to boot the system. Unlike the EVE representative making this statement I am going to blame Microsoft,"

The boot.ini file is actually protected. It is specifically marked as a System File, Read-Only and Hidden. This means that to modify the file you need to remove these attributes in a specific order first before you can modify or delete it, even if you are logged in as Administrator.

The only way to prevent software from doing bad things to important system files not to store important system information in text files on the filesystem...

Straightforward, sure.. but... | also, the bug

[Animaether]

what of the users who did lose valuable computer time due to this problem? The proverbial kid handing in their homework (or dissertation paper or whatever), for example. Apologizing and willing to pay for a third party tech support service (e.g. Geek Squad) is nice and all, but does that cover damages incurred? doubtful. Perhaps that EULA will finally get a test.

As for the bug itself... the installer code is NSIS script; quite powerful, but you do need to know what you're doing. Especially with a command such as "Delete", I can't help but wonder who failed to RTFM (TFM reads, as they point out, that "Delete" requires a full path to be safe or else it expects the path to be root) and instead made an -assumption- on how it would work.

Now, to their defense, NSIS is also a little inconsistent (RMDir needs /r to be recursive, but DeleteRegKey needs /ifempty to NOT be recursive; whatthe.) and I've wiped my entire root myself while developing an installer with it, although via a more complex bug.. NSIS simply doesn't have any built-in "you dumbass"-protection like most commercial installers.

Although I think it's nice of them to say that they're not blaming Windows for their own mistake, I do honestly think that Windows should protect such vital files at all cost - including against Administrator level process (e.g. a prompt "you dumbass - are you sure?" will do).

Re:Straightforward, sure.. but... | also, the bug

[TooMuchToDo]

what of the users who did lose valuable computer time due to this problem? The proverbial kid handing in their homework (or dissertation paper or whatever), for example. Apologizing and willing to pay for a third party tech support service (e.g. Geek Squad) is nice and all, but does that cover damages incurred? doubtful. Perhaps that EULA will finally get a test.

Almost never will damages be covered. Come to think of it, I think in this case I can say "Damages will never be covered." You have to show value and proof of destruction of that value. Your homework being destroyed? Your dissertation being destroyed? While it may have a large amount of value to you, monetarily it has very little value.

Re:Straightforward, sure.. but... | also, the bug

[zippthorne]

what of the users who did lose valuable computer time due to this problem?

That's a good point. And generates some good advice for future student/gamers: Do not install any new software of any kind a week or two before a paper is due*

*at least, not without having some kind of back-up which can be read and worked on on another computer and which you regular test.

TFM reads, as they point out, that "Delete" requires a full path to be safe or else it expects the path to be root

That sounds like the the opposite of a good way for delete to fail.

Re:Straightforward, sure.. but... | also, the bug

[Thomas M Hughes]

Almost never will damages be covered. Come to think of it, I think in this case I can say "Damages will never be covered." You have to show value and proof of destruction of that value. Your homework being destroyed? Your dissertation being destroyed? While it may have a large amount of value to you, monetarily it has very little value. Lost homework is usually only about 1-3 weeks of lost work. Often less. Dissertations are a whole different beast.

A lost dissertation has a lot more value than sentimental value. You've spent X years of your life working on it, with the clear expectation that you have a high probability of getting a PhD. Having a PhD means getting a job that pays better than the pay of a graduate student. If graduate student pay is $Y, and reasonable post-doctoral pay is $Z, and you lost X years of work due to the bug, the monetary value of a lost dissertation is X*(Z-Y). This is assuming the entire dissertation can be recreated again, which in some cases is not possible.

As a PhD student in the social sciences, my graduate student pay is ~$15k on average. Starting PhD pay is ~$40k. It takes 7 years to get a PhD on average, but for realistic sake, lets say I've been working on my dissertation for 2.5 years (ignoring the time spent on comprehensive exams and coursework). Using these values in the formula above, I'd expect to lose $62,500 dollars if I had to start my dissertation from scratch. These values go much higher in the hard sciences, with little question.

Knowing this, I've got backups of stuff, because I'm not an idiot. But were I to lose everything, it'd be really, really hard for you to claim that my loss was minimal in monetary terms, and only a large loss in personal sentimental value. If someone were to maliciously burn down my home, including a large portion of my notes and drafts for my dissertation, you better believe I'm going to claim monetary losses on those. I fail to see how it's any different if a corporation's negligence does similar damage.

Re:Straightforward, sure.. but... | also, the bug

[LarsG]

Do not install any new software of any kind a week or two before a paper is due

ITYM "Don't install an addictive game before a paper is due."

Re:Straightforward, sure.. but... | also, the bug

[DeadChobi]

Actually I've found that EVE provides such a nuanced experience that it would be totally lost on anyone not older than 16. The game isn't the GAME, it's the little political games that we play within the rules of the game. For example, having Band of Brothers, the largest alliance in the game, put in an appearance when we're fighting a smaller alliance indicates that that smaller alliance is allied with BOB.

There are more games to be played against other players than just fleet battles. You can fight alliances on the market by crashing it for their goods.

It's nice when companies are honest

[_Shad0w_]

If only more companies were so honest and straight forward when they cockup. It almost makes me feel like playing EVE again. CCP can consider themselves as being given a virtual karma bonus.

Although I can't help but wonder if the "honesty is the best policy" choice was because of their handling of the last PR cockup.

Re:okay windows guys...

[B3ryllium]

I would, but I had a power outage that terminated my 4-week uptime. :)

Worst part of it was, I was baking pecan butter tarts at the time. In an electric oven. They don't make UPSes for that.

Re:That's actually a really straightforward respon

[Ecuador]

Yeah, if only more businesses did not test products enough before deployment, or read TFM when using delete commands...

AND THEN they send GEEK SQUAD to "fix" your computer. Talk about adding insult to injury!

Alright!

[Cheezymadman]

Finally, a benefit to Vista! Vista users like myself were 100% unaffected by this. It was awesome.

I cant help but wonder...

[Nezer]

I have never been into MMOs. I just didn't get them. However, in the last week things have changed and it's due, in part, to this bug.

You see, until this bug happened EVE was totally off my RADAR screen. When I read about the bug on /. last week I went to the companies website and found myself intrigued. Further discovery that they didn't charge $50 for the box on top of the monthly fee was also appealing. Further, I see client software for Macs and Linux. Intrigued I download the Mac client and create the trial account. Two days later I'm hooked and sending them my CC #.

If it hadn't been for this bug, I probably would have never bought their product! They say that any publicity is good publicity and I think this is true. Sure the SNAFU was pretty bad yet the product was still compelling enough to buy it despite a pretty bad QA miss. This latest response from the company will only help further get their name out there and is truly an opportunity to make lemonade from lemons.

That was a very, very good analysis of the problem

[Antique Geekmeister]

Having violated /. policies and actually Read The Fine Article, it was a good analysis. I wish more people would write their bug reports this well, and explain how they're going to address the problem.

I also wonder if they wouldn't benefit from a nice virtual environment system to do QA testing of new releases with? Capturing the full graphical behavior of an OS is difficult in virtual systems, due to the overhead of the virtualization itself, but it might be a lot cheaper than keeping a dozen different hardware configurations around.

I kept reading it as CCCP

[Anonymous Coward]

Read it as CCCP and not CCP and everything becomes a whole lot clearer

Can you imagine how this happened?

[Daimanta]

Programming guy 1: So we're finally done with coding everything.
Programming guy 2: Yeah finally.
*Programming guy 2 tries to make a joke*
Pr. guy 2: Hey pr. guy 1, look at this
Pr. guy 1: lol, you appended a del boot.ini
Pr. guy 2: Well, I'm going to take a coffee break
Pr. guy 1: Yeah, me too
Pr. guy 2: Wait, lets put a sticky-note on the board that we're done
Pr. guy 1: Sure
*Pr. guy 2 puts sticky on the notice board*
*both walk off*
*manager walks in*
*manager looks at the board*
Clueless manager 1: Nice, the work is finally done.
Cl. manager 1: Ahhh, I'm on a tight schedule. Lets send this file to the head programmer so he can compile everything.
*Tries to click close*
Cl. manager 1: What, changes have been made? Whatever, save.
Cl. manager 1: Ok, open outlook. Send. Done. Wow, I know this will be a spectacular release.
*Cl. manager walks of*

Installer "technology"

[kabdib]

I'm not aware of a single installer package on Windows that isn't a useless, complicated, badly-documented, bug-ridden piece of crap. The times when I've had to use (say) InstallShield, I've seriously thought about finding a new job. This stuff sucks *and* blows. (Don't get me started on USB support and BlueTooth. Oh my God. Don't even think about reading about that stuff: Once you crack open the docs and see the wavy tentacles, the squamous mouths, the eyes, the eyes, the eyes that . . . well, you'll never be quite the same again. T-tr-trust me on th-that).

Remember the happy days of "just copy" installs, which worked great on MacOs in the 90s? Upgrade to a new system? Just copy your "apps" folder over.

The question, "What kind of installer should our OS have?" is like asking, "Should we drink the red poison or the green one?" Just asking the question seals your doom.