Slashdot Mirror
Computer Glitch Halts Seattle New Year's Fireworks
supersat writes "At the stroke of midnight New Year's Eve, Seattle's fireworks show ground to a halt. The source of the problem is reported to be a corrupted file that wasn't checked until the last minute. After two reboots, the fireworks had to be detonated manually. And yes ... one blog commenter, claiming to have worked on prior shows, said that the shows run on Windows."
Was actually running Linux, as detailed in this piece including screenshots[pyroblog.com]
Well, unless it was an operating system problem and not bad data or bad programming, what's the point in mentioning that other than childish bashing?
Is this some sort of newsflash? Why is Windows relevant to this story at all?
A friend of mine who was present at the event said people were booing/hissing hysterically. This was all over the local news in Seattle I am surprised it too so long to make /.
Have a squat over at the hobo house.
Shouldn't that be "the show doesn't 'run' on Windows" ?
Someone wasn't there click "Allow" when the dialog popped up asking "Are you sure you want to proceed with the fireworks extravaganza?"
Have a squat over at the hobo house.
http://www.blognotiziedigitali.blogspot.com/
Those fireworks were not vista Certified.
Gee, who can guess which version of Windows they were running?
Microsoft's Windows Home Server corrupts files?
Is this 2007th's last Microsoft caused disaster... ;-)
...or 2008th first ?
The best part is that it happened in Microsofts backyard.
--
Just trying to get my first "Funny" tag in 2008
Of course, because everybody knows Linux has always been much more reliable than anything that comes from Microsoft.
Well, you really must be dumb enough to really believe the blogentry from the person who claims to have worked there.. Unless the company involved says otherwise I won't believe anything (especially since booting windows normally takes longer than 1 minute LOL)..
Any company that puts the detonation of hundreds of pounds of explosives under the control of a windows machine is begging for a massive lawsuit.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
It's not. That snide comment was most likely made up, but that wont stop a site like this from running it anyway because they need to start the new year like they spent most of the last, bashing Microsoft.
I could have lied to these guys that it was running Vista on SP1, and I can guarantee that it would have been said by them.
talk about a 'glitch' in yOUR history, being buswhacked etc... our apologies to the nytimes.
http://www.nytimes.com/2007/12/31/opinion/31mon1.html?em&ex=1199336400&en=c4b5414371631707&ei=5087%0A
Looking at America
It was not the first time in recent years we've felt this horror, this sorrowful sense of estrangement, not nearly. This sort of lawless behavior has become standard practice since Sept. 11, 2001.
The country and much of the world was rightly and profoundly frightened by the single-minded hatred and ingenuity displayed by this new enemy. But there is no excuse for how President Bush and his advisers panicked -- how they forgot that it is their responsibility to protect American lives and American ideals, that there really is no safety for Americans or their country when those ideals are sacrificed.
Out of panic and ideology, President Bush squandered America's position of moral and political leadership, swept aside international institutions and treaties, sullied America's global image, and trampled on the constitutional pillars that have supported our democracy through the most terrifying and challenging times. These policies have fed the world's anger and alienation and have not made any of us safer.
In the years since 9/11, we have seen American soldiers abuse, sexually humiliate, torment and murder prisoners in Afghanistan and Iraq. A few have been punished, but their leaders have never been called to account. We have seen mercenaries gun down Iraqi civilians with no fear of prosecution. We have seen the president, sworn to defend the Constitution, turn his powers on his own citizens, authorizing the intelligence agencies to spy on Americans, wiretapping phones and intercepting international e-mail messages without a warrant.
We have read accounts of how the government's top lawyers huddled in secret after the attacks in New York and Washington and plotted ways to circumvent the Geneva Conventions -- and both American and international law -- to hold anyone the president chose indefinitely without charges or judicial review.
Those same lawyers then twisted other laws beyond recognition to allow Mr. Bush to turn intelligence agents into torturers, to force doctors to abdicate their professional oaths and responsibilities to prepare prisoners for abuse, and then to monitor the torment to make sure it didn't go just a bit too far and actually kill them.
The White House used the fear of terrorism and the sense of national unity to ram laws through Congress that gave law-enforcement agencies far more power than they truly needed to respond to the threat -- and at the same time fulfilled the imperial fantasies of Vice President Dick Cheney and others determined to use the tragedy of 9/11 to arrogate as much power as they could.
Hundreds of men, swept up on the battlefields of Afghanistan and Iraq, were thrown into a prison in Guantánamo Bay, Cuba, so that the White House could claim they were beyond the reach of American laws. Prisoners are held there with no hope of real justice, only the chance to face a kangaroo court where evidence and the names of their accusers are kept secret, and where they are not permitted to talk about the abuse they have suffered at the hands of American jailers.
In other foreign lands, the C.I.A. set up secret jails where "high-value detainees" were subjected to ever more barbaric acts, including simulated drowning. These crimes were videotaped, so that "experts" could watch them, and then the videotapes were destroyed, after consultation with the White House, in the hope that Americans would never know.
The C.I.A. contracted out its inhumanity to nations with no respect for life or law, sending prisoners -- some of them innocents kidnapped on street corners and in airports -- to be tortured into making false confessions, or until it was clear they had nothin
Because of their past history. It's a common legal practice to take into account former violations when accusing someone.
It happened in Cornwall, England, but this time blamed on a programming bug[cornwalltimes.co.uk]
Unless you know what the file was stored on, what interactions with the computer caused the halting of the program and on what basis they decided to continue manually, you are jumping to conclusions. One guy even claimed there was BSOD mentioned in the article (nowhere was it mentioned I can see). After years of supporting computers and servers, I can confidently tell you there is no way of knowing what caused the glitches from the article. A corrupted file on which several pieces of hardware are going to coordinate something as complicated as a fireworks display is probably not caused by the operating system, as the operating system has no reason to modify the file at all, and will only be reading it. More likely is a malfunctioning hard drive, possibly bad media that was used to transfer the file from one location to another, Or possibly a bad connection between the file storing device and the computer running the program. If you look up corrupted file you will see that every operating system known to man has to deal with that. There is no operating system that can magically correct the corrupted file and cause a fireworks display to run correctly. That is just silly talk.
"It says "BSOD" in several places in the article"
The PI article makes no mention of this.
Really though, anything automated like this that cannot be repeated should be designed to be testable as completely as possible, and should be tested several times in advance.
What makes you think they did not do this? They have been doing the same thing for years and you should expect they tested everything before hand this year too. You have to assume something changed between the last test and the actual firing.
for something as big as this I would expect no less than redundant computers. It's software for christ sakes.
The midnight timing makes this look like some kind of date roll over problem. Two identical computers would have the same problem. If it's really an OS issue, your laptop would have the same problem too.
The issue with Windoze is a lack of control. You don't know what changes when and can't ever be sure the system you qualify is the one you deploy. When things break, you never really know why and can't fix it even if you find the problem. Non free software is like that.
Friends don't help friends install M$ junk.
My best friend's sister's boyfriend's brother's girlfriend heard from this guy who knows this kid who's going with the girl who saw Ferris pass out at 31 Flavors last night. I guess it's pretty serious.
Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong fix.
I have a 70 Gb software RAID in my basement on a headless Pentium 133 or 166 powered by Slackware 11.0 serving over Samba. If a hard drive goes, I replace it and no harm is done.
And I digress, there are algorithms to correct corrupted data streams though there are limits to the level of corruption they can correct.
http://en.wikipedia.org/wiki/Forward_error_correction
If the corrupted data file was the firing sequence data then I would have to say that this type of programming is best left to proper software engineers considering the danger to human lives. And if a software engineer didn't at least run a CRC check on the input file, I would suggest job displacement or at the very least some serious retraining.
Engineering isn't for fly-by-wire hackers. Innovation is great and all but there are certain times when a profession's tried and true practices should be strictly adhered to.
An unemployed man
Pyro Spectaculars. They have been in business for 30 years, have done multiple Olympic game shows and do other high profile shows every year. You can compare that to M$'s reputation for screwing everything up.
Your ad could be here!
We waited such a long time for that real Y2K bug!
It's childish gratuitous stupid bashing. Mod parent up. I'm tired of it.
Everything you just said would've cost more money. I'm pretty sure that wasn't a priority.
Change is certain; progress is not obligatory.
Sorry. Couldn't resist.
"Win treats sysadmins better than users. Mac treats users better than sysadmins. Linux treats everyone like sysadmins."
A single, or more than one, file gets corrupted and the show doesn't go on.
Yes the underlying reasons for this happening are of interest to us, since many of us are charged with implementing or running systems, while not as showy, have to run to pay the bills.
Yes, the operating system in use is an issue, since it has part of the job of keeping files from being corrupted. It has the job of catching errors.
It was a flashy semi-failure in our collective business, we would be the stupid ones not to learn all we can about it.
I'm pretty sure that wasn't a priority.
I'm pretty sure it is now.
The higher the technology, the sharper that two-edged sword.
And yes ... one blog commenter, claiming to have worked on prior shows, said that the shows run on Windows
... this has not one goddamned thing to do with the reported problem.
And yes
I see Slashdot is going to resort to low brow sensationalism just as much in 2008 as they did in 2007.
I want a new quote. One that won't spill. One that don't cost too much. Or come in a pill.
Ballmer can help you launch chairs, not fireworks. Use FOSS for a happy new year.
And during those 30 years, they've been using Windows without problems. Your point?
There are many ways to keep the risk of corruption low. Hell, if you know the CRC of the original, valid table and have a live backup of the file available in case the primary is damaged, you can avoid problems like this. A better approach is to CRC individual records in the file: if one is found to be corrupt as the firing sequence is proceeding, restore it from a backup file and continue. If that doesn't work, skip the corrupt record and continue with show.
... sometimes it's the hardware, sometimes the software, and in an environment such as this you have to account for both.
Running a fireworks show is not a high availability application (I mean, it only has to run once for a few minutes) but it is a mission-critical app in that it damn well has to run successfully. In such an environment, you run a hot backup system with a watchdog timer and some kind of automatic switchover of the control outputs. The backup system constantly monitors the run state of the primary, so if the primary controller faults out the backup will take over seamlessly. For something as costly as a fireworks show, this would be a reasonable approach, and in fact is commonly used in industry.
Let's face it: when it comes to computers shit happens
The higher the technology, the sharper that two-edged sword.
Who in their ripe minds would put Windows in control of launching fireworks at a big event??? A Linux system would work much better :) Using a Windows machine was just asking for a "Hundreds hurt in fireworks accident, Windows to blame" headline in all major US newspapers.
Three strikes applies to sentencing. Accusations are usually made on the basis of evidence as to a party's guilt in the case at hand, or by idiots with an ax to grind.
And since there doesn't appear to be any of the former...
Because it's in good fun, just like when Microsoft showed that video with a spoof of the Matrix where the Agents have trouble with Neo/Ballmer because their copy of Linux needed the "kernel to be recompiled with a new device driver", or when all of us made fun of the Linux-sponsored car "crashing" in its first race. Ha ha Linux crashing, get it? It's not some evil propaganda trying to convince you that Windows is bad.
I hate explaining jokes but looking at the Wikipedia entry for Redmond and the looking at which city had this trouble, it's not hard to smile at that.
That's not the case in every country though - I don't believe that to be the case in the UK. A case is judged on the evidence presented at the time, relevant to the specific alleged offence.
Get your own free personal location tracker
From what little we have to go on in the article, it looks like there was a problem with syncing to time code. It's been a few years since I last did time as a tech at a major theme park, but our fireworks shows looked a little like this: There was a dedicated computer running the pyro controls, one running sound playback, one controlling the lasers, one controlling lights and fog, and one controlling some miscellany such as a 70mm film projector and the large water pumps that produced a screen the movie was rear-projected onto. All of these computers could be a little buggy given that this was in the days of windows 98 and getting device drivers to play nice was always a problem. But this stuff was worked out long before showtime. The biggest problem that could show up at the last minute would be the SMPTE time code that keeps it all synced up. One intermittent cabling problem somewhere in the system could cause a computer to get bad or no time code signal at all, causing at least that one element to not playback correctly. The best way to solve this problem was to notice it ASAP, usually in the few seconds of preroll before the show starts so that you can manually sync everything at a predetermined point. But there really is no ability to pause just one element or speed up others. Once you're off time code, you're going to have to go manual, and at least with pyro with all of the different fuse delays involved, manual just isn't going to be quite right. The only other last minute problem I could think of would be a corrupted file, or more than likely a revision that wasn't saved correctly or an outdated file being loaded automatically by the show control software and no one verifying that it was the proper version. These holiday shows are a one off, of course, so there is no dress rehearsel. You can run all the simulations you want, but you only get to fire off the pyro once.
Were they running MS Home Server? That has had some file corruption issues as of late. ;)
OS: UNKNOWN
Slashdotters are 12 years old pretending to be thirteen.
The only thing new in this world is the history that you don't know.[Harry Truman]
I've seen so many instances of BSOD's on things like gas pumps, ATM's, etc. Windows sucks. Am I using it, yes I am. I'm well familiar with its eccentricities. Would I use it for mission critical projects, hell no.
Yeah, it could have been worse, imagine if they used the algorithm from the program that determines how long a file will take to to transfer...
:)
10... 9... 80.. 6430... 6... -3..
happy new years
After years of dealing with people who deal with kernel code, I know how to spot a hardware problem when I see one. This kind of problem has a very high probability of being a hardware issue, mixed with an end user screw up. Probably because they didn't take all the precautions they should have, or possibly because it was unrealistic and too expensive to take all the precautions they would have liked to. If you think this was a kernel problem, you need to take a breath and realize other things besides kernel cause problems.
When you have a hammer, you start treating everything like it is a nail. Programmers treat everything like it is a programming issue. You don't know how many times I've replaced hardware on a server and repaired it after some hapless coder was trying to code around a hardware issue.
My server had a corrupt file last month. I replaced an old harddisk and now everythijn is ok again, but....
Yes, in my blog you can read I run Windows Etch.
I mean I THOUGHT I had installed Debian, but as I read the article it seems corrupt files are a Windows thing so I must be running Windows Etch....
Privacy is terrorism.
Silly Seattle. You don't shoot fireworks through windows. Double-glazed glass isn't cheap.
What's that on the horizon? Microsoft Fireworks 2008. oh yes.
A video of the event in H.264 video and AAC audio of if you're interested: http://aws.bluehome.net/2008.mp4 (Listen for the booing in the background.)
I don't get why windows folks are fine with files just getting "corrupted" on a daily basis. They don't swap out the hard disks though right? They just re-install, and somehow that makes sense to them. To Unix/Linux folk a corrupted file is serious, you don't just reinstall your kernel because a driver must have been corrupted for example... The Unix/Linux admin will freak out and try and make sure it doesn't happen again.
I see nothing hypocritical or crazy about wishing to hold your own government to basic ethical standards. If anything it shows a love of your country to care that the constitutional right to a fair trial is upheld.
The hypocrisy lies in building prisons and interrogation centres in foreign countries to avoid those pesky American laws that would otherwise prevent the activities that go on in them. You seem to be under the mistaken apprehension that allowing the indefinite detainment and torture of suspects for information (suspects, not even convicted of any crime) makes the country a safer place. Maybe right now you don't fit the right profiles to be whisked off to Guantanamo but that can change much more easily than you'd like to believe. It makes no difference whether the country falls to its enemies or voluntarily gives up its rights; the end result is still the same - tyranny and lack of freedom.
Doubtless there are many brave and honorable people working in the armed forces and intelligence agencies to safeguard our existence and they deserve our gratitude but it is an impossible task to keep the country 100% safe from threats without depriving people of their liberty (and even then it is unlikely to be fully effective). To remain free we must accept a certain amount of risk - I would gladly choose a small increase in the probability of dying in a terrorist attack in return for knowing that my own government cannot hold people without trial or torture them. It strikes me that to do otherwise is the cowardly option.
Pointing out that there are worse violators of human rights in the world is irrelevent - it does not make American abuses any less egregious in the same way that the existence of serial killers does not justify committing murder.
"Algebraical symbols are used when you don't know what you are talking about" - BCS
That's been fixed in Vista. Now it just starts out with an hour and counts down as it finds success. Like Scotty on the Enterprise - tell 'em it'll take an hour, so when it finishes sooner they're impressed.
``A corrupted file on which several pieces of hardware are going to coordinate something as complicated as a fireworks display is probably not caused by the operating system, as the operating system has no reason to modify the file at all, and will only be reading it.''
Yes, that's what you would expect. But if this really were a Windows glitch, it wouldn't be the first time. I remember something about a Windows file server system silently corrupting files...that was on Slashdot a few days back, IIRC.
Please correct me if I got my facts wrong.
Your point is valid: the failure was probably not directly attributable to the operating system. It may be that the automation was developed in Java/Eclipse and would have broken identically on any platform.
/. has been that Microsoft tools are explicitly designed to interfere with the freedom of its users and especially developers. This is not an especially contentions point in the debate because the Microsoft side of the argument is that this is their value-add. Deployments of Microsoft products are argued to be more consistent because the tools prohibit 'hacking'. Microsoft proponents also claim that there is less need for freedom in Microsoft tools because they tools integrate with each other seamlessly and 'just work'.
However, the point of many Microsoft 'haters' on
The reason some folks make these childish comments may be little more than "I told you so"-ism with a little confirmation bias, but the longer-winded version of these comments don't get any attention any more. The longer versions are basically re-hashes of essays by the likes of RMS, ESR, Bruce Perens, Linus Torvalds, Larry Wall and other FOSS folks.
So, in short, tools which are designed to prevent their users from modifying them attract and breed users who have no interest or experience in knowing how their tools work, how they don't work, and how to integrate them with tools which were not developed by the same company. "It only runs on Windows" is an indication to some that a tool is fragile, so whenever something breaks those people will naturally assume that thing runs on Windows even if Windows was not the cause of the breakage.
To answer your question, the point of mentioning that it runs on Windows is to re-iterate the above.
You must be new here. Nobody cares about your rational comments. The sole purpose of Slashdot is to provide a forum for snarky Microsoft bashing. Find another forum where you actually fit in.
The parent should be modded offtopic, since nowhere in the article are BSODs or anything like that mentioned, but you should be modded as flamebait. "Windoze"? "M$"? Seriously, it's 2008.
Yes, it is. The UK police tries, condemns, and executes on the spot people who look or behave in a suspicious way. This is absolutely normal, they have to start somewhere. Who are the usual suspects? In the case of computer programs misbehaving, Microsoft Windows gets the rap, if no other evidence is available. Or would you prefer the police to start investigating people at random when any crime is committed?
where did you find information on these guys have a 100% success rate until now? Had this not been blogged, who outside of a few thousand WA residents would have known what happened?
What blows me away is that with that kind of history, they didn't have a backup system so the show still went on and sync'ed to SOME music. Obviously, engineering is not something they thought was important when it comes to their computer control software and hardware.
Makes you wonder what kind of things are getting put into other automation systems? Wasn't there a story recently of a Windows computer getting hacked in a cab and the passenger was able to get admin control of the Windows PC with all its web browser, file access, and system control software available/installed? There's the CSX railroad signaling system which was taken offline by a Windows virus... How about the new voting machines? There sure are alot of hacks getting jobs putting a poorly designed OS places it should not be and using poor judgment right through to the application design and installation configurations.
please show the data stating they have not had any previous problems with their software/hardware.
LoB
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
From http://www.ibiblio.org/pub/linux/docs/HOWTO/Advocacy
Black Sky Of Death
Dear Alberto,
It doesn't matter which OS you use as your controller, you ALWAYS have a fail-safe machine, or two.
I bet you'll remember that in the future.
ALWAYS.
Yes, I WAS a show tech.
~hylas
Just a token post about the fireworks show itself and not about the operating system. There should be one, you know.
Okay, what was hilarious was not so much how everything on the Needle just stopped without warning (funny as that was) but that once it finally restarted, they finished playing Liza Minelli's "Caberet" as the fireworks shot -- and then for the rest of the pyrotechnics it was without background music. Or that's how it played out on KING-5 TV, couldn't say about in person cuz some of us geeks don't get the night off and gotta work the night shift.
Laughter is the Spackle of the Soul.
I was there down on 4th and Denny... man you shoulda heard the boos and cursing.
Man what a spoiled lot of babies....
Can you mod the Summary as troll?
Sign of things to come maybe??
I bet they spent 15 trillion dollars trying to get it right. I mean even the Canadians went off on time, how hard could it be.
The failure demonstrates why explosives, including this type, should have two sets of wires to the ignitors with two controller panels.
As to OS, when I worked on the Space Needle database, it was an Oracle DBMS on Linux boxen, which we did migrate to WinOS later. The main failure point is the constant lightning strikes that burn out the systems more frequently than other protected ones.
-- Tigger warning: This post may contain tiggers! --
That used to be true, at least for hardware-level corruption.
See: http://en.wikipedia.org/wiki/ZFS
Also, I'm betting they were using consumer-grade hardware, which is notorious for skipping the 5$ extra chip needed for ECC.
Hey, Windows hating dumbasses, KING 5 News just interviewed the director of the show, guess what? It wasn't Windows. It was a custom OS, wonder what OS they were able to customize? HMmmmmm.
On another front, most professional theater control systems for lighting and sound run on Apple systems.
So, guess this was just another slanted, knee-jerk, anti-MS rant when it was probably the poster's fanboy favorite OS that was the culprit.
It's a growing trend to synchronise major fireworks displays to music. The NYE fireworks in Sydney, Australia have had a 'soundtrack' for many years.
It never seems to work for me. The reason, is that they synchronise the mid-air explosion to the music, not the kaboom that accompanys the launch of the shell.
That is, every shell goes boom and a few seconds later when airborne, boom again. The second boom is usually a lot quieter. This is also before you add in the delay of the boom getting to you based on distance.
We are used to "booms" matching the music (think 1812 Overture). The only way it really works, is watching from a long way away, in which case the fireworks suck anyway.
Do you or your partner snore? - Visit www.snoring.com.au
...at least they have an OS driver for the fireworks.
Give the guys a break...they at least started it with the Star Wars theme. And the fireworks were almost in sync for the first few seconds! That gets them some credit...right?
You all have Oo.o and Firefox, so get World Wind.
Having had past experience with smpte re video editing etc... why on earth do those machines need CONTINOUS timing, surely all they need is a start
.old and do not overwrite the current one, but rename it to .tmp .new then rename to the .final when done, to avoid any sudden errors. More prudent hardcore coders will keep 10 historical copies just incase.
trigger and off they go using their own clocks. Its not like they need to individually slowdown/speed up because you just said that cannot be done, tho
it should have been possible with multi smpte signals from a generator with individual speed controls/pause/restart modes.
Even in more past earlier days on Amigas using 3 to do music+stuff, we only syncronized the trigger and bang it off went within 20ms of each other. Enough
for any audio/video sync to be perfect.
For the modern coders, please remember, do not trust the OS, it can go wrong, even windows doesnt keep two copies of the SYSTEM registry (idiots), so
I remind you, when ever you write to config files/data files, keep the previous copy as a
and write to a
Liberty freedom are no1, not dicks in suits.
Are you talking about Fantasmic at Disneyland? Thats what it sounds like to me at least.
There's alot of really buggy programs for Linux too. That's why there is a "force-quit" option in the Gnome panel and a kill command in the terminal. I guess with any software it's the quality of the code in the software itself that makes it reliable. The reason why you are more likely to see windows crash isn't necessarily because it is less stable than, it's because it is much more widely used.
Seattle's firework display is powered by the same computer that hosts the XBOX live service
Give them time guys, they'll fix the problems one at a time.
"We live in a global world" - Harvey Pitt, former Securities and Exchange Commission Chairman
I watched the fireworks on TV from a Seattle-area bar, and no one noticed anything was wrong. Of course, that was after 8 straight hours of drinking.