Voters News Service: What Went Wrong

ddtstudio writes "Baseline Magazine has a pretty good recounting of how even the national TV networks can have a computer network go wrong -- in this case the night of the last U.S. election. From the article: "VNS had been trying to rewrite and retool the system for years. This was just the most recent attempt and it failed miserably." Oracle, IBM, BEA Systems -- all crashed."

Comments

Re:What went wrong?

[cioxx]

They used Linux.

The systems in question were mainframe computers running IBM's Operating System 390.

Not that i'm a linux fanatic, just wanted you to get your facts straight.

Nearly ad free

[Anonymous Coward]

... and one page version of article.

Oh BooHoo

[billmaly]

Quote from article: "Also, the networks would be unable to give the type of detailed explanations as to why voters voted the way they did this time around. For example, according to TV network analysts working the election, the networks wouldn't be able to tell viewers why particular demographic groups voted for specific candidates nor the issues that they considered most or least important when voting. "

So, what this means is that people were able to go late to the polls, and cast a vote free from the influence of network prognostication. They were able to cast a vote that they thought was right, free from the spectre of "throwing a vote" as the election had already been "called" by *INSERT NETWORK NAME HERE*. Boo Hoo to the networks. Wow...why the hell is this a bad thing???

Up until the 1960's, most US citizens were able to vote just fine, all by themselves, without the need for knowing why *INSERT DEMOGRAPHIC HERE* people voted for *INSERT CANDIDATE NAME HERE*. Why does it need to be different today? There's already enough blather on TV, if we could eliminate it from just one night every 4 years....oh man, that'd be sweet! :)

Of course, I won't know because I'll be watching something that is entertaining, rather than a farce, on my TiVo!!! :)

Re:Oh BooHoo

[mccalli]

So, what this means is that people were able to go late to the polls, and cast a vote free from the influence of network

I'm hoping someone with more up-to-date knowledge will fill in for my sketchiness here, but...

In the UK, there are laws about broadcasting political material during (and I believe immediately preceding?) an election. Additionally, I seem to remember that you are not allowed to report on the progress of that election whilst the voting booths are still open. I'm open to correction on that last point though - I'm sure some news programmes broadcast latest exit polls during the last few General Elections. However, it's a rule I definitely recall from somewhere.

Regardless of my shaky memory, they both seem like a very good rules to me. An election's point is not to win ratings for some TV programme, and it really won't kill you to know the result a couple of hours later.

Cheers,
Ian

Re:Oh BooHoo

[kmellis]

In the UK the media are not allowed to report any exit poll information until *after* the polls have closed, precisely in order to remove any possibility of the media influencing the votes of the populace. I'm very surprised that the same isn't true in the US.

Well, we have that pesky Constutional guarantee of "freedom of the press". You can ask people how they voted (why wouldn't you be able to, and why wouldn't they be able to tell you?), and you can tell other people what you find out.

This is a good example--of which there are many, many more--of a situation where the strict and broad Constitutional prohibition makes less sense than a nuanced and particular law tailored to the situation. It would be better if exit poll results could be suppressed.

The thing that non-USAians don't quite understand about the USA and USAians is that built into the very fabric of our culture is a paranoia about abuses of power by the government. (Periodic lapses into naive trust during wartime, like now, notwithstanding.) All of the Bill of Rights are built upon the same sort of slippery-slope thinking that the gun rights folks use in talking about the Second Amendment: if you cut holes into the brick wall of blanket protections, the government is sure to come barreling through and effectively destroying the whole barrier. How libertarian-minded conservatives can tolerate Ashcroft is beyond my limited ability to comprehend human irrationality.

Anyway, I'm pretty sure that the reporting of exit poll data has been legally found to be protected speech in prior law. I could be wrong. A better answer is just to encourage a civic-minded sensibility among the news reporting agencies so that they voluntarily refuse to report exit poll data until after the polls close. Or even after all the polls close.

Re:Oh BooHoo

[tetranz]

In New Zealand we have pretty much a total political news blackout on election day until the polls close. They can talk about voter turnout estimates etc but nothing much else. Exit polling is illegal. All billboards must be done by midnight before election day. Party volunteers giving rides to elderly people etc to polling places can have coloured ribbons on their cars but no party or candidate names.

IMHO, these rules all make good sense. For one thing, we don't have billboard trash lying around the roadsides for weeks after an election.

I remember a radio interview with someone in South Korea the day before their recent election. The interviewer asked 'how was ???? doing in the polls this week?', the answer 'we don't allow polls in the week leading up to an election'.

Re:Oh BooHoo

[markhb]

First, the "cast a vote free from the influence of network prognostication" issue is a red herring in the case of the 2002 election, as that only really affects Presidential elections. US networks generally voluntarily refrain from projecting the results from any given state until all or most polls in that state have closed, so particular Congressional races would probably not be affected by early projections.

But generally, it is important to note that America essentially places no special restrictions on election reporting. With regard to Presidential elections, part of the issue is that the contiguous USA spans four time zones (UTC-5 thru UTC-9), with Alaska and Hawaii extending at least another couple of zones west. Each state (and in many cases lower levels of government) sets its own polling hours for the convenience of its citizens (usually closing c. 8 or 9 Pm local time), so the poll closings roll East to West. Since the Eastern part of the country holds enough electoral votes to elect a President, it is possible for the networks to project a winner before the most populous state (California) finishes voting. Bills to introduce a uniform poll closing time for Presidential elections have been introduced in Congress (notably following the 1980 election which was called for Reagan before 9 Pm EST (UTC -5)), but they have never passed. Florida was hit in the 2000 race by the fact that the western panhandle extends into the Central time zone, and the state had polls close an hour later in that (demographically very different from the southern part of the state) area... so the networks (which at that time were following a "most of the polls have closed" criterion) called the state while people were still voting in Pensacola.

-- Remainder of my .sig: be the majority of voters.

Too many cooks?

[jlanthripp]

Quoth the article:

All members, including 19 newspapers, shared in the management of the company and oversaw its $33 million operating budget for the current four-year election cycle.

Could the failure of VNS be the fault of having far too many PHB's droning on about mission statements and TPS reports?

sensationalism

[abhikhurana]

Well, no where in the article is it mentioned that there was a problem with IBM and Oracle. It just says that there were delays in transferring data. So I don't know why Oracle and IBM were named in the original post.

BZZZZZZ!

[M.C. Hampster]

From article:

Insufficient testing of the new Java-based WebLogic application server that replaced mainframe computers running IBM's Operating System 390.

Now, it does not mention what OS they were running WebLogic on in the article, but it was definately not OS/390.

I'll tell you what went wrong

[dubbayu_d_40]

So they're having a meeting dicussing requirements for a stable system, and an idiot say's "we must have voice recognition."

Java, Oracle, DB2, BEA - nope, those were symptoms of a deeper failure...

Re:That wan't the *last* US election

[A Big Gnu Thrush]

Yes it was.

From the article:

Back up to Election Day, Nov. 5. The balance of power in Congress was up for grabs. Yet by 10 a.m., the TV networks confirmed what they had feared for months: They couldn't derive any meaningful exit-polling data from a system they had just spent between $10 million and $15 million to overhaul.

That's 2002.

Let's get an open source solution for this

[Znonymous Coward]

Who wants to start an open source project to replace this failed service. We'll use Linux, MySQL, Perl, PHP and Apache. Any takers?

First hand account

[Anonymous Coward]

OK Folks,
I'm one of the lead programmers of one of the members

We KNEW this was going to happen a LONG time before November. At the end of the article, they talk about "set a deadline 2 months ahead of the real deadline"

Guess what? They did!!! They were supposed to be ready for the NJ Primary, which was before the summer - they missed it, BIG TIME - That's when the alarm bells went into overdrive for me

I understand (This up in the levels above me) that the steering committee didn't realize that the technical committee was saying "We've got a BIG problem"

Another warning sign was when their test data generator that they sent us in the spring generated XML that didn't match their own XSDs - and that was with all the fields declared as cdata - the field names didn't match

The first test, which was supposed to be months ahead, came weeks ahead, and even the most basic message (just a heartbeat) didn't work. That's they day I knew it was doomed for sure. Our prime efforts switched to our backup data source at that time. THAT worked fine. I had a boring election night, watching VNS crash, and laughing

Re:First hand account

[Grab]

Interesting stuff.

My problem with this article is that it's describing the scenario as a "perfect storm", ie. it only happened bcos a whole bunch of unlikely things occurred together at precisely the wrong time, and there wasn't anything ppl could do about it.

In fact, as you've shown, the project went into freefall, and no-one at any oversight level had the balls to say so. As usual, it seems they committed the standard IT sin of saying "let's put all this incompatible data together, with a new architecture, a new interface and a new team", which has a well-tested track record of producing failures.

I'm constantly amazed by failures of IT projects being categorised as "one-off" events. History has shown that the *success* of a major IT project is a one-off event, and can only be achieved by major effort and good organisation. And in general, the guys at the coal face know full well that the project is screwed, but the layers of management filter out the bad news, so it ends up that managers don't know quite how bad it is until the iceberg actually hits. Some software guru (Yourdon?) said only half-jokingly that the chance of success is in inverse proportion to the cost of the project, and above some cost (or some number of people) the project is basically doomed to fail. ;-)

Grab.

Re:First hand account

[johnalex]

OK, the gentleman in me compels me to edit this, but you've all seen the story before:

In the beginning was the Plan and then came the Assumption.

And the Assumptions were without form, and the Plan was completely without substance, and darkness was upon the faces of the workers.

And they spoke amongst themselves saying, "It is a crock of crap, and it stinketh."

And the workers went unto their supervisors and sayeth, "It is a pail of dung and none may abide the odor thereof."

And the supervisors went unto their managers and sayeth unto them, "It is a container of excrement and it is very strong, such that none may abide by it."

And the managers went unto their directors and sayeth, "It is a vessel of fertilizer, and none may abide its strength."

And the directors spoke among themselves saying to one another, "It contains that which aids plant growth, and it is very strong."

And the directors went unto their vice presidents to sayeth unto them, "It promotes growth and is very powerful."

And the vice presidents went unto the president and sayeth unto him, "This new plan will actively promote the growth and efficiency of this company, and these areas in particular."

And the president looked upon the Plan, and saw that it was good.

And the Plan became Policy.

And this is how Crap Happens.

What happened?

[z_gringo]

One stipulation: That the new system use more flexible and current programming languages--Java and the Extensible Markup Language-- rather than OS 390 to gather, compute and deliver data to the media outlets.

That sounds great. People who have no idea how to accomplish the goal telling the people tasked with doing it, how it should be done. I can't believe it failed. They should have laid out what they wanted to acheive and left the rest up to the designers on how to meet those goals...

Also, some interesting older information on the VNS can be found a the Votescam website. Although they seem to have a few extreme views, along with some wild conspiracy theories..

Okay, I'll burn karma...

[Tsar]

"Oracle, IBM, BEA Systems -- all crashed!"

Doesn't that sound like a line from a bad disaster movie?
DIE HARD IV: DIE HARDWARE.

Two words....

[primebase]

"Poor Management".

As someone who finally bailed out of an extremely poorly run company (WebMD) burdened with dumb management, it's easy to see the echoes.

The list on the last page of the article is nearly perfect, with one small addition:

6) Listen to your employees!! You hired them because you thought they were good at what they do. Why would you ignore their input into the process now?

Nothing in this article is the "fault" of the technology (Oracle, Java, IBM, Linux, or anything) itself any more than it's the fault of a head of cabbage.

It's just poor management.

OS 390

[Zayin]

One stipulation: That the new system use more flexible and current programming languages--Java and the Extensible Markup Language-- rather than OS 390 to gather, compute and deliver data to the media outlets.

Ah, yes. The programming language OS 390. Are there any O'Reilly books on that subject?

Cry me a friggin river

[whovian]

the TV networks confirmed what they had feared for months: They couldn't derive any meaningful exit-polling data from a system they had just spent between $10 million and $15 million to overhaul.

Projecting winners and losers in various races would take several hours longer than in the past.

(sarcasm)
Y'know, it is truly a sad day when you can no longer count on the media to tell you what might happen and instead have to settle for what did .
(/sarcasm)

Re:Stress testing

[Minna Kirai]

Or make sure you know what kind of better hardware you could buy, if needed.

When developing a system you should try to overload it so you can recognize what a failure state looks like. This may give your engineers valuable insight.

What is the resource that gets exhausted first? What is the system's behavior when it is completely overloaded? Does it just stop functioning, or does it lose data? Or maybe generate bad data?

These things could be nice to know, and may suggest quick improvements so that, if 6 years later the customer puts in 20 times as much usage as was originally budgeted, the failure isn't completely embarrassing.

Re:Amazing. It crashed.

[ChaosDiscord]

Now why do they use this? And why is it government funded?

The "this" in question, the Voters News Service, is funded by news networks and newspapers, not the government. They use it so that they can provide as up to the minute information on voting results as possible, and so that they can provide in depth analysis of voter behavior.

Voting in this country is a fraud. Voting machines of any kind can be rigged. They don't count the ballots at the polling place. How do I know that my ballot box is the same one that arrives at city hall.

I suggest volunteering to work at the polls, or to be a monitor of the polls. You're free to watch the entire process yourself. In hotly contested races, the various parties will send people down to monitor things themselves.

It is a fraud. I don't vote because of it. Our rights were stolen from years ago.

If you are absolutely convinced that the system is fraudulent, what are you doing about it? Might I suggest:

• Vote anyway. It doesn't take long and gives you a certain credibility. Many people will hear, "I don't vote" and hear "I'm lazy and have no right to complain." It sounds more impressive if you say "I vote and I feel my vote was illegally discarded."
• Monitor the election. Get a few friends and watch the ballot boxes from start to finish to ensure no tampering occurs.
• Run for office yourself. Then vote for yourself. They can't steal that vote from you (after all, you can find out exactly how many votes you got, and if you don't get at least one, it will look suspicious).
• Run your own polls. The various parties certainly do. If your polling results significantly diverge from the actual results, use it as evidence to...
• Demand recounts. Collect as much evidence as you can and present it to as many people as you can. Hand recounts happen every election cycle in some places.
• Demand accountable voting systems. The new touch screen systems are a sham that provides an opportunity for fraud. Voting should be done on paper in an easily human readable fashion. Locally (Madison, Wisconsin), we use a paper ballot with broken arrows next to each candidate. You just draw a line to complete the arrow of the candidate you want. It's easy for an automated tabulator to read, but if you need to recount, humans can trivially read your vote.

Your claim that our election system is rigged is extremely serious. If you seriously believe it, don't you owe it to yourself and your country to fight back?