White House Email Follies

Presto Vivace forwards a link detailing a recent House Committee on Oversight and Government Reform hearing on the White House missing emails mess. David Gewirtz's report, carried in OutlookPower and DominoPower (in 6 parts, keep clicking), makes for scary reading. "If, in fact, the bulk of the White House email records are now stored in bundles of rotting PST files, all at or above their maximum safe load-level, that ain't good in a very big way... I object to using the inaccurate and inflated claim of excessive cost as a reason to avoid compliance with the Presidential Records Act."

Comments

What?

[palegray.net]

all at or above their maximum safe load-level What exactly is the safe load level for a PST file? If you're talking about stuff that's not reliably archived, the answer is "there isn't one." I recall reading a story a while back about a debacle wherein several thousand emails were "inadvertently" deleted... what the hell is so hard about implementing a sane backup policy? It's email, not terabytes of images or anything.

Re:What?

[cridanb]

the safe load level of a pst is around 2gb , after that it is subject to corruption
as to the amount of data , email systems are the largest systems on earth these days encompassing tens of terabytes of data in their live stores and tens of petabytes on tape

and yes they should have an archiving system not just doing tape back up tape

p.S if they used an enterprise email system like lotus domino this would not be a problem after all thats what the CIA uses

Re:What?

[vertinox]

What exactly is the safe load level for a PST file?

About 1.9 GB on an older PST file and anymore will crap out.

Outlook 2003 and greater will allow 20gb files, but they become horrendously slow after 5 to 10 gb.

And yes.. People will store gigabytes of email on an exchange server... Usually when they are emailing large videos, photoshop files, or do Desktop publishing work. Though I wonder what the Whitehouse doing to take up that much space.

Certainaly it wasn't powerpoints on intelligence reports.

Re:What?

[Anonymous Coward]

the safe load level of a pst is around 2gb , after that it is subject to corruption

Depends. Starting with outlook 2003, there is a new format of .pst file which doesn't have size limits. Prior to that, 2 gig was the limit.

and yes they should have an archiving system not just doing tape back up tape

Well, that depends if you want your email to be subpoenaed/leaked or not! G.W. Bush is on the record saying he deliberately doesn't use email for that reason.

p.S if they used an enterprise email system like lotus domino this would not be a problem after all thats what the CIA uses

There are lots of enterprise email systems that can archive email, including exchange. I think this case is more of a deliberate choice not to archive, or plausibly deny archiving.

Re:What?

[NeverVotedBush]

I think it's more likely that G.W. doesn't use e-mail because he can't find the "sendification" button.

Re:What?

[NilObject]

Certainaly it wasn't powerpoints on intelligence reports.

Based the general stream of diarrhea coming from the White House (especially this past week), I'll wager $500 that it's torture/bondage porn.

Re:What?

[aeschenkarnos]

after that it is subject to corruption

This is the Bush Administration. It started subject to corruption.

Lost

[arigram]

They just need some excuse for "losing" dangerous email messages...

PST files for archiving

[call-me-kenneth]

Really, that's an accurate write-up - click past a couple of pages to get to the technical details. It'd be hilarious if it weren't so tragic.

After all, it's not like there aren't answers to the question "how shall I archive my user's email for legal and regulatory purposes?" (Disclaimer- I work for a player in that market, but we're not on the first page of results for that search. So I don't feel too bad. Oh, wait - )

digitally signed and time stamped archive

[wkk2]

Given all the convenient archival problems, every executive branch email should be archived as a PDF and digitally signed and time stamped by a secure server with the private key in protected hardware. The archive needs to be outside of the executive branch.

Re:digitally signed and time stamped archive

[martin-boundary]

No. PDF is bad. The correct format is MIME encoded RFC2822, one file per message, tarballed. It's the best way to keep the information for future readers. Any mail admin worth his pay can convert PST/OST to that format.

You can sign the tarball if you like afterwards.

The real question

[WindBourne]

is why are the dems allowing the White house off? They should be paying to have all the PST's restored. By now somebody has told them that the white house lied about the costs of the PST files. The need to go after them for perjery as well as getting the emails.

What really bothers me is that not this white house makes nixon and reagan look like boy scouts, but that the dems PROMISED to go after them, and really has done nothing.

Re:The real question

[Stanislav_J]

What really bothers me is that not this white house makes nixon and reagan look like boy scouts, but that the dems PROMISED to go after them, and really has done nothing.

Politicians making promises and then failing to keep them? I'm shocked....SHOCKED...

Re:The real question

[Adambomb]

And the captain-obvious-esk sentiment you and the rest of us all feel is EXACTLY the problem.

Just because something has always been done a certain way doesn't mean a time won't come where its necessary to put re-election odds by the wayside and do whats right. The caveat we all despise being that such people do not seem to win elections beyond the small to mid-sized municipal level from what I've seen.

I do not know of a better system overall myself, but this is definitely one of the biggest issues with democracy. Not only can doing whats right get you on your ends without any means (like say, doing nothing) but it can also be entirely undone shortly thereafter. Of course, I do not expect this to change unless we survive the next worldwide readjustment when we either can no longer maintain the food supply thats maintaining worldwide overpopulation, blow our selves the hell up, or simply forget that water isn't just for toilets.

If the current level of strife in the world isn't enough to make people want to think for themselves to be able to navigate the sea of bullshit on all sides, i doubt anything will until we see massive imminent worldwide peril with projected massive die-offs within a generation. Then the question will be, will we survive it.

Delete the White House

[Doc Ruby]

Is anyone out there still thinking that this White House operates at all near the level of minimum performance required from people in its job?

Anyone still think all this incompetence that always protects Bush and his team is some kind of accident?

Re:Delete the White House

[dpilot]

> operates at all near the level of minimum performance required

We should be so lucky to see such a high standard.

> Anyone still think all this incompetence that always protects Bush and his team is some kind of accident?

I would rather. The alternative explanation is EVIL and probably treasonous.

Re:Delete the White House

[Master of Transhuman]

Treasonous? Ever heard of Sibel Edmonds? She's got your treason in detail. Do a Google or go here. And here's the Wikipedia entry.

Lotus Notes?

[sunderland56]

I find the fact that the US Government runs on Lotus Notes more scary than the fact that they don't have any sort of backup strategy.

I bet if you go over to the IRS, those guys have a rock-solid backup going back many years.....

Who cares?

[MillionthMonkey]

All their interesting stuff went through private mail servers at the RNC to evade responsibility for document retention under the Presidential Records Act. The RNC systematically destroys its emails and Bush has even invoked executive privilege in ordering the RNC to defy Congressional subpoenas to produce them.

Re:Who cares?

[cfulmer]

There is a completely acceptable reason for going through the RNC, and it's the same reason that the Clinton administration used similar services at the DNC: there are two competing acts. You've mentioned the Presidential Records Act, which is intended to protect the official records of the White House. There's also the Hatch Act, which (among other things) prevents government computers from being used for political activities. Emails regarding political activities went through the RNC servers (or, in the case of the Clinton Administration, the DNC servers); emails regarding activities as President, i.e. the Presidential Records, are supposed to go through the White House email system, where they are backed up and archived. So, you cannot infer an intent to violate the Presidential Records Act merely from the fact that outside services were used.

Re:Who cares?

[MillionthMonkey]

There's also the Hatch Act, which (among other things) prevents government computers from being used for political activities. Emails regarding political activities went through the RNC servers (or, in the case of the Clinton Administration, the DNC servers); emails regarding activities as President, i.e. the Presidential Records, are supposed to go through the White House email system, where they are backed up and archived. So, you cannot infer an intent to violate the Presidential Records Act merely from the fact that outside services were used.

There, I helped you out with a few BOLD tags. Your mistake is assuming that an email either falls under the scope of partisian political activity or represents communication at official levels regarding government business. They sent emails that were both.

When you're having an email conversation (for example) about which U.S. Attorneys should be fired by the president for prosecuting Republican offenses or for not going after Democrats in election years, and what the cover stories for the firings should be, you're mixing political partisan activity and official government business. Since these emails were illegal for government officials to be sending, they obviously didn't use the White House email infrastructure to send them. Even these guys weren't that stupid. They were dumb enough, though, to indicate in WH emails when they were going to continue certain conversations, regarding planned activities to be carried out in an official capacity, in nongovernmental channels (RNC, gwb43.com, Yahoo Mail) to avoid them from ever becoming public.

But the purpose of the Hatch Act (passed in 1939) isn't just to protect Outlook servers from private or partisan use- it forbids the use of any federal agencies or resources to assist in partisan activities. That would include both WH email servers and the U.S. Department of Justice.

wow

[grasshoppa]

Here I am, some lowly line level system tech for a smallish town, and I'd be handed my marching orders were I even a quarter as incompetent as the white house staff seems to be. Which leads me to suspect:

1) Either they are that incompetent, and it's just a symptom of big government not knowing it's ass from it's face
OR
2) These people are purposefully appearing this inept.

Either option isn't pleasant, and both lead to a serious problem with our government where there will likely be no repercussions from this.

But then, we all knew that already, didn't we?

Re:wow

[laird]

The problem, I believe, is that the Presidential Records Act has no enforcement provisions or penalties for non-compliance. Thus, if the White House prefers to ignore it, there's no risk in doing so. So if the value of non-compliance is higher than the value of compliance, which is the case right now, the PRA loses.

This is not simply a case of incompetent IT staff setting up a system badly. The White House had an email system that by all accounts worked very well, archiving everything properly, and it was shut down and the staff let go, and the new system was set up by someone over-ruling their own IT staff in order to make sure that it couldn't work properly. That means that someone made the decision to spend a lot of time and money to eliminate a system that worked properly, to replace it with a system that didn't, over-ruling the recommendations of their own IT staff, which can only have been done intentionally.

What would be ideal would be for the PRA to be given real teeth so that the cost of violating it becomes clearly higher than the cost of not hiding whatever it is you want hidden. Given the extremely high value of keeping embarrassing or illegal behavior secret, the penalty needs to be extremely high as well, as it is for destroying evidence. That is to say, courts should presume that the records that were destroyed were incriminating. Judges take destroying evidence of a crime quite seriously.

Re:wow

[postbigbang]

While there's no seeming penalty, civil or criminal, there is a bigger penalty: ongoing confidence in government. The hubris and arrogance has become intolerable. This is just one symptom of a government gone berserk. Vetoing a bill on torture was another. Sliming the US House of Representatives because it won't pass a bill allowing the telcos to violate the very tenets of liberty in the constitution is another. The list is long. The list is sad. These are evil days, my friends.

Solution

[Cuppa 'Joe' Black]

Someone just needs to 'leak' that one of those archived emails contains a transcript of Osama bin Laden endorsing Barack Obama for president. Then stand back because we are all going to be sprayed with WH archived emails.

I watched this on TV

[value_added]

So much for those that say watching CSPAN coverage of legislative hearings is as boring as watching paint dry.

The article, despite being spread across multiple pages, characterises the hearing fairly, so I won't bother reiterating except to say that the committee members were indeed uninformed, the witnesses were somewhere between clueless and dishonest, and the politics injected into the situation (notably from the Republicans) was so thick that I wondered whether anything could be agreed upon or any of the issues resolved. Hell, by the end of it, I doubt anyone really knew what the technical issues were, myself included.

The saving grace was watching (no one could hear what he was saying) the soft-spoken White House archivist and remembering the joke about how to tell the difference between an introverted and extroverted geek. Instead of shoes, it was microphones.

Your government in action, folks. The bad guys trying to cover up, the good guys trying to find out what's going on, and both groups taking its cues Microsoft weenies.

Big, corrupted PST files? No problem.

[Animats]

Big, corrupted PST files? No problem. Just get Stellar Phoenix PST Repair. "Stellar Phoenix can repair PST files in all scenarios including the common issues listed below ... Oversized PST files with 2Gb problem. Recovers from encrypted files. Recovers deleted e-mails." U.S. Government price $249 with CD. Immediate download available. Recommended by PC Magazine.

This little problem can be overcome. Just get some image copies of those tapes out to the Internet Archive or Wikileaks, and all the technical problems will be quickly dealt with, the data will go on line, and it will all be indexed.

Re:Mod this guy up

[Anonymous Coward]

Elect a President who is a failure (Business, Baseball, Economics, & English) and see what happens. He's the Gym Teacher president

Those who can't do Teach, those who can't teach, Teach Gym.

Re:Strangling kittens

[Entropius]

Elsewhere in the world destruction of evidence is taken as guilt. Is that not the case in the USA?

In the USA, it matters a whole lot who you're talking about whether or not XYZ counts as guilt.

Mission Critical Email - why I won't use MS Prods

[kilodelta]

I cannot stand Microsoft Exchange in any of it's versions. It is nothing but an I.T. headache of the worst kind. Try backing up the mail store, I dare you. After spending several thousand more dollars you'll be close but no cigar.

In my former place of employment we used a lot of OSS for things like web, email, database, etc. Even Samba. We had a few MS-SQL environments but I stayed as far from those as I could. For email we used Qmail with a SquirrelMail front end, and for web it was Apache/Plone and databases were MySQL.

The nice thing about Qmail is it stores email in user home folders. They're flat files that are easily replicated and backed up.

When the new administration came in the Director of Admin was paranoid about the fact that I.T. could see her email folder. So they went out and spent a shitload of money on AD, Exchange, etc.

That was a year ago. They still don't have it all running.

Re:Mod this guy up

[jlowery]

Those who can't do Teach, those who can't teach, Teach Gym.

Those who, cant Use Punctuation; post to Slashdot

Outlook storage

[DragonHawk]

What exactly is the safe load level for a PST file? There's been lots of replies to this, but I figured I'd organize a coherent and correct one.

Outlook has PST (Personal Store) and OST (Offline Store) files. PSTs are basically just local mail folder collections. OSTs are used to maintain local replicates of Exchange server mailboxes (so you can still use your email even if you're on the road). In Outlook 2003 "Cached Mode", Outlook also uses OSTs even when connected to the Exchange server, and synchronizes to the server in the background.

http://support.microsoft.com/kb/208480

PST and OST files -- I'll call them "Outlook stores" -- are both built around the same file format. There are two variants. The original format, which Microsoft sometimes called "ANSI", is limited to 2 Gi byte total size, and 64 Ki items per table. The table limit affects the number of items you can have in a folder, as well as the total number of folders you can have in a PST. (Outlook stores from Outlook 97 and earlier also had a table limit of 16 Ki items, but could be auto-upgraded in place to large tables in newer Outlook versions.)

http://support.microsoft.com/kb/197430

These store limits affected OST and PST alike, so even if you had a nice, capable Exchange server, you could still encounter problems with Outlook store limits.

http://support.microsoft.com/kb/288283

With Outlook 2003, Microsoft introduced a new Outlook store format. It's sometimes called the "Unicode" format. I'm aware of no documented limits on the file format. I'm sure there are some, but Microsoft doesn't document them. Microsoft didn't document the ANSI PST limits until long after they started causing data loss, either.

http://support.microsoft.com/kb/830336

In versions of Outlook prior to 2002, if you exceeded the store format limits, Outlook would give no immediate indication. The file would keep getting bigger, as the software didn't have checks for the limits. But it would corrupting things, too. In short, silently loosing data.

Eventually, the Outlook store would get so damaged it would stop working. Microsoft provided a utility to truncate the file to 2 GiB to make it work again, loosing more data in the process.

http://support.microsoft.com/kb/296088

In Outlook 2002, Microsoft added some code to check the limits of the store, and warn/stop if you reach them.

http://support.microsoft.com/kb/305108

In Outlook 2003, along with the Unicode format, Microsoft added a parameter at which it would consider a Unicode store "full", even though the format can keep going. The stock limit is 20 GiB; you can increase it with a registry tweak.

http://support.microsoft.com/kb/832925/

"ANSI PST" does not mean PST is a standard file format; that refers to the character sets/encodings the file uses.

Exchange Server uses an entirely different on-disk storage format, called EDB. There are technical limits, but they're insanely huge (16 TiB per store, 5 stores per database group). Exchange starts to run out of hardware resources (memory, mainly) long before you hit the file size limits. There are license-based size limits in some versions/editions of Exchange. 16 GiB in 2000 Standard, and 75 GiB in 2000 Standard SP2.