Data Sharing, Government Style

rowama writes "The Department of Homeland Security and the Justice Department have been collaborating to develop an XML-based model for data sharing. After less than a year since the initial release, in October 2005, the National Information Exchange Model (NIEM) 1.0 Beta is out. It's big, really big. There are no less than 9 namespaces and plans for future expansion. Contact your local government contractor, with resume in hand, and you may be one of the lucky developers to implement NIEM-capable software."

Comments

Bonus advantage

[nizo]

Contact your local government contractor, with resume in hand, and you may be one of the lucky developers to implement NIEM-capable software.

As an added bonus you can add a wee bit of code to make sure your name never ends up in these databases.

Re:Bonus advantage

[tb3]

Hey, don't laugh. This could be bigger than Ada.

Re:Bonus advantage

[Lally Singh]

There are some large ethical questions programmers have to ask themselves when taking on jobs these days. After my last DoD gig, I've really started filtering what opportunities I'll consider. Mass surveillance, for example, is something most of my contacts know I won't touch.

As for the 9 namespaces, it's actually pretty reasonable. From TFA:
xmlns:u="http://niem.gov/niem/universal/1.0"
xmlns:s="http://niem.gov/niem/structures/1.0"
xmlns:c="http://niem.gov/niem/common/1.0"
xmlns:j="http://niem.gov/niem/domains/justice/1.0"
xmlns:emer="http://niem.gov/niem/domains/emergency -management/1.0"
xmlns:im="http://niem.gov/niem/domains/immigration /1.0"
xmlns:ip="http://niem.gov/niem/domains/infrastruct ureProtection/1.0"
xmlns:int="http://niem.gov/niem/domains/intelligen ce/1.0"
xmlns:it="http://niem.gov/niem/domains/internation alTrade/1.0"

Re:Bonus advantage

[Kesch]

They need more terrorist references. I also see a distinct lack of Thinking of the Children(TM). Also, the namepsace count just isn't bloated enough. I don't believe this spec is up to government work yet.

Re:Bonus advantage

[Quiet_Desperation]

After my last DoD gig, I've really started filtering what opportunities I'll consider.

I also go looking for the projects that have the potential to kill the most people, but then again I'm an utter misanthrope. :D

Although I have to admit that cybernetic, remote controlled stealth shark thing DARPA announced a while back had my interest. No killing the enemy, but it's fricken stealth sharks, man! You know I'd fight for comm lasers to burst the data back to base.

Don't worry, I'll take those mass surveillance jobs. I'll do them really well, too. Sleep tight. :)

Re:Bonus advantage

[Soong]

under which namespace will I find the tag?

thank Government for databases

[yagu]

Meanwhile grandma is still taking off her shoes and getting wanded at the airport. Nice to know yet another debacle is launched. Here's hoping they're as successful as they have been with the new Air Traffic Control System.

Re:thank Government for databases

[DoubleRing]

Meanwhile grandma is still taking off her shoes and getting wanded at the airport.

As objectionable as this is, I think the bigger problem is the racial scanning that goes on at these airports. There are large groups of Middle Eastern people living in the US. Have they attempted any massive terrorist operation? To grandma, it's just an inconvenience. To these people, this is prejudice. Why do people go crazy over some dumb psp ad (which didn't even make it to the US) and skip over these issues?

Re:thank Government for databases

[Anonymous Coward]

Meanwhile grandma is still taking off her shoes and getting wanded at the airport.

The knowledge that she will get the same treatment as the rest of us is the one thing keeping grandma from demanding body cavity searches for the rest of us.

I, for one, am glad that we don't live in a world where grandma gets waved through security with a smug little smile on her face while I get directed to the body cavity search room to take it in the rear to appease grandma's paranoid fears of "all those terrorists".

Re:thank Government for databases

[jrumney]

which didn't even make it to the US

Yeah, well neither do the people who've been racially profiled onto the no-fly-list once they've left. (registration free link)

Re:thank Government for databases

[Anonymous Coward]

As objectionable as this is, I think the bigger problem is the racial scanning that goes on at these airports.

No, the bigger problem is making government agents into robots. They only follow procedures and aren't allowed to think for themselves (or heaven forbid, take initiative), for fear that someone could say that they were performing racial profiling. We are more afraid of the political repercussions of a few racial discrimination cases than the repercussions of planes being bombed or flown into major landmarks.

Aliens?

[SatanicPuppy]

Okay...I can see the need for u:SuperType->u:ActivityType->c:ActivityType->im:Al ienEncounterType...I mean, we're bound to encounter aliens at some point, right?

But im:AlienStudentDisciplinaryActionType? Planning for Alien encounters is one thing, but planning for dealing with them in our school systems seems like bureaucratic bloat to me. I don't think the Red Staters will be down with their taxes going to teach godless little green people.

(end humor tags)

Re:Aliens?

[__aaclcg7560]

You got to account for AlienIllegal, AlienET and AlienOfTheWeirdAndPissedOffVariety to cover all your bases. Assuming that some AlienET doesn't already own all your bases.

TSDB

[9x320]

Neato. Maybe now they'll make less errors in that Terrorist Screening Database they have. You know, the one that has the names of over 250,000 people tagged as terrorists used in everything from no-fly lists to border crossings ever since the administration wanted all such watchlists to be consolidated into a single big one. That one the NSA probably uses. That one that, according to Department of Justice Inspector General reports, may be riddled with errors.

Read the Department of Justice and Department of Homeland Security Inspector General reports. They redact sensitive information in some cases, but based on context you can identify information in some places they've failed to redact in others. Keep on reading and you'll remember things to fit together a bigger picture.

Re:TSDB

[9x320]

3,673 records had been removed from the Terrorist Screening Database since its creation in June 2004 until this DOJ Inspector General report came out in May 2005. The page of the Inspector General report clarifies that when a possible misidentification of a suspect with Terrorist Screening Database records is found by the Terrorist Screening Center, the Quality Assurance team reviews the information with the agency (either the National Counterterrorism Center or a certain FBI unit) that nominated the record to be included in the database. Removal of the name from the Terrorism Screening Database is an option.

Previously, two databases were maintained, a Terrorist Threat Integration Center database that was classified, which would have information from files removed before being moved to an unclassified Terrorist Screening Database for use by law enforcement.

Local law enforcement centers, and certain international airports, would get a copy of the database, and if they saw a face and name that matched up with a file in their copy, they would call a phone number. The Terrorist Screening Center would advise them on what to do based on four handling codes, which were redacted by the FBI as sensitive information in Department of Justice Inspector General reports, but I have them right here. There was a computer malfunction that resulted in Handling Code 4's being tagged as "armed and dangerous" in the database due to an error in the programming language of a program that was supposed to automatically merge together a certain database into the larger one. I wonder if this resulted in any false arrests. The handling codes have been updated since they were first released.

Handling Code 1: WARNING - APPROACH WITH CAUTION. Arrest this individual. This individual is
associated with terrorism. Once this individual is arrested, immediately contact the Terrorist Screening
Center at (866) 872-9001 for additional information and direction. If you are a border patrol officer
immediately call the NTC [National Targeting Center]

Handling Code 2: WARNING - APPROACH WITH CAUTION. Please detain this individual for a
reasonable amount of time for questioning. This individual is of investigative interest to law enforcement
regarding association with terrorism. Immediately contact the Terrorist Screening Center at (866) 872-9001 for additional direction. (As appropriate, the TSC will facilitate an immediate response from an FBI Joint Terrorism Task Force [JTTF] or other appropriate law enforcement entity.) If you are a border patrol officer immediately call the NTC.

Handling Code 3: DO NOT ALERT THIS INDIVIDUAL TO THIS NOTICE. The person queried through
this search may be an individual identified by intelligence information as having possible ties with terrorism. Contact the Terrorism Screening Center at (866) 872-9001 for additional identifying information available to assist you in making this determination. DO NOT ARREST THIS INDIVIDUAL UNLESS THERE IS EVIDENCE OF A VIOLATION OF FEDERAL, STATE OR LOCAL STATUTES. Conduct a logical
investigation using techniques authorized in you jurisdiction and ask probing questions to determine if this individual is identical to the person of law enforcement interest. WARNING - APPROACH WITH CAUTION. If you are a border patrol officer immediately call the NTC.
Handling Code

4: DO NOT ALERT THIS INDIVIDUAL TO THIS NOTICE. The person queried through
this search may be an individual identified by intelligence information as having possible ties with terrorism. DO NOT ARREST THIS INDIVIDUAL UNLESS THERE IS EVIDENCE OF A VIOLATION OF
FEDERAL, STATE OR LOCAL STATUTES. Attempt to obtain sufficient identification information to
positively identify this individual in a manner consistent with the techniques authorized in you jurisdiction. You may be contacte

Obvious bloat.

[SatanicPuppy]

Just glancing at it, I can see problems. XML is too often used for databases when it shouldn't be, but there are similarities, and just looking at it I can see that it violates one of the most basic database design principles: normalization

Just as an example, there are three different namespaces dedicated to the various FIPS (Federal Information Processing Standards)...To three different STANDARDS.

I'm no expert on government info, and I just looked at this thing for the first time, so maybe it's brilliant and I'm ust not seeing it, but it sure looks a lot like they've fallen victim to a database noob mistake, and created a monster tree with disproportionate crazy branches everywhere, and that is bound to cause relational problems, redundant data, and warped design challenges.

Re:Obvious bloat.

[punkinabox]

Well, they said data sharing, not data storage.

Re:Obvious bloat.

[kfg]

that is bound to cause relational problems, redundant data, and warped design challenges.

Do me a favor, don't tell them.

KFG

Re:Obvious bloat.

[truthsearch]

Exactly. We're talking about data interchange between systems, not single system efficiency.

Re:How big?

[Tackhead]

> > The National Information Exchange Model (NIEM) 1.0 Beta is out. It's big, really big.
>
> But that's peanuts to space.

I mean you think there's a long list of entities in the markup for your CSS/AJAX/Web2.0 project's folksonomy, but that's just peanuts to the NIEM," and so on.

After a while, the spec settles down a bit and tells you things you really want to know, like the fact that the fabulously corrupt city of Washington D.C. is now so enamored of the cumulative fiscal erosion by ten billion visiting lobbyists a year that any net imbalance between the amount you donate and the amount you receive in federal contracts whilst on the take is surgically removed from your bank account when you leave: so every time you go to K Street, it is vitally important to get a receipt... and falsify it.

War of the Worlds

[Lord Grey]

XML is, in part, supposed to make it easier to manipulate data by providing unambiguous definitions. It clarifies the data. So we throw the U.S. Government into the mix and wind up definitions like the following (pulled at random from the 'Definition' column within the niem-1.0beta.xls spreadsheet buried in the download):

Authorized dissemination control portion mark abbreviation(s). Either (a) a single abbreviation or (b) a space-delimited list of abbreviations in the order shown in the CAPCO Register. Exception: For the REL abbreviation, omit the country code trigraph(s) and instead place the trigraph(s) in the releasableTo attribute value.

WTF? This is perhaps a use of the word "Definition" that I am not acquainted with. It reminds me strongly of trying to decode the income tax rules while filling out those yearly forms. Possibly, those that actually understand the above will believe it to be a brilliant explanation. I guess I won't be one of those "lucky" contractors looking to implement NIEM-compliant software. Unless it's a "spook->human" translator.

Re:War of the Worlds

[EQ]

Actually that statment is pretty clear to someone with domain knowledge. Like any other knowlege domain, its probably very abstruse to the outside. Remember, Feynman was not famous for only being a physicist, but for being a physicist that could make himself understood to those outside of his domain of expertise (c.f. Feynman's lectures).

Its actually a very concise and clear explanation of that part of the data plan. The problem for you is that you do not have the context, nor subject matter expertise, so it appears to make no sense to you. I, on the other hand, have handled and created classified compartmented documents "back in the day", so its meaning is perfectly clear to me. Its also quite obvious this is from a section about how to carry across message-handling markings ("Classification" and "Dissementation" restrictions & caveats) from one agency to another, or even intra-agency stuff. This indicates to me that you probably pulled it from the Intelligence part of the namespace.

Bascially, the part you quoted says, in more coloquial English:

To control who gets to see this portion of data, the document is marked over-all AND portions are marked individually. To properly mark a portion of a document, (usually a paragraph), ther may be some paragraphs in a document that are "secret", some may be "unclassified", some may be US-only, some may be releasable to NATO, or various and sundry combinations of these types of things. To designate these "portion classifications, caveats and dissementation controls" and properly "mark" this portion fo the document, there is either (a) a single abreviated term, or else (b) a list of abbrevaited terms delimited by spaces. These terms can be found in a document called the "CAPCO Reigster". The only exception to this rule is the "REL" term, which means "Releasable To". Therefore, the values normally found after the REL term in a portion of a document should be put into the "releasableTo" attribute of this portion of a document, instead of the normal dissemenation control data block part of the document.

Thats a lot of context that isnt needed by someone reading a spec, governmentor otherwise. The spec assumes a given level of subject matter and domain expertise. To dumb it down would be wrong - that is the best way to lard up and bloat a spec, or else allow a spec so loose as to be useless in constraining the data properly. And, as you mention, "XML is upposed to make it easier to manipulate data by providing unambiguous definitions". The quoted text in your post is an example of a *very* _un_ambiguous definition of a data field. And contrary to your belief, its not just goverment that created such hard-to-scan (for outsiders) documents/specs, I've seen banks, health companies, telecom companies, aerospace [and other places that cannot afford a "loose" data type] write very similar specifications that contain similar definitions.

You'll see much of the same once you get out into the world.

HTH.

Re:bah

[deathy_epl ccs]

On some level, information has always been exchanged between these powers. Now they're using XML. Cool.

Yes, but see... with the advent of XML, that information exchange is now more than just "Uh-uh, not gonna tell ya!"

Now, they have a name-space that includes the ability to tack a "NYAH NYAH" on to the end of the statement.

58 pages of spec; 3 instances of "security"...

[kneecapd]

I skimmed through the 58 page spec document which was mostly filled with describing the vast levels of bureaucracy that they're putting place to manage this beast. I also did a simple word find on the word "security". I only found 3 instances of the word that weren't coupled with the word "homeland" as in the Department of. No instances of the word "authentication".

I know this is doc isn't intended to show the exact structure of the messages to be passed, but gee whiz, wouldn't you think they would address the topic of "how do we make sure that only members with access to the NIEM can retrieve/exchange this data."????

All I found was a quick reference to one of the committees that they're going to form - who has the responsibility of (paraphrasing) "helping member organizations handle data security".

That's kinda scary. Or does this thing just run on the super-duper-secret world-wide government inter-network? I mean, they never have any problems with data security on that thing. (see: Los Alamos Lab, Dept. of Veteran Affairs, etc.)