Data Sharing, Government Style

rowama writes "The Department of Homeland Security and the Justice Department have been collaborating to develop an XML-based model for data sharing. After less than a year since the initial release, in October 2005, the National Information Exchange Model (NIEM) 1.0 Beta is out. It's big, really big. There are no less than 9 namespaces and plans for future expansion. Contact your local government contractor, with resume in hand, and you may be one of the lucky developers to implement NIEM-capable software."

Comments

Bonus advantage

[nizo]

Contact your local government contractor, with resume in hand, and you may be one of the lucky developers to implement NIEM-capable software.

As an added bonus you can add a wee bit of code to make sure your name never ends up in these databases.

Re:Bonus advantage

[Lally Singh]

There are some large ethical questions programmers have to ask themselves when taking on jobs these days. After my last DoD gig, I've really started filtering what opportunities I'll consider. Mass surveillance, for example, is something most of my contacts know I won't touch.

As for the 9 namespaces, it's actually pretty reasonable. From TFA:
xmlns:u="http://niem.gov/niem/universal/1.0"
xmlns:s="http://niem.gov/niem/structures/1.0"
xmlns:c="http://niem.gov/niem/common/1.0"
xmlns:j="http://niem.gov/niem/domains/justice/1.0"
xmlns:emer="http://niem.gov/niem/domains/emergency -management/1.0"
xmlns:im="http://niem.gov/niem/domains/immigration /1.0"
xmlns:ip="http://niem.gov/niem/domains/infrastruct ureProtection/1.0"
xmlns:int="http://niem.gov/niem/domains/intelligen ce/1.0"
xmlns:it="http://niem.gov/niem/domains/internation alTrade/1.0"

Re:Bonus advantage

[Kesch]

They need more terrorist references. I also see a distinct lack of Thinking of the Children(TM). Also, the namepsace count just isn't bloated enough. I don't believe this spec is up to government work yet.

Re:Bonus advantage

[Quiet_Desperation]

After my last DoD gig, I've really started filtering what opportunities I'll consider.

I also go looking for the projects that have the potential to kill the most people, but then again I'm an utter misanthrope. :D

Although I have to admit that cybernetic, remote controlled stealth shark thing DARPA announced a while back had my interest. No killing the enemy, but it's fricken stealth sharks, man! You know I'd fight for comm lasers to burst the data back to base.

Don't worry, I'll take those mass surveillance jobs. I'll do them really well, too. Sleep tight. :)

thank Government for databases

[yagu]

Meanwhile grandma is still taking off her shoes and getting wanded at the airport. Nice to know yet another debacle is launched. Here's hoping they're as successful as they have been with the new Air Traffic Control System.

Re:thank Government for databases

[DoubleRing]

Meanwhile grandma is still taking off her shoes and getting wanded at the airport.

As objectionable as this is, I think the bigger problem is the racial scanning that goes on at these airports. There are large groups of Middle Eastern people living in the US. Have they attempted any massive terrorist operation? To grandma, it's just an inconvenience. To these people, this is prejudice. Why do people go crazy over some dumb psp ad (which didn't even make it to the US) and skip over these issues?

Re:thank Government for databases

[Anonymous Coward]

Meanwhile grandma is still taking off her shoes and getting wanded at the airport.

The knowledge that she will get the same treatment as the rest of us is the one thing keeping grandma from demanding body cavity searches for the rest of us.

I, for one, am glad that we don't live in a world where grandma gets waved through security with a smug little smile on her face while I get directed to the body cavity search room to take it in the rear to appease grandma's paranoid fears of "all those terrorists".

Aliens?

[SatanicPuppy]

Okay...I can see the need for u:SuperType->u:ActivityType->c:ActivityType->im:Al ienEncounterType...I mean, we're bound to encounter aliens at some point, right?

But im:AlienStudentDisciplinaryActionType? Planning for Alien encounters is one thing, but planning for dealing with them in our school systems seems like bureaucratic bloat to me. I don't think the Red Staters will be down with their taxes going to teach godless little green people.

(end humor tags)

Re:Aliens?

[__aaclcg7560]

You got to account for AlienIllegal, AlienET and AlienOfTheWeirdAndPissedOffVariety to cover all your bases. Assuming that some AlienET doesn't already own all your bases.

Obvious bloat.

[SatanicPuppy]

Just glancing at it, I can see problems. XML is too often used for databases when it shouldn't be, but there are similarities, and just looking at it I can see that it violates one of the most basic database design principles: normalization

Just as an example, there are three different namespaces dedicated to the various FIPS (Federal Information Processing Standards)...To three different STANDARDS.

I'm no expert on government info, and I just looked at this thing for the first time, so maybe it's brilliant and I'm ust not seeing it, but it sure looks a lot like they've fallen victim to a database noob mistake, and created a monster tree with disproportionate crazy branches everywhere, and that is bound to cause relational problems, redundant data, and warped design challenges.

Re:Obvious bloat.

[punkinabox]

Well, they said data sharing, not data storage.

Re:Obvious bloat.

[truthsearch]

Exactly. We're talking about data interchange between systems, not single system efficiency.

Re:How big?

[Tackhead]

> > The National Information Exchange Model (NIEM) 1.0 Beta is out. It's big, really big.
>
> But that's peanuts to space.

I mean you think there's a long list of entities in the markup for your CSS/AJAX/Web2.0 project's folksonomy, but that's just peanuts to the NIEM," and so on.

After a while, the spec settles down a bit and tells you things you really want to know, like the fact that the fabulously corrupt city of Washington D.C. is now so enamored of the cumulative fiscal erosion by ten billion visiting lobbyists a year that any net imbalance between the amount you donate and the amount you receive in federal contracts whilst on the take is surgically removed from your bank account when you leave: so every time you go to K Street, it is vitally important to get a receipt... and falsify it.

Re:TSDB

[9x320]

3,673 records had been removed from the Terrorist Screening Database since its creation in June 2004 until this DOJ Inspector General report came out in May 2005. The page of the Inspector General report clarifies that when a possible misidentification of a suspect with Terrorist Screening Database records is found by the Terrorist Screening Center, the Quality Assurance team reviews the information with the agency (either the National Counterterrorism Center or a certain FBI unit) that nominated the record to be included in the database. Removal of the name from the Terrorism Screening Database is an option.

Previously, two databases were maintained, a Terrorist Threat Integration Center database that was classified, which would have information from files removed before being moved to an unclassified Terrorist Screening Database for use by law enforcement.

Local law enforcement centers, and certain international airports, would get a copy of the database, and if they saw a face and name that matched up with a file in their copy, they would call a phone number. The Terrorist Screening Center would advise them on what to do based on four handling codes, which were redacted by the FBI as sensitive information in Department of Justice Inspector General reports, but I have them right here. There was a computer malfunction that resulted in Handling Code 4's being tagged as "armed and dangerous" in the database due to an error in the programming language of a program that was supposed to automatically merge together a certain database into the larger one. I wonder if this resulted in any false arrests. The handling codes have been updated since they were first released.

Handling Code 1: WARNING - APPROACH WITH CAUTION. Arrest this individual. This individual is
associated with terrorism. Once this individual is arrested, immediately contact the Terrorist Screening
Center at (866) 872-9001 for additional information and direction. If you are a border patrol officer
immediately call the NTC [National Targeting Center]

Handling Code 2: WARNING - APPROACH WITH CAUTION. Please detain this individual for a
reasonable amount of time for questioning. This individual is of investigative interest to law enforcement
regarding association with terrorism. Immediately contact the Terrorist Screening Center at (866) 872-9001 for additional direction. (As appropriate, the TSC will facilitate an immediate response from an FBI Joint Terrorism Task Force [JTTF] or other appropriate law enforcement entity.) If you are a border patrol officer immediately call the NTC.

Handling Code 3: DO NOT ALERT THIS INDIVIDUAL TO THIS NOTICE. The person queried through
this search may be an individual identified by intelligence information as having possible ties with terrorism. Contact the Terrorism Screening Center at (866) 872-9001 for additional identifying information available to assist you in making this determination. DO NOT ARREST THIS INDIVIDUAL UNLESS THERE IS EVIDENCE OF A VIOLATION OF FEDERAL, STATE OR LOCAL STATUTES. Conduct a logical
investigation using techniques authorized in you jurisdiction and ask probing questions to determine if this individual is identical to the person of law enforcement interest. WARNING - APPROACH WITH CAUTION. If you are a border patrol officer immediately call the NTC.
Handling Code

4: DO NOT ALERT THIS INDIVIDUAL TO THIS NOTICE. The person queried through
this search may be an individual identified by intelligence information as having possible ties with terrorism. DO NOT ARREST THIS INDIVIDUAL UNLESS THERE IS EVIDENCE OF A VIOLATION OF
FEDERAL, STATE OR LOCAL STATUTES. Attempt to obtain sufficient identification information to
positively identify this individual in a manner consistent with the techniques authorized in you jurisdiction. You may be contacte

War of the Worlds

[Lord Grey]

XML is, in part, supposed to make it easier to manipulate data by providing unambiguous definitions. It clarifies the data. So we throw the U.S. Government into the mix and wind up definitions like the following (pulled at random from the 'Definition' column within the niem-1.0beta.xls spreadsheet buried in the download):

Authorized dissemination control portion mark abbreviation(s). Either (a) a single abbreviation or (b) a space-delimited list of abbreviations in the order shown in the CAPCO Register. Exception: For the REL abbreviation, omit the country code trigraph(s) and instead place the trigraph(s) in the releasableTo attribute value.

WTF? This is perhaps a use of the word "Definition" that I am not acquainted with. It reminds me strongly of trying to decode the income tax rules while filling out those yearly forms. Possibly, those that actually understand the above will believe it to be a brilliant explanation. I guess I won't be one of those "lucky" contractors looking to implement NIEM-compliant software. Unless it's a "spook->human" translator.