Slashdot Mirror
NSA Shopping For Data Mining Tech
prostoalex writes "The National Security Agency paid a visit to Silicon Valley venture capitalists, the New York Times learned, to talk about potentially 'interesting' technologies that the Feds would be interested in purchasing. Data mining technologies that could link arbitrary facts into logical events and find dependencies, technologies for quick voice transcription - all these technologies usually get to market faster if developed by private companies."
Why didn't they turn to the open source community? They wouldn't have to pay for it and they'd get free support, too.
The owls are not what they seem
I wonder if it hs anything to do with this. To be fair to the government, this isn't actually too bad an idea. I mean if spammers and dvertisers can gleam information to find potential targets, why can't the same technology be utilized by the defense department, who is typically an early pioneer of technology adapted for public use. Then again, a similar project 'Able Danger' identified Mohammed Atta over a dozen times.
Any fool can criticise, condemn, and complain, and most fools do. - Benjamin Franklin
It's getting time we put a stop to these people.
I'm hearing more and more about the idea of a national strike.
We technicians bitch and complain about this kind of flagrant privacy violation.
It would be much more difficult for these people, I'd think, if there were some sort of technician union that had technical rights as well as civil rights as part of its platform.
It's real simple:
1) Don't help these fucks in any way.
2) Harm them in any way you can get away with. Small needling, over and over again. Refusal to cooperate. Take their money and do nothing.
Intolerance for ambiguity is the mark of the authoritarian personality.
I can't help thinking the authorities are still way too star-struck by tech and don't value human intel enough. We have seen the shortcomings of a lack of human intel in Afganistan, Iraq, 9/11 and so on. When will improving the human intel get the focus it needs so gov'ts can make informed decisions about our security. Maybe then we can forget about Dept of Homeland Security type fiascos.
Why do you hate America so much?
arbitrary:
adjective: based on or subject to individual discretion or preference or sometimes impulse or caprice
fact
noun 1 a thing that is indisputably the case
So an arbitrary fact would be something that is indisputably the case based on individual impulse of caprice.
I write code like that after I smoke a phat dubbie but I didn't know the NSA would be interested in paying a big buck for it. I'm gonna get right onto that.:)
"Academicians are more likely to share each other's toothbrush than each other's nomenclature."
Cohen
Someone will sell 'em Oracle Reports or Cognos or some other bloated "Enterprise scale solution" and they'll cane billions of taxpayer dollars over 5-10 years with little to show for it except happy BMW dealers in the areas where the middleware agile on-demand service bus oriented architecture consultants live. Bad news if you're an American taxpayer, but not as bad as it'd be if the US Govt actually capable of developing functional large-scale systems. You don't have to look far to see that it ain't necessarily so... and that if the intelligence generated by the system can be systematically ignored, marginalised and worked-around by the executive branc anyway, what difference would it make? (hint: it makes none...)
Everything I needed to know about life, I learnt from Blake's Seven
find / -name supersecretstuff -print
I believe government exists to defend the liberty of its citizens. Got me? I'm a conservative libertarian, no caps. That means that while I more-or-less agree with the Libertarians, I don't march lockstep with anybody.
... keep it to yourself.
But I used to. I used to march lockstep with my fellow Marines, wanting only a chance to use my rifle, or its bayonet, on some terrorist bent on destroying all I hold dear.
I value my privacy, too. But there's a difference between what I do in private (or even a semi-public area like a restaurant or pseudonominous posting on a blog) and what I do in public.
If someone stands on the corner shouting "Down with America! We will blow up your orphanages, unholy capitalist swine!", I'd like to know who he is and whether he's actually in contact with anyone else. If there's no "we", then he's just a nutcase and can be told so and otherwise ignored.
But if that fellow is in contact with others doing the same thing, I'd like to know about it. I'd like the government to defend my liberty by infringing his.
Similarly, if he's smart enough just to be in contact with his terrorist buddies, I'd like the government to know it so he doesn't get a chance to blow up Disneyworld or something.
I want the government to sift through all publicly available information to find people planning or engaged in activities which would cause me or another 2,966 of my countrymen to be deprived of life, liberty, or pursuit of happiness.
So take your "strike", and your call for people to interfere with government's only legitimate role, and
Raise your children as if you were teaching them to raise your grandchildren, because you are.
This isn't like the Dept. of Motor Vehicles pouring a billion down the tubes and getting nothing in return.
The NSA is made up of very smart and capable folks. Give them a budget and incentives, and they can probably do a pretty good job of sticking their noses into the public's affairs.
Sadly for our privacy, the US has no real concept of data privacy. If you've bought something and told someone, they can tell the NSA.
So if the data is available, the NSA can just go out and but it. That's perfectly fine, but it means the NSA can easily acquire mind-bogglingly large amounts of data. Also, the phone company (AT&T) has no qualms cooperating with the govt. It isn't like Google, willing to fight it out in court. Just about nobody is -- so the NSA has an easy time, if it wants to get the goods on you.
http://www.thebricktestament.com/the_law/when_to_
all these technologies usually get to market faster if developed by private companies
But after 10 years' worth of liscensing, ultimately cost the economy more.
It means dealing with the type of people who'd sell out their country for money. They're usually not the bedrock-of-society types.
Anyone remember the outcries when the CIA was found to actually be paying those types of folks?
The question is only (!) whether such software is really able to create "real" hits. And the answer is, no it won't, simply because I know a person who knows a person who knows a person who knows Billy Clinton. Or so. And also, if you can't cover your tracks, what worth are you anyway?
So they're not for real anyway. They're trying to "look good".
And it's nothing new. Like, if you have to wait for 2 hours for US airlines to check your luggage, and if you speak up about anything like seat reservations, they'll "SSSS" intimidate-search you, you know these guys are in the wrong movie anyway. Do I feel the absolute craving to fly to the US and prostitute myself on their work market? Do others feel like it? That's what they should be really worried about: the connections that do NOT happen, the applications they do NOT get. They should be really worried sleepless about what's all NOT going on in the USA.
Otherwise, people like that Canadian muslim (whom they dragged all the way to Syria and back just for a bunch of bad press), will condensate in front of each and every investigational journalist's microphone or camera, and that'll be the end of "USA being a cool country to live in" for a broad mass of really intelligent folks for quite a while. They'll exhaust their Patriot Act and screw the country. Be it! It's theirs to screw!
If you're not in these deranged guys target line, you may at least profit from cheaper RAM, cheaper mass storage, et cetera, as a consequence of clueless data mining.
So if you are mobile and thinking, why not leave the US? Any other nation that is after intelligent people for just about any purpose (industrial, academic, developement, teaching, research) should lick their fingers AS WE SPEAK. The most important thing: Create job opportunities for the language impaired! They don't teach them much else than English over there...
Instead of approaching venture capitalists, wouldn't it have been better to do a survey of existing technology?
For example, Xanalys http://www.xanalys.com/ sells exactly this type of software. I'm familiar with Xanalys because, for some time, they were the second largest common lisp vendor -- before a buyout by their programmers. (Their most interesting software is built with common lisp).
Good grief. That's 40 years of DARPA funding majestically dismissed.
arbitrary facts into logical events and find dependencies
Doesn't this sound like a paranoid?
American companies can now stop working for repressive foreign governments.
quiquid id est, timeo puellas et oscula dantes.
You're going to incite the children with that remark.
If I'm a terrorist I am going to make my suspicious purchases with cash. I will communicate in code with written letters. I won't talk about my evil plots in chat rooms or on the phone. I will assume they are listening. This will render however many Billions of dollars useless for catching terrorists. The NSA is in love with the TV idea of sitting in a dimly lit room using the keyboard to zoom-in-and-enhance the picture of the terrorist.
It WILL be useful for catching ordinary Americans who look at a little porn, smoke a little pot, drive a little over the speed limit and disaggree with the govt.; maybe about abortion, maybe about party affiliation, etc.
nobody is going to stand on a corner and admit that fucking shit you just said.
and nobody has to help the fucking government shread the rest of the constitution.
if your a fukin libertarian then you believe in a constitutional republic.
if you believe in a constitutional republic then you should have goal #1 to get fucking rid of all those fucking electronic voting machines!
if your are serious abou hiding shit
if you want to hide your communication.
all you fucking got to do is load up a god damn pen drive
or a DVD...
burn 8GB data..
all our country's secrets.
all your evil plans.
or a nice movie..
carry it on the plane.
carry in an ipod.
your fucking argument doesn't hold water!!!!!
the only reason the patriot act spy's on americans is for political reasons.
that's the bottom line.
they're so corrupt they don't want to get caught.
that is why they ignore fisa
and now they have stirred up a fucking hornets nest with islam.
the rest of the world's patience is wearing thin.
we gotta fix this problem first.
We already know what George Bush thinks about the constitution (http://www.capitolhillblue.com/artman/publish/art icle_7779.shtml), we already know that he doesn't care if weather We The People want him to do what he is doing (http://rawstory.com/news/2005/Controversial_data_ mining_program_continues_after_0224.html) and we also know that Mr. Bush is making preparations for martial law. After he retired, Tommy Franks toured the country announcing that after the next terror attack we will have to go to a military form of government (http://www.infowars.com/print/ps/franks.htm). This was unfortunate, he said, but necessary -- the people would demand it. He insisted that he was just a concerned citizen expressing his opinion -- but it turns out that he was payed over $400,000 by the White House to express this opinion.
Please get informed about what your government is doing.
Problem is ( and we all know it ) this stuff wont just be used properly, to comabat terrorism/threats to this country. It will be expanded to monitor 'dissidents' that actually love this country, but are upset how things are going.
---- Booth was a patriot ----
Unfortunately, getting quality human intel isn't simple, and there can be problems with overvaling it, too. There are problems when you start only looking for evidence supporting what you expect to find.
Tech-based intel is too limited in coverage; humans go places machines don't. Human intel has low accuracy; machines don't lie for their own benefit (yet). You need a mix of both.
//Information does not want to be free; it wants to breed.
Try looking up Apophenia; there's a fine line between creative genius and madness. Not that the twilight between can't be entertaining.
//Information does not want to be free; it wants to breed.
It will be posted on Slashdot if just some blogger mentions it on their blog...
Ahem,
Dont mind me, but doesnt data mining require data? Dont these bad guys use snail mail, secret meetings & public payphones etc? Data mining and monitoring of electronic communications is only effective when the enemy e-mail or otherwise use electronic means of communication - the Brits have been mining all Internet data too and from the UK for a few years, but were still subject to an attack.
While voice calls are routed through an underground massive network of computers looking for key-words, these bad guys will just continue to use old and proven methods like coded letters (snail mail), carrier pigeon (not a real pigeon but some unconnected messenger), coded newspaper advertisements, and all the type of techniques the CIA used to communicate with their operatives within Russia. The enemy just revert to Old School. Its slower, but secrecy is probably far more important than speed to them.
I'm all for 'warranted' electronic eavesdropping and surveillance, but the real aim is probably to prevent the use of electronic means (e-mail, blogs etc) and push the enemy to using other means where eventually they will be caught out.
Once electronic civil liberties have eroded and the enemy have reverted to (if they ever left) 'old school' methods, it will require DHS to have road blocks, physical searches, ID, questioning... It will probably be much like living under a Taliban regime. (Except of course the women will be treated the same as men...)
I already know who's going to get the fat contracts for this. I already know. I'm so amused I could practically pee myself. I'll probably get picked up by the NSA by the end of the day over this... but I already know!
At my last performance evaluation, at a non-profit federal military contractor, my manager was attempting to explain to me why my job sucks so much and why he couldn't do anything about it. At the end, though, he said,"You see that 30 acre construction project we're building across the street? I don't know what kind of business the upper management is going to put in it, but I hear it's going to be all office space and it's going to have something to do with IT. I know you have some interest in IT so maybe if you can just wait until it's built we can get you into a position where we can make better use of your skills."
I promptly left. I wasn't sticking around for 3 years, fighting with myself not to sleep at my desk, just so that I could be a database jockey. But gosh-darn I already know... and what this gives me insight to is that the company already knows, and that means that the policians sitting on the committees who will dole out these contracts already know, because there's no way that a military contractor is going to drop down a $200 million dollar construction project unless they have a good idea of which contracts are going to fill it.
Haha! It's off to Gitmo I go...
The government itself is not stealing your liberties. Their new programs are enabling criminals who will.
"7. There are two kinds of oil business. 1) There are business people who find, pump, refine, and deliver oil. 2) There are people who manipulate the government and government purchases to make a profit."
I am familiar with most of the points you make except #7. Do you have examples?
The Bush/Cheney administration has become deeply corrupt because they think that since they have fought for and obtained power, it is theirs to weild as they see fit. This is completely wrong. They have achieved positions of awesome responsibility, not power for personal profit. And oversight is key to making sure they are acting responsibly. They have (or should not have) any right whatsoever to keep any secrets as they serve in public roles, except when it directly pertains to classified information.
But what can be done about this mess? I tried this approach, http://breakthelink.org/ but to my surprise and extreme disapointment nobody was willing to actually work the problem. I think this is because they fear to actually cross swords with our authorities. Bitching is OK, because it is useless. Actually doing something is apparently not.
And now, since Halliburton has just been awarded a $350 million contract to build detention centers in the US, perhaps those who have kept a low profile are the smart ones.
"Work will make you free!"
They could fund a prize competition such as the following:
Let anyone submit an open source program that produces, with no inputs, one of the major natural language corpora as output.
S = size of uncompressed corpus
P = size of program outputting the uncompressed corpus
R = S/P (the compression ratio).
Award monies in a manner similar to the M-Prize:
Previous record ratio: R0
New record ratio: R1=R0+X
Fund contains: $Z at noon GMT on day of new record
Winner receives: $Z * (X/(R0+X))
Compression program and decompression program are made open source.
Explanation For an idea of why the C-Prize can solve the AI problem, if it is solvable, see Matthew Mahoney's comment on it:
Matt Mahoney is the author of Text Compression as a Test for Artificial Intelligence which states: (Mahoney is also a competitor who has some winnings from The Calgary Corpus Compression ChallengeSeastead this.
Wel, you would not only want a compression method that simply compresses some text effectively, but that also would proceed to produce sensible and interesting output when the compressed file is padded.
This means that the usual methods of compression that favor blocks and limited dictionaries(or Huffman buffers or whatever) are not those to look at, because the only input files (corpus) that represent what is sensible are the complete body of accessible human works. It is interesting that this body is not sequential but that grouping of related topics of this corpus together would faciliate traditional compression.
I'm still trying to figure out what people mean by 'social skills' here.
Data Mining and Data Warehousing is increasingly being used not just at Governmental levels but even in Colleges. We're going to be investigating this as a way of reducing load on the main online transaction processing where I work. Data mining is going to be very common in the near future.
Gorkman
Eric B
ebresie@gmail.com
So FDR was a dictator when he organized D-DAY???
Was he a dictator when he had our intelligence agencies secretly break the codes of the Germans and the Japanese?
Was the secret of the spy plane which Kennedy used to find the missiles in Cuba created by Dictator Kennedy?
Your post is moderated up, so people agree with you. Extremely sad. Kennedy was a good man.
"I'd like the government to defend my liberty by infringing his."
Ignoring the other issues present, let me make a clarification:
The government isn't defending your "liberty" when they catch such a person. They're defending your safety. Important difference. Not that your safety isn't important. It is.
But these sorts of programs infringe upon the liberty of the guilty by infringing upon the liberty of everyone . That's how they work. It's like a tuna net. If you want tuna caught that's fine, but are you volunteering to be the dolphin? Because plenty of dolphins will get caught in that net if it gets built.
So don't say they're defending your liberty. They're not, their defending your safety by infringing upon your liberty. That's the way of it.
Re-ordering the corpus prior to compression is fine so long as it is reversible. That means the ordering information must cost less than the gain in compression by reordering.
A classic example is the bzip algorithm. Beautiful.
Seastead this.
You can apply that same principle to tanks or fighter planes. And it would be just as invalid. Possesion of these tools, whether they be bombs or data mining software, does not make the government a tyranny. How and WHY they're used will determine that. You can't deny the nation the use of them, nor automatically label the government despotic by the mere posession of them. National Security is a legitimate function of any government. What is both ironic, and even a touch hypocritcal, is that many of the same people here bitching about data sweeps from the government, have absolutely no probably giving out their info willy nilly across the web, from porn sites to blogs. But when the government uses the technology to try and protect citizens? Nope, can't have that...
Life is hard, and the world is cruel
"I am familiar with most of the points you make except #7. Do you have examples?"
When Cheney was Secretary of Defense, he changed the rules, allowing no-bid contracts in some cases. Now that Cheney is vice-president, his former company got a huge no-bid contract, in secret. This contract was partly to provide oilfield services. The attraction of oil and weapons for corrupters is that there is so much money involved, and many contracts.
There are too many other examples for a Slashdot comment. Note that when Secretary of "Defense", Cheney engineered one of the major problems between Osama bin Laden and the United States, that there were U.S. military weapons in Saudi Arabia.
So, Cheney helped create the problem, arranged that the problem would be very profitable for his company (no bids, just a secretly arranged contract), and then arranged for his company to get the contract. It's difficult to imagine more conflict of interest than that.
if you go to http://www.in-q-tel.org/ you will find the VENTURE CAPITAL arm of the CIA, NSA, FBI and several other TLA's (Three Letter Acronyms) Every tech you mentioned in the article is ALREADY being funded and working in the commercial world. This isn't wonderland anymore, this stuff is real and well funded. Check out http://www.callminer.com/ or http://www.agentlogic.com/ if you dont believe me.
AC, this is a common trick used by some people on Slashdot, with several steps:
1) Pretend that a Slashdot comment is an exhaustively researched 500-page book that discusses all sides of an issue.
2) Find something that isn't explained in the Slashdot comment.
3) Assume that the Slashdot comment represents all that the comment poster knows.
4) Claim that it is a tragedy that the poster of the Slashdot comment could be so stupid.
Answer to the issue you raised: The proper time to deal with Hitler was long before he was elected. Then there would have been no need for secrecy surrounding D-Day, because there would have been no D-Day.
Again, this is not the complete explanation.
Halliburton Subsidiary Gets Contract to Add Temporary Immigration Detention Centers (Free registration required.)
Quote from wikpedia: "As of late 2003 Gator was installed on an estimated 38 million PCs"
http://en.wikipedia.org/wiki/Claria_Corporation
We don't see the world as it is, we see it as we are.
-- Anais Nin
"government officials and departments wanting these tools, not a private corporation"...
Honest,no offense intended at all but.. you really think there's any really huge difference? Maybe take a peek at who gets the bulk of the "government" money again, once it is all distilled down. Then take a peek at where "retired" generals and civil "public servants" go after they leave government.
For all *practical* intents and purposes, the so called "US government" is just the stealth frontman for a few dozen large transnational corporations now. Our own president, Eisenhower, explicitly warned us this might happen in his last public speech he gave..and he wasn't joking about it.
Just about everything the "government" does now is geared towards increasing those companies profits and geopolitical/economic power. Just *look* at it with a critical eye and no preconceived notions, it becomes obvious then. We no longer have much in the way of a representative republic. That is merely an illusion they promulgate in order to maintain "law n order".
some google search terms to explore: corporatism, technofeudalism, neofeudalism
see if all of this isn't coming true right in front of our faces, the old wired "scared shitlist" story
h tml
http://www.wired.com/wired/archive/3.01/shitlist.
...if you want your job outsourced to India and your ass outsourced to Gitmo.
I'd guess it's more like this: there is military/intelligence computer technology that's decades ahead of anything in the corporate world. But feigning incompetence ("Duh, we had no idea planes might fly into buildings") is what you have to do if you are the nsa/cia/fbi. I bet every fbi guy walking into the Hoover building knows that. What, you think J Edgar really was a crossdressing homo? That's more of a symbol of what they really do: deceive. Otherwise, why does the nsa make a public announcement of going shopping unless you want to 'advertise' into public space. Sheesh.
The NSA's motivations are political.
Bright they may be, but the NSA is primarily a politically motivated org that answers to the president. It would be more appropriately know as the NSC (Non-Suborned by the Constitution).
Full faith and credit should also be given the NSA for their integral role in the creation of al Qaeda.
Carter's National Security Advisor, Zbigniew Brzezinski, should get his notice as the originator of the plan to trick the Soviet into their own Vietnam, and to use the radical Arab fundamentalists as a blade to bleed them. Reagan's NSA should get their proper attribution for expanding upon this sanguineous plan.
And who can forget the words of the ole gimper himself:
On a more contemporary note, GW Bush's NSA has been alleged to have pulled an end-around the CIA station chief in Rome, violating the logical protocols which were in place at the time, accepting the dubious Niger Yellowcake to Iraq story from the Italian Intelligence Agency, SISMI, first hand, and then sourcing it into the prewar claims.
(The Italian paper "La Repubblica", ran a good 3-part expose. There is a good English translation available: 1 - 2 - 3 - (decent mirror starts here.)
The NSA was left unscathed by the Silberman/Robb Commission, that one hit wonder recognized for their top 40 silver bullet, "Blaming it all on the CIA".
When actors, orgs and/or segments of the US government, in the dispatch of their official duties, act covertly and extra-Constitutionally, they are rogue, and a criminal enterprise. They should be identified as such, their intelligence, and their stated altruistic rationalizations notwithstanding.
Rush Limbaugh is a perfect real world example of an oxycontinmoron
What records does the Postal Service keep? What records of public telephone usage are available. What about cash withdrawals from banks, or POS cash transactions from retailers. The list could go on and on.
Then there are going to be DB's bought on the black market. The Wahington Post's blogger, William Arkin discussed one of Able Danger's mad forays into that market:
Connections will be found, but their actual credibility will remain in doubt, and there will be no such thing as an individual's privacy.
All because of a government so arrogant, self-centered and incompetent, they didn't see 911 coming.
Rush Limbaugh is a perfect real world example of an oxycontinmoron
NeocCons, who presently hold great sway in the NSA, don't trust the CIA, and have attempted to disparage them, as well as DeGoss their upper management.
The CIA doesn't trust the NSA, and views them as being political. they also see the FBI as pussies.
The FBI views the NSA and the CIA as immoral and unlawful actors, and holds them at arms length when dealing with them.
If the present NSA is shopping for their own Intel software tools, their inherent paranoia will preclude their usage of In_Q_Tel's enabled tech.
Rush Limbaugh is a perfect real world example of an oxycontinmoron
You got the NSC (and Iran/contra) mixed up with the NSA.
http://www.thebricktestament.com/the_law/when_to_
"After the election Enron was illegally fleecing the good citizens of the State of California (who voted overwhelmingly against Bush)."
Before Bush was inaugurated in January 2001:
-Nov. 2000, Ken Lay began selling his Enron stock...
-Jan. 2001, California declared a state of emergency over rolling blackouts and energy prices...
-Jan. 2001, California governor Gray Davis called Enron and other energy companies "out-of-state profiteers" during the 2000 California energy crisis...
The truth is bad enough for Bush, you don't have to embellish. The California problems began and were identified while Clinton was still President. In January 2002 the Department of Justice started investigating Enron. Enron cooked the books while Bush was in office (and got busted for it), the fleecing of Californians was mostly during Clinton.
A simple Google(California Enron) will get you lots of information, almost all of which contradicts your statement. Similarly a Google(Bush California price caps) will show you that starting before the inauguration and lasting until at least the end of May, 2001, Bush refused to impose price caps to stop the gouging.
For Bush's earlier take on the problem, take a look at this reprint of a San Francisco Chronicle article that states:Finally take a look a what CBS News had to say:
If anything, instead of embellishing (as you suggest), I was probably understating the problem.We don't see the world as it is, we see it as we are.
-- Anais Nin
You are of course, correct in my lumping the National Security Agency with members of the National Security Council. My trying to be cute, and some contemporary reading subjects are to blame.
Recently I have been reading the Walsh Iran/Contra Report, as well as some other related reading, and it seems that I confused the president's Naional security Advisor (NSA - a the time of the Reagan ongoing criminal enterprise it was Robert McFarlane) as someone who is a part of, and holds great policy control over the NSA.
It didn't help that some of the other reading I've been doing are transcripts of the NSA archive's interviews for the CNN series: The Cold War Experience. The direct Iran/Contra link being Duane Clarridge's Interview.
If you visit, don't just read Dewey's silly condensed history of the Monroe Doctrine and run. Ziggy Brzezinski's interview is quite illuminating, especially when taken in context with his Le Nouvel Observateur Interview in Paris, January 15-21, 1998, which has been translated and published by that incorrigible lefty for life muckraking journo, Alexander Cockburn. For a more contemporary flavoring of the ties that grind there are also interviews by: Condi Rice, Richard Perle, and the long-term Machinator of US policy, John Negroponte; Part One and Part Two.
Rush Limbaugh is a perfect real world example of an oxycontinmoron