I work as a data-mining professional and aside from creating statistical models on flat-files, I manage the process of transforming and joining relational databases into a a flat file for model building.
Currently we use Oracle for this work, but in the past we tried switching to MySQL but found that it lacked some of the key features such as materialized views, nested sub-queries and a variety of Oracle SQL functions that we find useful. MySQL seemed to be geared towards maintaining a real-time database to support customer interaction, rather than as an environment for assembling static data sources.
Could Firebird be a viable open-source alternative, or are there others?
The scenarios you list may aid in tracking down a known terrorist, but without prior knowledge that can be effectively inserted into the data mining algorithm neither scenario will discover novel information about unkown terrorist rings.
Consider that there are approximately 300 million people in the united states. At 10 phone calls a day for 365 days you get 1 trillion phone calls per year. Suppose I have 100 known terrorists telephone activity for 6 months, and I want to find similar patterns in the remainder of my data to identify other terrorist rings. That means I have 100*10*365/2 = 182,500 training examples. I.e., 200k / 1,000,000,000k ~= two millionths of a percent of my data for training a predictive model is labeled as positive for terrorist activity. Even with an amazingly accurate algorithm, this will lead to hundreds of thousands of false positives and a few true positives, for no net gain of actionable information. Certainly you can narrow these results by orders of magnitude through very intensive effort, but the margin will not be overcome.
To track down known terrorists can't the data be requested on an as-needed basis through the courts?
As a data-mining professional I find myself using the term data-mining less frequently in my interactions with clients and colleagues. That is because data-mining is going the way of artificial intelligence: over hyped and under delivered. The ARDA Novel Intelligence from Massive Data web-site summarizes the principle failures of data-mining.
"The techniques fail to acquire or to use the prior knowledge - the "thread of logic" - that analysts bring to their tasks. As a result, discoveries made by machines prove to be trivial, well-known, irrelevant, implausible, or logically inexplicable"
95% of what is "discovered" in data-mining falls into one of the above categories. The value is provided by leveraging the data to quantify the "well-known" effects, and is obtained by using modern applied statistics to tackle specific problems such as:
Use these 100,000 measurements of 10 known varibles and outcomes to build a model to predict unkown outcomes for new variables.
DARPA and ARDA's goal of predicting terrorist behavior, or
"spotting the telltale signs of strategic surprise in massive data sources"
will fail due to a paucity of observed terrorist behavior, an inability to precisely define the objective and an enormous amount of poorly collected, noisy and irrelevant data.
About a year ago I purchased a wireless mouse, and I eventually threw it out. Not only was the mouse substantially heavier but constantly replacing the batteries was expensive and annoying. I think wireliess mice would be much more practical if the mouse-pad doubled as a charging pad.
With narrow bandwidth, limited geographic reach and poor sound quality, why haven't the alternatives to FM radio caught on? There is satellite radio, cable radio, internet radio, yet all combined the size of their audiences pale in comparison to those of good old FM, a technology that hasn't changed for decades.
While advancing leaps and bounds in personal mp3 players, are we skipping the next generation of broadcasted music?
Re:Optical gets bypassed by other denser tech?
on
Storing Light In Chips
·
· Score: 4, Interesting
I wonder if optical will simply be bypassed by other, already denser technologies.
There are two primary restrictions on current micro-processors. One is our ability to manufacture large deformity free wafers of silicon. The other is the excessive heat generated by the electricity. Both have been slated to slow our progress along Moores Law using conventional micro-processor technology.
What are the alternatives? It is possible to build deformity free cubes of silicon. However, in a 3-dimensional chip the heat generated (grows with the cube of the height of the chip) is dissapated through surface area (grows with the square of the height of the chip) so it compounds the second problem.
A probable alternative is the substitution of man-made diamond wafers for silicon. Diamond is far more heat-resistant than silicon, and can be created deformity free by plasma layering processes. Unfortunately the technology is still nacent and wafer sizes are still miniscule.
Optical computation would clearly provide a heat advantage. Imagine the newest supercomputer powered by a flashlight. But regardless, the greatest advantage of this technology, if realized and implemented for even a small set of basic algorithms, will be quantum computers.
Many people who have heard the phrases "Business Intelligence" and "Executive Dashboard" do not understand what they mean in practice, nor how they are useful.
As a data mining consultant in the Insurance industry who has participated in Sales meetings with Business Intelligence vendors such as Insight, I have come to understand some of the limitations of these products.
First, an executive dashboard is a pivot table dressed up with web based features. It is a flexible interface that sits upon "data cubes" that are generated by your analysts and IT departments. You can view your inventory distribution broken down by any other variable you want, or you can see a map of the world color-coded to your recent sales successes. What is new about these products is that they automate the generation of these graphical data representations, and give you a lot of power and flexibility with little effort.
The principle disadvantages to these products are that:
Executives often don't know or have the time to learn how to maximize their use of the tools
The aggregated results that you see on a dashboard are only as good as the data going in
They focus on lagging indicators, such as what your sales activity was yesterday, rather than what's going on today and what is likely to happen tomorrow, you can end up driving while looking out the rear window
They do not allow the executive to tweak constants, such as percent of inventory in San Francisco, and automatically regenerate their data - you have to cycle back through the analysts, you often have to re-work everything if you want to turn the steering wheel or step on the gas pedal
We are working with these vendors to try to address many of these issues, and incorporate modern statistical pattern recognition algorithms such as Neural Networks and Support Vector Machines, so that the Executive can look out their front window, steer, and out-race the competition.
The ideal source for random numbers has always been physical sources, such as the white noise you see on your television screen when tuned to an unused channel. The noise is generated by remnants from the big bang, and is cryptographically unusable (since the numbers are recordable by anyone). But is a good test for statistical algorithms such as evolutionary computation (which depend on randomn initial states).
The idea of using electrical currents secured on a chip is much sounder - since the noise is locally generated and very difficult to tap. I project that as quantum mechanics become more mainstream, the random quantum effects of electrons will be tapped to generate even sounder and accessible random signals.
The man then can urinate, with the liquid conducted down the tube, and from there into another tube or container of your choice.
Solves a lot of long term bondage problems!
The promise of quantum computers is doing computations (as Damian says)
"in multiple universes, in constant time" and Q::S obviously can't do this. It can and does, however, act like you're programming a quantum computer by allowing you to give one scalar multiple simultaneous values.
To expound upon Damian's phrase, consider that a traditional computer will take a binary string of length N such as
B = {1,0,0,1,1,0,1,1,1,0,...} (N bits)
and compute a function of B
F(B) = {0,1,1,0,1,0,...} (M bits)
Where for example B could be an image and F(B) is it's compressed.jpg form (compressed if M is less than N). The power of a quantum computer is that it can compute F(B) for all possibleB's in one computational pass!
While this not terribly useful for our compression problem (who would want to compress all possible pictures simultaneously?) it is immensely useful for other tasks, such as testing the primality of a number. If F(B) is 1 when B represents a prime number, and F(B) is 0 when B represents a composite number, then a single quantum run of F would test the primality of all numbers from 0 to 2^N simultaneously!!
Blank paper - 40% of all kiddie porn drawings are drawn with crayon on blank paper!
Fresh air - 30% of all traffic violations are commited while breathing fresh air! (the remainder occur in LA)
Unused sperm - 99% of all unborn children die in plastic sheaths and on bathroom walls!
Blank CD sales??? WTF would you do that?
As a native Midwesterner (born and raised in Kansas) who has migrated to the City (Manhattan to be specific) and who misses the countryside, I must protest.
To proceed somewhat logically:
1. You are not traveling for the purpose of arriving at your destination, for clearly
airfare is cheaper, therefore you must be traveling for some alternative reason.
2. The Midwest, by definition as only partway to somewhere else, is an intermediary place. You don't travel to the Midwest, you travel through it. Therefore stopping is tantamount to etymological excoriation.
Since you are not traveling to arrive at a destination, you must be traveling to either:
1. Enjoy the ubiquitous spaciousness of the scenery, or
2. Hyper-inflate your ego
The very act of bringing your computer along with you obfuscates the beauty of the Midwest. Attempting to remain connected to the internet and actually blogging your journey, undeniably proves the latter reason.
I can therefore conclude, as my final non sequitur, that you are a Mac user, since they obviously enjoy appending useless hype to revered foundations.
Industry leading Games Development Company seeking talented software developers to work for both stock options and salary!!
Excellent opportunity for pseudo-elite narcissistic code-monkeys with mediocre GPA's, 2 years Everquest experience and a predilection for sleazy pornography.
Starting salary $75k-$100k ***
*** In order to be eligible for this introductory salary offer, please sign and overnight the available forms at reputable recruiting services.
The basic idea is to coerce a ton of DNA into producing random potential solutions to the problem, and to then use chemical processes to select "good" solutions in mass. Since the space of possible solutions to Traveling Salesman problems of any reasonable size is tremendous (larger than the national debt expressed in pesos) DNA computing has an edge over traditional methods, because solutions are easy to generate and then weed out.
Unfortunately, this is really just a gigantic parallel processor - with each strand of DNA the memory of a processor induced by the chemical manipulations, and a small subset of useful algorithms are parallelizable (can be broken up into small "chunks" that can be computed independently and tied back for a larger result.
The immense benefit that this technology will have will be in fields like evolutionary computation. Evolutionary computation relies upon generating large populations of solutions, and then applying simple rules (which could be chemically encoded) to "improve" the generation, towards the pursuit of some ultimate goal. This could be training a neural network to predict coronary artery disease, or optimizing the design of a jet engine without tackling fluid dynamics - truly wondrous!
What about a watch that periodically samples biometric data from the skin beneath it?
There are wristwatches that sample glucose for monitoring diabetic's blood-sugar level, such as the GlucoWatch.
There are also devices for processing fluids in a microsystem, such as the MEMS's Biochip.
In the near future the wristwatch could eliminate the need for visible user-token authentication, or at least reduce its frequency. This would greatly increasing both the security of the system and its ease of use.
I called the Markham Police department's phone number, and got a Sargent Knapp's voice mail, so that checks out. However, it doesn't mean the author didn't just look this guy up to add more credibility to the story.
This could all be a hoax, meanwhile people are reading the article and sending money to his girlfriend's Paypal account. Some of the things that struck me:
We are set up to empathize with him:
I almost always buy used, so don't get any ideas about me being rich.
In case you didn't feel bad enough yet:
I was already beyond broke, but I figured $85.00 more wouldn't kill me.
Are you heartless???:
all of your student loans for the next semester are going to cover this counterfeit check.
He'd give to charity if he had money:
I urge you to choose a local charity
Make the check payable to:
If you really must, you can send money to my girlfriend's Paypal account, cranberry_coyote@hotmail.com
Slashdot Mirror
Typing SHIELDEDPROTONMAGNETICMOMENTTONUCLEARMAGNETONRATIO with the SHIfT key twice an hour adds insult to injury (coding in Fortran).
I have limited experience mining text/unstructured data. In the past I have used word-frequency counts to mine semi-structured text.
I work as a data-mining professional and aside from creating statistical models on flat-files, I manage the process of transforming and joining relational databases into a a flat file for model building.
Currently we use Oracle for this work, but in the past we tried switching to MySQL but found that it lacked some of the key features such as materialized views, nested sub-queries and a variety of Oracle SQL functions that we find useful. MySQL seemed to be geared towards maintaining a real-time database to support customer interaction, rather than as an environment for assembling static data sources.
Could Firebird be a viable open-source alternative, or are there others?
The scenarios you list may aid in tracking down a known terrorist, but without prior knowledge that can be effectively inserted into the data mining algorithm neither scenario will discover novel information about unkown terrorist rings.
Consider that there are approximately 300 million people in the united states. At 10 phone calls a day for 365 days you get 1 trillion phone calls per year. Suppose I have 100 known terrorists telephone activity for 6 months, and I want to find similar patterns in the remainder of my data to identify other terrorist rings. That means I have 100*10*365/2 = 182,500 training examples. I.e., 200k / 1,000,000,000k ~= two millionths of a percent of my data for training a predictive model is labeled as positive for terrorist activity. Even with an amazingly accurate algorithm, this will lead to hundreds of thousands of false positives and a few true positives, for no net gain of actionable information. Certainly you can narrow these results by orders of magnitude through very intensive effort, but the margin will not be overcome.
To track down known terrorists can't the data be requested on an as-needed basis through the courts?
Use these 100,000 measurements of 10 known varibles and outcomes to build a model to predict unkown outcomes for new variables.
DARPA and ARDA's goal of predicting terrorist behavior, or will fail due to a paucity of observed terrorist behavior, an inability to precisely define the objective and an enormous amount of poorly collected, noisy and irrelevant data.
By the time this ships maybe the mouse pad will be a power pad.
About a year ago I purchased a wireless mouse, and I eventually threw it out. Not only was the mouse substantially heavier but constantly replacing the batteries was expensive and annoying. I think wireliess mice would be much more practical if the mouse-pad doubled as a charging pad.
With narrow bandwidth, limited geographic reach and poor sound quality, why haven't the alternatives to FM radio caught on? There is satellite radio, cable radio, internet radio, yet all combined the size of their audiences pale in comparison to those of good old FM, a technology that hasn't changed for decades.
While advancing leaps and bounds in personal mp3 players, are we skipping the next generation of broadcasted music?
What are the alternatives? It is possible to build deformity free cubes of silicon. However, in a 3-dimensional chip the heat generated (grows with the cube of the height of the chip) is dissapated through surface area (grows with the square of the height of the chip) so it compounds the second problem.
A probable alternative is the substitution of man-made diamond wafers for silicon. Diamond is far more heat-resistant than silicon, and can be created deformity free by plasma layering processes. Unfortunately the technology is still nacent and wafer sizes are still miniscule.
Optical computation would clearly provide a heat advantage. Imagine the newest supercomputer powered by a flashlight. But regardless, the greatest advantage of this technology, if realized and implemented for even a small set of basic algorithms, will be quantum computers.
As a data mining consultant in the Insurance industry who has participated in Sales meetings with Business Intelligence vendors such as Insight, I have come to understand some of the limitations of these products.
First, an executive dashboard is a pivot table dressed up with web based features. It is a flexible interface that sits upon "data cubes" that are generated by your analysts and IT departments. You can view your inventory distribution broken down by any other variable you want, or you can see a map of the world color-coded to your recent sales successes. What is new about these products is that they automate the generation of these graphical data representations, and give you a lot of power and flexibility with little effort.
The principle disadvantages to these products are that:
Executives often don't know or have the time to learn how to maximize their use of the tools
The aggregated results that you see on a dashboard are only as good as the data going in
They focus on lagging indicators, such as what your sales activity was yesterday, rather than what's going on today and what is likely to happen tomorrow, you can end up driving while looking out the rear window
They do not allow the executive to tweak constants, such as percent of inventory in San Francisco, and automatically regenerate their data - you have to cycle back through the analysts, you often have to re-work everything if you want to turn the steering wheel or step on the gas pedal
We are working with these vendors to try to address many of these issues, and incorporate modern statistical pattern recognition algorithms such as Neural Networks and Support Vector Machines, so that the Executive can look out their front window, steer, and out-race the competition.
The ideal source for random numbers has always been physical sources, such as the white noise you see on your television screen when tuned to an unused channel. The noise is generated by remnants from the big bang, and is cryptographically unusable (since the numbers are recordable by anyone). But is a good test for statistical algorithms such as evolutionary computation (which depend on randomn initial states).
The idea of using electrical currents secured on a chip is much sounder - since the noise is locally generated and very difficult to tap. I project that as quantum mechanics become more mainstream, the random quantum effects of electrons will be tapped to generate even sounder and accessible random signals.
And if going to the bathroom becomes too inconvenient, try some of these.
B = {1,0,0,1,1,0,1,1,1,0,...} (N bits)
and compute a function of B
F(B) = {0,1,1,0,1,0,...} (M bits)
Where for example B could be an image and F(B) is it's compressed
While this not terribly useful for our compression problem (who would want to compress all possible pictures simultaneously?) it is immensely useful for other tasks, such as testing the primality of a number. If F(B) is 1 when B represents a prime number, and F(B) is 0 when B represents a composite number, then a single quantum run of F would test the primality of all numbers from 0 to 2^N simultaneously!!
(We) unfortunately found that one sheet of paper is too heavy for the blimp to lift reliably
Anything usefull that has to be transferred as paper is going to be much larger than a Post-It-Note, so how large must this thing be?
Would filling it with hydrogen instead help? I would love to see a Hindenburg go down at my office...
To proceed somewhat logically:
Since you are not traveling to arrive at a destination, you must be traveling to either:
The very act of bringing your computer along with you obfuscates the beauty of the Midwest. Attempting to remain connected to the internet and actually blogging your journey, undeniably proves the latter reason.
I can therefore conclude, as my final non sequitur, that you are a Mac user, since they obviously enjoy appending useless hype to revered foundations.
Industry leading Games Development Company seeking talented software developers to work for both stock options and salary!!
Excellent opportunity for pseudo-elite narcissistic code-monkeys with mediocre GPA's, 2 years Everquest experience and a predilection for sleazy pornography.
Starting salary $75k-$100k ***
*** In order to be eligible for this introductory salary offer, please sign and overnight the available forms at reputable recruiting services.
A company called Seisint has created a product called Accurint that was used by the government to catch the DC snipers .
Their price list alone is reminiscent of Gattaca which offers the ability to retrieve for most any U.S. citizen their:
6 Neighbors at 10 Different Addresses
Possible Relatives
Possible Drivers Licenses
Criminal (Felony) - 10 Year
It's bad enough if our government uses it to catch terrorists, what happens when AOL uses it to target their mass marketing?!?
I was first introduced to DNA computing by Leonard M. Adleman's article Molecular Computation of Solutions to Combinatorial Problems which describes using DNA computers to solve problems such as the notorious Traveling Salesman problem.
The basic idea is to coerce a ton of DNA into producing random potential solutions to the problem, and to then use chemical processes to select "good" solutions in mass. Since the space of possible solutions to Traveling Salesman problems of any reasonable size is tremendous (larger than the national debt expressed in pesos) DNA computing has an edge over traditional methods, because solutions are easy to generate and then weed out.
Unfortunately, this is really just a gigantic parallel processor - with each strand of DNA the memory of a processor induced by the chemical manipulations, and a small subset of useful algorithms are parallelizable (can be broken up into small "chunks" that can be computed independently and tied back for a larger result.
The immense benefit that this technology will have will be in fields like evolutionary computation. Evolutionary computation relies upon generating large populations of solutions, and then applying simple rules (which could be chemically encoded) to "improve" the generation, towards the pursuit of some ultimate goal. This could be training a neural network to predict coronary artery disease, or optimizing the design of a jet engine without tackling fluid dynamics - truly wondrous!
Self-introspection is redundant, or maybe just indicative of our obession with our selves?
Click, ponder, drag, click again?
Click, click... pirouette, double-click, sashay?
Triple-click, behind the back, no-look double-pump click?
What about a watch that periodically samples biometric data from the skin beneath it?
There are wristwatches that sample glucose for monitoring diabetic's blood-sugar level, such as the GlucoWatch.
There are also devices for processing fluids in a microsystem, such as the MEMS's Biochip.
In the near future the wristwatch could eliminate the need for visible user-token authentication, or at least reduce its frequency. This would greatly increasing both the security of the system and its ease of use.
I called the Markham Police department's phone number, and got a Sargent Knapp's voice mail, so that checks out. However, it doesn't mean the author didn't just look this guy up to add more credibility to the story.
This could all be a hoax, meanwhile people are reading the article and sending money to his girlfriend's Paypal account. Some of the things that struck me:
We are set up to empathize with him:
I almost always buy used, so don't get any ideas about me being rich.
In case you didn't feel bad enough yet:
I was already beyond broke, but I figured $85.00 more wouldn't kill me.
Are you heartless???:
all of your student loans for the next semester are going to cover this counterfeit check.
He'd give to charity if he had money:
I urge you to choose a local charity
Make the check payable to:
If you really must, you can send money to my girlfriend's Paypal account, cranberry_coyote@hotmail.com
So now who's the sucker?