It sure would be interesting read an article or a study that showed reading of RAM after power had been removed for some time. I am sure there is some decay function associated with the persistency of the state of the memory, but I would have thought it was on the order of milliseconds not hours or days.
Remember, according to the NISPOM, when the DoD is certifying a system for classified processing they are looking at the whole enviroment, the builiding, the floor, the room, the people, etc. So that a vulnerability at one level (the computer) is mitigated by security at the other levels (the computer is inside an locked, alarmed room, inside an locked, alarmed building).
The actual computer system is pretty trivial, the only difference may be, just as you identified, the removable hard drive. Just get any of the IDE or even SATA removable hard drive kits and you are set. This is definitely something you can do yourself.
You see the security is in the whole system DoD will be looking for security in layers, many layers. How is the building secured, who has access to the building, the same floor, the floor above & below, the room, etc. What kind of security patrol, alarms, alarms response? What kind of physical security? What kind of walls, ceiling, floor, doors? What kind of electrical service, telecommunication service? The last layer will be the actual computer. What will be attached to the computer, a small LAN, a printer? Don't even think about wireless!
Now, I've said that setting up the computer is trivial, but the administration is NOT. The NISPOM specifies a lot of documentation. Something like writing down the serial number of every component, maybe keeping logs of certain types of activities (loging in, loging out, installing software, updating software, etc.). Checking the logs weekly for suspicious acitivity, etc. If you've heard the old adage that good system administrators write everything down, double it... twice... then you are on the right track.
If your having problems with controllers, drives, enclosures, etc., going bad, then maybe you need to buy better hardware (i.e. more expensive).
I have been working with compaq proliant servers for several years (support for RedHat Linux is good) with nary a hardware problem.
http://h18004.www1.hp.com/products/servers/proli an tml530/index.html http://h18004.www1.hp.com/produ cts/servers/prolian tstorage/arraycontrollers/index.html http://h1800 4.www1.hp.com/products/servers/prolian tstorage/drives-enclosures/4300enclosure/
I know that expensive is not always better or more reliable but failing (or nearly failing) to meet a SLA should get management buy-in to buy just about anything in the $K range.
Yeah, I'll take it, but given that medical care costs have increased by gigantic orders of magnitude in that same time span, we're not getting that much for our dollar.
This gives some details about how much medical spending has increased over the past 40 years.
Looking at "National health expenditures as percent of GDP" (a measure that somewhat takes into account inflation and increased standard of living) there has been a 276% (4.1% vs 15.1%) increase in medical spending.
Life expectancy at ages 20 and up haven't changed all that much...
The National Center for Health Statistics doesn't quite agree with you. Life expectancy by age, race, and sex, 1900-2000
U.S. Life Tables, 2000, table 11
Summary: A person that reached 20 years of age between 1900-1902 could expect to live until they were 62.79 years of age. A person that reached 20 years of age in the year 2000 could expect to live until they were 77.8 years of age.
15 extra years sounds tremendous to me.;-)
What you are proposing will work for general spam checking, but not for Bayes, which is what the original poster asked about. In reality, it's hard to test Bayes in a general case.
The original question was regarding testing to see how they perform in relation to themselves and to other, non-Bayesian filters. So while it is of course best for you to test all of the different spam filters with your spam, it is not as practical as having each developer test their own spam filter again a common, known spam database. If the algorithm is "robust" then it should perform consistently well on lots of different, large training and testing databases.
Ideally, someone, probably an academic, should make a repository of spam available for testing. Software spam filters can say things like, "Correctly classified 99.9% of the email in the UCI spambase 1999-08-20 repository"
You can also save the $72 annual cost of replacing the Friedrich's auxiliary carbon filter. Designed to remove odors, carbon filters have not been very effective in our tests. We found you can leave the Friedrich's original filter in place without losing cleaning performance.
Since you didn't provide any references I did a little googling and found an article from the Washington Post that talks about the results of the media's recount 1 year after the election.
The study by The Post and other media groups, an unprecedented effort that involved examining 175,010 ballots in 67 counties, underscores what began to be apparent as soon as the polls closed in the nation's third most populous state Nov. 7, 2000:
that no one can say with certainty who actually won Florida. Under every scenario used in the study, the winning margin remains less than 500 votes out of almost 6 million cast.
WARNING: Snide remarks follow.
(1) Doing something for an extended period of time means that you can't possibly be killing anything pretty fast.
(2) only a year of spinning while being moved around
One year of spinning is 8760 hours of recording. Sounds like of a lot of birthdays & weddings to me.
Read it slowly, a couple of pages every day. I didn't understand much of what he was trying to say until I read the book the second time.
But why make up my own clever things to say... From http://www.oreilly.com/catalog/regex/desc.html
"There can be certain subtle, but valuable, ways to think when you're using regular expressions, and these can be taught."
I find that books that teach you how to think about problems and solutions are few and far between, and books that do it well are almost impossible to find. This books is one of those.
Once learned regular expressions are one of those things that can profoundly effect the way you work. And once your there "you wonder how they [other people] use UNIX w/o them".
1. Let's say, some time in the future, they get the face-scanning technology to work right. 0.000001% false-positive rate.
2. Let's also say that, among the 250 million people in the United States, one or more people had facial structures similar enough to terrorists' that they would trigger those scanners. In fact, they'd trigger every scanner that person was surveiled by.
This is strawman argument. There needs to be more justification that this scenario really would happen with some nontrivial probability.
I don't think that it is unreasonable to say that right now the police mistake one person for another all the time. But no one is proposing that police stop trying use their eye and look for people. And yet your assumptions lay out a scenario where a computer system that is more accurate than most people can't distinguish between a terrorist and one particular other individual.
Yes, that person would definetly have trouble (if such a person ever exists). But it seems that your assumptions about the accuracy of the computerized recognition system make it even more unlikely to occur than it probably does right now.
Emotionalizing the argument by saying, "What would you do if...", reminds me of sitting around thinking about "What would you do if you won a million dollars... ten million...", "What would you do if someone offered you a million dollars to... (insert something people don't do)." While this can be a fun game to sit around and play on a lazy Saturday afternoon it doesn't have any basis in reality. (Execpt of course some reality TV shows.)
I had a problem similair to the original question.
However, I did not bother the with OCR step. (I can't electronically search my paper books right now, and I still find them awfully usefull:-)
I scanned an out of print technical book (no pictures) two pages at a time on a flatbed scanner, 300 pages total, 300 DPI, black and white (NOT grayscale). I did 15-20 scans at a time and spread my time over couple of days. Yes this is somewhat labor intensive but you will reap the savings next time you have to move a single Zip disk instead of "over 100 pounds of 'essential' technical books."
I saved this as a multi-page TIFF document with Group 4 compression. Total disk space used 8 MB. Yes, that right eight megabytes.
It look just like a nice photocopy. What more do you want? The OCR requirement seems to add a significant amount of risk to the project, both in terms of time and loss of data.
Summary: Black and white scanned images saved in the same format that fax machines use will get you what you want. The OCR is secondary, if you can get anything out it, great, but you don't need it for the project to be valuable.
I sooo wish that the Fortune 500 company that I work for had a couple of these.
Finding anything on the company intranet is next to impossible. The search engine that we have now returns links to documents 2 or 3 years old with URLs on a network that is no longer in service. Arrgg!!
$20K is an insigificant cost for a large company. Yet I think that (at least where I work and I suspect other large corporations) being able to effectively search the company intranet just doesn't seem all that important to the people making the money spending decisions.
Looks like June 6 of last year.
http://www.forbes.com/2001/06/06/0606topblock.html Late fees--or what Blockbuster calls its
"extended-view policy"--have accounted for 19% of the company's revenue in
the past.
Slashdot Mirror
http://igtfy.com/?q=using+gtd+with+google
This event is no big deal, just teaching the fifth lesson.
It sure would be interesting read an article or a study that showed reading of RAM after power had been removed for some time. I am sure there is some decay function associated with the persistency of the state of the memory, but I would have thought it was on the order of milliseconds not hours or days.
Remember, according to the NISPOM, when the DoD is certifying a system for classified processing they are looking at the whole enviroment, the builiding, the floor, the room, the people, etc. So that a vulnerability at one level (the computer) is mitigated by security at the other levels (the computer is inside an locked, alarmed room, inside an locked, alarmed building).
The general specifications for DoD computer systems are freely available to all. NATIONAL INDUSTRIAL SECURITY PROGRAM OPERATING MANUAL OPERATING MANUAL. Specifically, see chapter CHAPTER 8. AUTOMATED INFORMATION SYSTEM SECURITY.
... twice ... then you are on the right track.
The actual computer system is pretty trivial, the only difference may be, just as you identified, the removable hard drive. Just get any of the IDE or even SATA removable hard drive kits and you are set. This is definitely something you can do yourself.
You see the security is in the whole system DoD will be looking for security in layers, many layers. How is the building secured, who has access to the building, the same floor, the floor above & below, the room, etc. What kind of security patrol, alarms, alarms response? What kind of physical security? What kind of walls, ceiling, floor, doors? What kind of electrical service, telecommunication service? The last layer will be the actual computer. What will be attached to the computer, a small LAN, a printer? Don't even think about wireless!
Now, I've said that setting up the computer is trivial, but the administration is NOT. The NISPOM specifies a lot of documentation. Something like writing down the serial number of every component, maybe keeping logs of certain types of activities (loging in, loging out, installing software, updating software, etc.). Checking the logs weekly for suspicious acitivity, etc. If you've heard the old adage that good system administrators write everything down, double it
The Internet Archive: Open Source Audio has a lot of free audio with staff picks, popularity stats, etc.
I can never seem to find _anything_ at the HP site in just 5 clicks.
/ proliantml570/index.html.
n tdl145/index.html.
So here is link to HP's ProLiant with an Intel processor that will support up to 32GB of memory.http://h18004.www1.hp.com/products/servers
Or a rack-mount box with the AMD Opteron(TM) that supports up to 16GB of memory http://h18004.www1.hp.com/products/servers/prolia
If your having problems with controllers, drives, enclosures, etc., going bad, then maybe you need to buy better hardware (i.e. more expensive).
i an tml530/index.htmlu cts/servers/prolian tstorage/arraycontrollers/index.html0 4.www1.hp.com/products/servers/prolian tstorage/drives-enclosures/4300enclosure/
I have been working with compaq proliant servers for several years (support for RedHat Linux is good) with nary a hardware problem.
http://h18004.www1.hp.com/products/servers/prol
http://h18004.www1.hp.com/prod
http://h180
I know that expensive is not always better or more reliable but failing (or nearly failing) to meet a SLA should get management buy-in to buy just about anything in the $K range.
"If you were trying to pick up bunch of little bugs and seeds from a field, which would you rather use? Two strong oxen of 1024 chickens?"
The point is: It is often necessary to match the worker (linux cluster or cray) to the work at hand (crypto or weather sim or whatever).
This gives some details about how much medical spending has increased over the past 40 years.
National Health Expenditures Health, United States 2002, Table 112
Looking at "National health expenditures as percent of GDP" (a measure that somewhat takes into account inflation and increased standard of living) there has been a 276% (4.1% vs 15.1%) increase in medical spending.
The National Center for Health Statistics doesn't quite agree with you.
Life expectancy by age, race, and sex, 1900-2000 U.S. Life Tables, 2000, table 11
Summary: A person that reached 20 years of age between 1900-1902 could expect to live until they were 62.79 years of age. A person that reached 20 years of age in the year 2000 could expect to live until they were 77.8 years of age.
15 extra years sounds tremendous to me.
The original question was regarding testing to see how they perform in relation to themselves and to other, non-Bayesian filters. So while it is of course best for you to test all of the different spam filters with your spam, it is not as practical as having each developer test their own spam filter again a common, known spam database. If the algorithm is "robust" then it should perform consistently well on lots of different, large training and testing databases.
Actually what I am talking about is basic design and testing of statistical pattern recognition algorithms. Check out: The seminal work on the subject Fukunaga, Keinosuke. Introduction to statistical pattern recognition. New York, Academic Press, 1972. And it's revised edition Introduction to Statistical Pattern Recognition (Computer Science and Scientific Computing Series) by Keinosuke Fukunaga Or another classic: Pattern Classification (2nd Edition) by Richard O. Duda, Peter E. Hart, David G. Stork
Maybe someday someone will take the ideas of David B. Fogel and apply them to spam filtering.
Ideally, someone, probably an academic, should make a repository of spam available for testing. Software spam filters can say things like, "Correctly classified 99.9% of the email in the UCI spambase 1999-08-20 repository"
Something like say, the UCI Machine Learning Repository. In fact, look at the UCI spambaseA couple of problems with the UCI spambase. Too old / out of date. And too small.
I looks like there is a more recent community effort going on over a SpamArchive
Looks like you should have googled.
Florida Recounts Would Have Favored Bush But Study Finds Gore Might Have Won Statewide Tally of All Uncounted Ballots
Here's a qoute that emphasizes the point.
(Italics mine.)
Wrong, for just one example see page 2 Maxtor DiamondMax Plus9
Operating Mechanical Shock: 60G
WARNING: Snide remarks follow.
(1) Doing something for an extended period of time means that you can't possibly be killing anything pretty fast.
(2) only a year of spinning while being moved around
One year of spinning is 8760 hours of recording.
Sounds like of a lot of birthdays & weddings to me.
$ dd if=/dev/zero of=/dev/hda
/dev/hda is usually the name give to the first IDE hard drive under Linux.
:)
Note: This is a "Linux-centric" answer to the question since
You may also want to fill the hard drive with (semi)random data.
$ dd if=/dev/urandom of=/dev/hda
If you do this for a couple of weeks you should be fine
Here is a PDF document produced by a company that wants to use this portion of the spectrum, Loea Communications, Service Rules For The Point-to-Point Use Of The 71.0-76.0 GHz and 81.0-86.0 GHz bands
The petition listed above and the company's website give a little more background information on the subject.
Regular expressions aren't so much either a trick or a tool exactly, but you can use them with all the "good" tools.
Get the book "Mastering Regular Expressions," by Jeffrey E. F. Friedl. http://www.oreilly.com/catalog/regex2/
Read it slowly, a couple of pages every day. I didn't understand much of what he was trying to say until I read the book the second time.
But why make up my own clever things to say... From http://www.oreilly.com/catalog/regex/desc.html "There can be certain subtle, but valuable, ways to think when you're using regular expressions, and these can be taught."
I find that books that teach you how to think about problems and solutions are few and far between, and books that do it well are almost impossible to find. This books is one of those.
Once learned regular expressions are one of those things that can profoundly effect the way you work. And once your there "you wonder how they [other people] use UNIX w/o them".
Okay, let's look at your suppositions.
...", reminds me of sitting around thinking about "What would you do if you won a million dollars ... ten million ...", "What would you do if someone offered you a million dollars to ... (insert something people don't do)." While this can be a fun game to sit around and play on a lazy Saturday afternoon it doesn't have any basis in reality. (Execpt of course some reality TV shows.)
1. Let's say, some time in the future, they get the face-scanning technology to work right. 0.000001% false-positive rate.
2. Let's also say that, among the 250 million people in the United States, one or more people had facial structures similar enough to terrorists' that they would trigger those scanners. In fact, they'd trigger every scanner that person was surveiled by.
This is strawman argument. There needs to be more justification that this scenario really would happen with some nontrivial probability.
I don't think that it is unreasonable to say that right now the police mistake one person for another all the time. But no one is proposing that police stop trying use their eye and look for people. And yet your assumptions lay out a scenario where a computer system that is more accurate than most people can't distinguish between a terrorist and one particular other individual.
Yes, that person would definetly have trouble (if such a person ever exists). But it seems that your assumptions about the accuracy of the computerized recognition system make it even more unlikely to occur than it probably does right now.
Emotionalizing the argument by saying, "What would you do if
I had a problem similair to the original question.
:-)
However, I did not bother the with OCR step. (I can't electronically search my paper books right now, and I still find them awfully usefull
I scanned an out of print technical book (no pictures) two pages at a time on a flatbed scanner, 300 pages total, 300 DPI, black and white (NOT grayscale). I did 15-20 scans at a time and spread my time over couple of days. Yes this is somewhat labor intensive but you will reap the savings next time you have to move a single Zip disk instead of "over 100 pounds of 'essential' technical books."
I saved this as a multi-page TIFF document with Group 4 compression. Total disk space used 8 MB. Yes, that right eight megabytes.
It look just like a nice photocopy. What more do you want? The OCR requirement seems to add a significant amount of risk to the project, both in terms of time and loss of data.
Summary: Black and white scanned images saved in the same format that fax machines use will get you what you want. The OCR is secondary, if you can get anything out it, great, but you don't need it for the project to be valuable.
I sooo wish that the Fortune 500 company that I work for had a couple of these.
Finding anything on the company intranet is next to impossible. The search engine that we have now returns links to documents 2 or 3 years old with URLs on a network that is no longer in service. Arrgg!!
$20K is an insigificant cost for a large company. Yet I think that (at least where I work and I suspect other large corporations) being able to effectively search the company intranet just doesn't seem all that important to the people making the money spending decisions.
Looks like June 6 of last year.l
http://www.forbes.com/2001/06/06/0606topblock.htm
Late fees--or what Blockbuster calls its "extended-view policy"--have accounted for 19% of the company's revenue in the past.
Historical information about this bill
. /t emp/~bd1zT4:@@@L&summ2=m&|/bss/d107query.html|
http://thomas.loc.gov/cgi-bin/bdquery/D?d107:1:
Tommorow the following link should become active with todays activities.
http://thomas.loc.gov/r107/r107d12oc1.html