The popular press is very good at promoting the line that computers are dangerous. The courts and "ergonomic consulting" firms seem to buy into the danger as well. But where's the evidence?
Here's a contrary hypothesis: carpal tunnel syndrome and chronic back pain are stress related. That's not to say they aren't real, it's to say that the primary contributory agent is stress. People in repetitive data entry employment may have stressful lifestyles. The stress may be partially caused by the job, or there may be some other non-causal association.
Now sitting differently -- or any other intervention -- may even cause a measurable improvement due to the Hawthorne effect. That's not proof that sitting one way or the other was the cause of the problem.
Exercise -- just getting up and walking, running, swimming, and so on -- probably has more effect than changing posture at the job. Whether that's physiological or psychological or, more likely, both, is unimportant; it works.
I encourage people to check out primary sources or research on these issues, not just statements from consultants who have something to gain from a particular point of view, or trade unions or employers or insurers who have somewhat different axes to grind.
[Says he slouched in bed with wrists heavily on laptop keyboard, who is about to go out for a jog, so as to prevent the chronic back pain that he has suffered from time to time in his life.]
Once again, Slashdot parrots second-hand marketing hype.
If Microsoft has better compression methods, they should demonstrate them in a scholarly venue. Press releases and trade shows are not such a venue.
If Microsoft thinks that a better codec must be supported by a whole new standard, they'd better justify that -- technically, in the marketplace, and in the competition review processes throughout the world.
The definition of spam includes the relationship between sender and recipient. Not the relationship between the sender and you. Admittedly, you don't know as much about that relationship as the actual recipient would, but there are clues.
Perhaps you'll consider these issues next time somebody claims their filter is 99.987% accurate.
After all - this isn't a rigorously-applied, double-blind t-tailed test.
Whatever a double-blind t-tailed test is I do not know.
A double-blind test this most certainly is. The sample is chosen by computer, and you have no idea of the system judgement before you click yours.
As for being a t-test the statistics I gave in the GP constitute an exact binomial test which is more appropriate for discrete judgements than a t-test, which applies to continuous (normally distributed) quantities. Like I said if you got 10/10 that means the probability that you are no better than anybody else (contrary to your claim) is 0.35. That is, the evidence does not even come close to validating your claim.
Since you asserted that you were better than the average, not merely "different" from the average, a single-tailed test is appropriate. In this particular case, since you got 10/10 there is only one tail anyway.
So your ego may tell you that you're better than the rest, but your result doesn't.
"Manually type in the link" is incomplete and dangerous advice. Look up the link using an independent source.
Recent phishing even include a warning to the effect: WE NEVER INCLUDE LINKS. Type in the URL of our homepage paypal-for-you.com and enter your security info.
10 for 10 is perfectly consistent with TFA's observation; If you had a 10% error rate you'd still have a good shot (35%, in fact) at getting ten right. So you cannot conclude from 10 messages that you're good at predicting what the filter will say -- at least not any better than average.
On the other hand, the chance of going 100 for 100 if you're average is.0026% Even 100/100 would be unremarkable if you had say, a 1% error rate.
You're not training any filter. You're participating in a community effort to assess the accuracy of the corpus -- and, as a side-effect, the accuracy of the community judging effort. The corpus is free of charge subject to a usage agreement.
You got 100% on how many messages? If you got did 100 messages and agreed with the corpus on all 100, I'd be impressed. If you agreed for 200 in a row, I'd be amazed.
Just post the final page where it says how many you've classified and how many you agree with the filter on.
"Linux" doesn't manage dependencies. Package systems like.deb and.rpm enforce consistency, and applications like apt and yum do a fine job of supporting distributed repositories of consistent.deb and.rpm packages.
"Windows" on the other hand, uses dlls with no version compatibility information at all. It relies on application-specific installers and uninstallers that rarely work properly. Sure if you use automatic update (and reboot all the time) you get a consistent base OS, but have you ever tried to run Acrobat 5 along with Acrobat Reader 7, for example? Or perhaps a different version of Photoshop? I'm not picking on Adobe in particular. If I happen to use foo.dll and so do you, there's nothing can be done to ensure compatibility (or to update foo.dll without rebooting).
Once again, contest results are announced and people want to over-generalize in one of two directions: either that a country's entire prowess rests on the competition results, or that the results show nothing at all about anything [with the subtext that If I wanted to I could win but I have better things to do].
TopCoder, unlike ACM, actually claims to identify the best coder. What a load of crap. TopCoder is a sport, like ACM, like soccer, or anything else. I sometimes watch hockey. I've even played it. I would never claim that I could be in the Olympics if I wanted to. I could not be in the Olympics or the NHL no matter hard I tried, and that probably has relation to the fact that I don't want to be.
First, if you look at the long-term TopCoder country rankings, they show the US third after Poland and Russia:
Poland is exceptional given its size; they have a very strong contest culture. Canada is also exceptional, currently fourth, followed by China.
If you look at the top schools you'll see two Polish, two Russian, two Chinese and two Canadian schools in the top ten, with Canadian and Polish teams taking slots 11 and 12.
Why is this? Partly culture. TopCoder has a thriving on-line community and is, like any other sport, more popular in some countries than others. I think that secondary school education has something to do with it, too. University/college education is less of a factor.
In any event, the United States for its population ranks just fine at TopCoder. And at the IOI. At the ACM contest, not so well...
Only its impossible to buyout ODF and OpenOffice.org.
Of course, but ODF and OpenOffice.org have insufficient resources to do battle. People will simply assume that "OpenOffice.org doesn't even work with.odf documents (that were created by Word) so it's junk." Exactly the same way they said "Netscape can't render Web pages (created by Microsoft software) so I'm using IE from now on." And no remedy will be forthcoming.
Nothing new or encouraging about this. Microsoft ruined html, Java, and so on by embedding non-standard features supported only by their software. They're well on the way to embedding Windows dependencies in Windows-generated Postscript and PDF files, too.
Transparent as it is, the strategy is remarkably effective. The masses blame the standards-compliant software for "not working", not Microsoft for having poisoned the standard. The courts will sit on their hands and a couple of billion-dollar buyouts will silence the commercial opposition.
Why does Slashdot not report on real spam research? They report puff pieces like this and the phishing talk from the MIT Spam Conference, but not the results of TREC 2005 Spam Track (Hint: an outsider using compression techniques was very strong; open source filters like crm114, dbacl, bogofilter and spamassasin were close behind; DSPAM was middle of the pack.) No filter came close to demonstrating those widely-claimed 99.9-whatever% accuracy figures. I guess "news for nerds -- stuff that matters" includes testimonials but not results.
The TREC tests involved tests on 350,000 email messages. A 92,000 message public corpus from this effort is available for free download.
John Graham-Cumming (no relation to TREC) has created SpamOrHam -- a community-based effort to adjudicate the judgements in the TREC corpus. This'll let us test in a big way Yerazunis' contention that spam filters are better than humans.
Any filter writer can participtate in TREC 2006 by submitting a letter of intent now and a filter in due course.
There's also an upcoming scientific spam conference this summer - CEAS.
I have made or attempted to make many modifications over the years:
- configured several Linuxes to run on laptops
- modified the way SpamAssassin self-trains
- a hack to make "IE view" work with Firefox/Linux/Wine
- [tried] to hook into Mozilla's spam filter interface
(a) to replace the filter
(b) to drive the filter with a test script
[successfully downloaded and built the 70MB source
and experimented but found the interfaces impenetrable]
- various other hacks and patches to dozens of bits
of software: drivers, utilities, printer configs,
dialup scripts, etc.
I suceeded on most but I have no viable way to give what I've learned back to the open source community. Why? The development tools and/or controlling clique are impenetrable. There's no easy plugin or source patch that I can communicate to others -- I either have to join the development community and learn its tools and culture or fork or resort to an informal "howto" web page. These "howto" pages are replicated over and over and over.
I have tried to give back this info via the various community forums (all of which are different and require a fair amount of effort to participate in) and have largely been met with "you're not one of us so your ideas can't be any good." My aim is to make useful hacks not to spend my time arguing about why they're useful. So I end up simply posting my ideas to the 'net where sophisticated searchers/hackers may be able to take advantage of them. But they're no help at all to "joe user."
If we had a lightweight but mechanical way of capturing the hacks that people do to software to make it work on their configuration (especially if their configuration is identical to that of thousands of other users) we'd be better off.
But that requires an architectural change, and a mindset change of the developers. I don't want to join their club in order to use and adapt free software any more than I want to fork over a license fee.
Slashdot Mirror
The popular press is very good at promoting the line that computers are dangerous. The courts and "ergonomic consulting" firms seem to buy into the danger as well. But where's the evidence?
Here's a contrary hypothesis: carpal tunnel syndrome and chronic back pain are stress related. That's not to say they aren't real, it's to say that the primary contributory agent is stress. People in repetitive data entry employment may have stressful lifestyles. The stress may be partially caused by the job, or there may be some other non-causal association.
Now sitting differently -- or any other intervention -- may even cause a measurable improvement due to the Hawthorne effect. That's not proof that sitting one way or the other was the cause of the problem.
Exercise -- just getting up and walking, running, swimming, and so on -- probably has more effect than changing posture at the job. Whether that's physiological or psychological or, more likely, both, is unimportant; it works.
I encourage people to check out primary sources or research on these issues, not just statements from consultants who have something to gain from a particular point of view, or trade unions or employers or insurers who have somewhat different axes to grind.
[Says he slouched in bed with wrists heavily on laptop keyboard, who is about to go out for a jog, so as to prevent the chronic back pain that he has suffered from time to time in his life.]
Once again, Slashdot parrots second-hand marketing hype.
If Microsoft has better compression methods, they should demonstrate them in a scholarly venue. Press releases and trade shows are not such a venue.
If Microsoft thinks that a better codec must be supported by a whole new standard, they'd better justify that -- technically, in the marketplace, and in the competition review processes throughout the world.
That's definitely something to be wary of. Spammers do set up fake sites to use social engineering to crack CAPTCHAS.
In this case, you can verify jgc's reputation using external evidence.
Let me see, 92,000 messages at 1 min/message. A mere 1500 hours. Think you can do it faster with Gimp? OK, 10 sec/message. You do the math.
it is just common sense that needs to be used here, nothing else. Oh wait was that not a nerdy enough comment
Lack of social awareness is a classic nerd characteristic. Often combined with an affectation of superiority. Your comment qualifies.
The definition of spam includes the relationship between sender and recipient. Not the relationship between the sender and you. Admittedly, you don't know as much about that relationship as the actual recipient would, but there are clues.
Perhaps you'll consider these issues next time somebody claims their filter is 99.987% accurate.
Were they really from 2001/2002? How do you know?
After all - this isn't a rigorously-applied, double-blind t-tailed test.
Whatever a double-blind t-tailed test is I do not know.
A double-blind test this most certainly is. The sample is chosen by computer, and you have no idea of the system judgement before you click yours.
As for being a t-test the statistics I gave in the GP constitute an exact binomial test which is more appropriate for discrete judgements than a t-test, which applies to continuous (normally distributed) quantities. Like I said if you got 10/10 that means the probability that you are no better than anybody else (contrary to your claim) is 0.35. That is, the evidence does not even come close to validating your claim.
Since you asserted that you were better than the average, not merely "different" from the average, a single-tailed test is appropriate. In this particular case, since you got 10/10 there is only one tail anyway.
So your ego may tell you that you're better than the rest, but your result doesn't.
"Manually type in the link" is incomplete and dangerous advice. Look up the link using an independent source.
Recent phishing even include a warning to the effect: WE NEVER INCLUDE LINKS. Type in the URL of our homepage paypal-for-you.com and enter your security info.
10 for 10 is perfectly consistent with TFA's observation; If you had a 10% error rate you'd still have a good shot (35%, in fact) at getting ten right. So you cannot conclude from 10 messages that you're good at predicting what the filter will say -- at least not any better than average.
.0026%
On the other hand, the chance of going 100 for 100 if you're average is
Even 100/100 would be unremarkable if you had say, a 1% error rate.
He wrote, along with several text books, Minix, which Linus used as an inspiration, and development platform, for Linux.
You're not training any filter. You're participating in a community effort to assess the accuracy of the corpus -- and, as a side-effect, the accuracy of the community judging effort. The corpus is free of charge subject to a usage agreement.
You got 100% on how many messages? If you got did 100 messages and agreed with the corpus on all 100, I'd be impressed. If you agreed for 200 in a row, I'd be amazed.
Just post the final page where it says how many you've classified and how many you agree with the filter on.
Stop using HTML or convert it plain text and it's hard not to spot a phish.
SpamOrHam.org displays the raw message below the image. Just scroll down a bit.
(a) Avoiding the use of email for business is surrending to the s[pc]ammers.
(b) Do you have a trustworthy white list? I doubt it.
"Linux" doesn't manage dependencies. Package systems like .deb and .rpm enforce consistency, and applications like apt and yum do a fine job of supporting distributed repositories of consistent .deb and .rpm packages.
"Windows" on the other hand, uses dlls with no version compatibility information at all. It relies on application-specific installers and uninstallers that rarely work properly. Sure if you use automatic update (and reboot all the time) you get a consistent base OS, but have you ever tried to run Acrobat 5 along with Acrobat Reader 7, for example? Or perhaps a different version of Photoshop? I'm not picking on Adobe in particular. If I happen to use foo.dll and so do you, there's nothing can be done to ensure compatibility (or to update foo.dll without rebooting).
OK, TV not film, but a great effort: http://imdb.com/title/tt0078037/
Once again, contest results are announced and people want to over-generalize in one of two directions: either that a country's entire prowess rests on the competition results, or that the results show nothing at all about anything [with the subtext that If I wanted to I could win but I have better things to do].
...
TopCoder, unlike ACM, actually claims to identify the best coder. What a load of crap. TopCoder is a sport, like ACM, like soccer, or anything else. I sometimes watch hockey. I've even played it. I would never claim that I could be in the Olympics if I wanted to. I could not be in the Olympics or the NHL no matter hard I tried, and that probably has relation to the fact that I don't want to be.
First, if you look at the long-term TopCoder country rankings, they show the US third after Poland and Russia:
http://www.topcoder.com/stat?c=country_avg_rating
Poland is exceptional given its size; they have a very strong contest culture. Canada is also exceptional, currently fourth, followed by China.
If you look at the top schools you'll see two Polish, two Russian, two Chinese and two Canadian schools in the top ten, with Canadian and Polish teams taking slots 11 and 12.
Why is this? Partly culture. TopCoder has a thriving on-line community and is, like any other sport, more popular in some countries than others. I think that secondary school education has something to do with it, too. University/college education is less of a factor.
In any event, the United States for its population ranks just fine at TopCoder. And at the IOI. At the ACM contest, not so well
Easy. PDF embeds Postscript and Postscript can do all kinds of nasty system-dependent stuff.
Only its impossible to buyout ODF and OpenOffice.org.
.odf documents (that were created by Word) so it's junk." Exactly the same way they said "Netscape can't render Web pages (created by Microsoft software) so I'm using IE from now on." And no remedy will be forthcoming.
Of course, but ODF and OpenOffice.org have insufficient resources to do battle. People will simply assume that "OpenOffice.org doesn't even work with
Nothing new or encouraging about this. Microsoft ruined html, Java, and so on by embedding non-standard features supported only by their software. They're well on the way to embedding Windows dependencies in Windows-generated Postscript and PDF files, too.
Transparent as it is, the strategy is remarkably effective. The masses blame the standards-compliant software for "not working", not Microsoft for having poisoned the standard. The courts will sit on their hands and a couple of billion-dollar buyouts will silence the commercial opposition.
I don't see how this is any different from tainted binary kernel drivers. They'll allow redistribution of the JRE run-time environment. Big deal.
If they allowed redistribution of JDK compiler and libraries, we'd be making progress.
Nothing to see here. Move along.
The TREC tests involved tests on 350,000 email messages. A 92,000 message public corpus from this effort is available for free download.
John Graham-Cumming (no relation to TREC) has created SpamOrHam -- a community-based effort to adjudicate the judgements in the TREC corpus. This'll let us test in a big way Yerazunis' contention that spam filters are better than humans.
Any filter writer can participtate in TREC 2006 by submitting a letter of intent now and a filter in due course.
There's also an upcoming scientific spam conference this summer - CEAS.
I have made or attempted to make many modifications over the years:
- configured several Linuxes to run on laptops
- modified the way SpamAssassin self-trains
- a hack to make "IE view" work with Firefox/Linux/Wine
- [tried] to hook into Mozilla's spam filter interface
(a) to replace the filter
(b) to drive the filter with a test script
[successfully downloaded and built the 70MB source
and experimented but found the interfaces impenetrable]
- various other hacks and patches to dozens of bits
of software: drivers, utilities, printer configs,
dialup scripts, etc.
I suceeded on most but I have no viable way to give what
I've learned back to the open source community. Why? The development tools and/or controlling clique are impenetrable. There's no easy plugin or source patch that I can communicate to others -- I either have to join the development community and learn its tools and culture or fork or resort to an informal "howto" web page. These "howto" pages are replicated over and over and over.
I have tried to give back this info via the various community forums (all of which are different and require a fair amount of effort to participate in) and have largely been met with "you're not one of us so your ideas can't be any good." My aim is to make useful hacks not to spend my time arguing about why they're useful. So I end up simply posting my ideas to the 'net where sophisticated searchers/hackers may be able to take advantage of them. But they're no help at all to "joe user."
If we had a lightweight but mechanical way of capturing the hacks that people do to software to make it work on their configuration (especially if their configuration is identical to that of thousands of other users) we'd be better off.
But that requires an architectural change, and a mindset change of the developers. I don't want to join their club in order to use and adapt free software any more than I want to fork over a license fee.