Slashdot Mirror

Nobody's concerned about normal waves hitting coasts 2 inches higher up. What does concern them is how the rise affects the more extreme events; coastal floods from king tides and storm surges are getting worse, and more frequent - unusually high floods that only happened once a century (1% chance) are now happening once a decade (10% chance).

These floods don't just inundate streets and underground cables, they can contaminate coastal wetlands, aquifers, and farmland with salt. In flat coastal deltas, a small rise in flood levels can extend a much longer way inland, salting the ground and the water table for miles and affecting the livelihoods of many - particularly in poorer countries with large populations depending on once-fertile river deltas. This is a very real problem for many countries without the funds to relocate farms and farmers, especially those like Bangladesh.

Nobody's concerned about normal waves hitting coasts 2 inches higher up. What does concern them is how the rise affects the more extreme events; coastal floods from king tides and storm surges are getting worse, and more frequent - unusually high floods that only happened once a century (1% chance) are now happening once a decade (10% chance).

These floods don't just inundate streets and underground cables, they can contaminate coastal wetlands, aquifers, and farmland with salt. In flat coastal deltas, a small rise in flood levels can extend a much longer way inland, salting the ground and the water table for miles and affecting the livelihoods of many - particularly in poorer countries with large populations depending on once-fertile river deltas. This is a very real problem for many countries without the funds to relocate farms and farmers, especially those like Bangladesh.

What if the gold-plated ones actually sounded much better because they were better made, and more expensive, and didn't rust sitting on the warehouse shelves. What if they also didn't destroy as much groundwater in the making and the workers we're happer and better treated?
The gold/organic doesn't matter, it just makes things easier to find.

If we all demanded organic Orange Juice from Florida, the reduction in fertilizer alone could solve a lot of problems:
 
    https://www.fit.edu/research/p...
http://unclematts.com/dev/conv...
  http://www.nytimes.com/2013/04...

That's just orange juice, in one state. There are thousands of examples.

It often tastes better too, which is a bonus.

Stars are a bit far to resolve, and these stars are about the size of the earth, so no pictures of this will be available in your lifetime. But assuming you accept that we understand physics and can simulate what things look like, please visit http://astro.fit.edu/wood/visualizations.html. This is not new. It's called the Roche lobe and is simply an equipotential surface. Here's an image from a textbook http://physics.uoregon.edu/~jimbrau/BrauImNew/Chap20/FG20_22.jpg

Matt Mahoney to Hutter show details 9:33 AM (7 hours ago) [google.com]

I have uploaded a mirror of Alexander Ratushnyak's new submission to the Hutter prize [hutter1.net] to http://cs.fit.edu/~mmahoney/compression/text.html#1323 [fit.edu] It is in the paq8hp12 section. Scroll down to the bottom of the list of versions just above the table. The submission is decomp8.zip which contains 2 files, decomp8.exe and archive8.bin, the decompressor and compressed file. There is no compressor. To decompress:

decomp8 archive8.bin enwik8

The direct link is http://cs.fit.edu/~mmahoney/compression/decomp8.zip [fit.edu] Decompression took about 2 hours on my computer and used a little over 924 MB memory. The total size of the 2 files is 15,986,677 which passes the 3% threshold improvement from his previous submission of 16,481,655 bytes on May 14, 2007.

The submission was Mar. 23. The 30 day comment period before awarding the prize ends Apr. 22, 2009.

That's exactly what a bot would say.

Matt Mahoney to Hutter show details 9:33 AM (7 hours ago) [google.com]

I have uploaded a mirror of Alexander Ratushnyak's new submission to the Hutter prize [hutter1.net] to http://cs.fit.edu/~mmahoney/compression/text.html#1323 [fit.edu] It is in the paq8hp12 section. Scroll down to the bottom of the list of versions just above the table. The submission is decomp8.zip which contains 2 files, decomp8.exe and archive8.bin, the decompressor and compressed file. There is no compressor. To decompress:

decomp8 archive8.bin enwik8

The direct link is http://cs.fit.edu/~mmahoney/compression/decomp8.zip [fit.edu] Decompression took about 2 hours on my computer and used a little over 924 MB memory. The total size of the 2 files is 15,986,677 which passes the 3% threshold improvement from his previous submission of 16,481,655 bytes on May 14, 2007.

The submission was Mar. 23. The 30 day comment period before awarding the prize ends Apr. 22, 2009.

That's exactly what a bot would say.

I have uploaded a mirror of Alexander Ratushnyak's new submission to the Hutter prize to http://cs.fit.edu/~mmahoney/compression/text.html#1323 It is in the paq8hp12 section. Scroll down to the bottom of the list of versions just above the table. The submission is decomp8.zip which contains 2 files, decomp8.exe and archive8.bin, the decompressor and compressed file. There is no compressor. To decompress:

decomp8 archive8.bin enwik8

The direct link is http://cs.fit.edu/~mmahoney/compression/decomp8.zip Decompression took about 2 hours on my computer and used a little over 924 MB memory. The total size of the 2 files is 15,986,677 which passes the 3% threshold improvement from his previous submission of 16,481,655 bytes on May 14, 2007.

The submission was Mar. 23. The 30 day comment period before awarding the prize ends Apr. 22, 2009.

I have uploaded a mirror of Alexander Ratushnyak's new submission to the Hutter prize to http://cs.fit.edu/~mmahoney/compression/text.html#1323 It is in the paq8hp12 section. Scroll down to the bottom of the list of versions just above the table. The submission is decomp8.zip which contains 2 files, decomp8.exe and archive8.bin, the decompressor and compressed file. There is no compressor. To decompress:

decomp8 archive8.bin enwik8

The direct link is http://cs.fit.edu/~mmahoney/compression/decomp8.zip Decompression took about 2 hours on my computer and used a little over 924 MB memory. The total size of the 2 files is 15,986,677 which passes the 3% threshold improvement from his previous submission of 16,481,655 bytes on May 14, 2007.

The submission was Mar. 23. The 30 day comment period before awarding the prize ends Apr. 22, 2009.

Your CSS is teh suck

1) As several people have already mentioned, Stennis Space Center is on the coast. That's nowhere near Oxford, where Ole Miss actually is, of course.

2) Ole Miss is a Space Grant school. Why it's Ole Miss and not Mississippi State, where the Aerospace Engineering program is, I'll never understand. But that's how it is. There are several lines of space related research going on at both schools, though.

3)I'm not sure what you mean by "Extra [FAA] offices." What are you talking about?

4) "Like studying Oceanography in Colorado." I can't but feel insulted. What would make Texas a better choice than Mississippi? Is it somehow more on the border to space? Space access is hard, no matter where you are at, unlike ocean access. In fact, it's pretty much equally inaccessible to every college program in the country. The Florida schools might have a slight advantage, FIT did launch a rocket a couple years back for the Pioneer Cup. Experts on the matters (space, space law, launch vehicles) are scattered across the country -- and this includes Mississippi. I suspect that the program is at Ole Miss because that's where (at least some) experts in space law happen to work. There is a law school there, you know.



"Even Alabama would be better." God, I hope you're trolling, because you've got the prejudiced asshole act down. Neither Mississippi nor Alabama is perfect, but they are American states in the 21st century. Like pretty much anywhere, there are plenty of poor people, stupid people, ignorant people and prejudiced people (your act could help you fit in). They may even have a larger proportion of such people than other places, but that doesn't preclude them from having plenty of knowledgable, intelligent, reasonable people who can practice space law or build rockets or speak intelligently about nearly anything else.

--sabre86

Sorry, pretty NO compression algorithms are asymetric. The only one i would know of on a hat that is signifficantly asymetric would be LZ77 and 7zip

BWT,RAR,PPM, PAQ and others are more or less symetrical.
PAQ is an especially bad offender. Look here: http://www.cs.fit.edu/~mmahoney/compression/text.h tml . Some version of paq decompressed with 1mbyte/HOUR.

PPM and context based algorithms generally have fairly similar compress and decompress times as well as similar resource usage. The algorithms work by keeping a huge table of statistics based on text already seen (and maybe some initial statistics based on some external corpus) and using that to come up with a table of probabilities as to what the compressor thinks the likelihood is of the next 'symbol' being any of its possible values. This table of probabilities is then used to encode the symbol using an arithmetic encoder. To decode the same symbol, the arithmetic decoder needs to be in the exact same state as the encoder was. So, the same table's must have been generated in the same manner, using the same resources. Also, arithmetic encoding is pretty much symmetric, so there is no real speed gain there. As such, I imagine the algorithm requires a fairly lengthy decompress time as well.

Matt Mahoney lists compression time, decompression time and memory usage (as well as other statistics) for the best Wikipedia compressors: http://www.cs.fit.edu/~mmahoney/compression/text.h tml. A brief look shows that most PPM and CM based algorithms are quite symmetric in performance.

Assymetric compression algorithms are ones that tend to spend a configurable amount of time looking for patterns in the data. The compressed stream ends up being an encoding of the pattern, and the decompressor simply grabs data according to the pattern and outputs it, which is quite fast. Think LZ based algorithms that spend a lot of time looking for string matches, and deciding the optimal way to output them. Or video encoding that spends a lot of time calculating how the scene has moved (how blocks in one frame are related spatially to those in another).

I now await my many awards for searching the Internets - and then another award for each 1% improvement in time that I demonstrate...

http://cs.fit.edu/~mmahoney/compression/paq8hp11an y_src.rar

As linked by other folks on this thread, maximumcompression.com will show that WinRK (proprietary) and PAQ8 (GPL) take the crown in compression. The free PAQ series (wiki, homepage) kick some serious butt...

(Tested on a Project Gutenberg text "The Man who was Thursday")
79105___thurs.paq8l-7
79112___thurs.paq8l
96495___thurs.bz2-9
96708___thurs.rz
107583__thurs.7z
123847__thurs.gz-9
320553__thurs.txt

--
Slashcode bug # 497457 - unfixed since December 2001 - Go look it up!

The PAQ data compressor (best of the open source ones in most comparisons, sometimes overall best) is avaliable from http://www.cs.fit.edu/~mmahoney/compression/paq8l. zip

Yes, that's right. They seem to agree with you...

I recommend that you read this before making any further irrelevant comments.

For all the people laughing at this contest: Read this and this before making any more posts about how ridiculous this contest seems, how stupid it is to compress wikipedia, how compression is such a dumb process, and how bad the compressor was for taking a lot of time to process the text.

I've actually learned about this contest and the theory from a previous story on slashdot. There are some nice mathematics and theoretical results that support this claim. This is also the motivation to have this 'contest', which for all people saying "20% is not much", "it takes too long to be used in applications"; well, this is not about creating better/faster applications to .zip your files, but more about testing the applicability of some interesting theoretical results.

More information can be found here:
http://cs.fit.edu/~mmahoney/compression/rationale. html

http://www.cs.fit.edu/~mmahoney/compression/text.h tml

Info about contenders and results of common compression programs on the testset. (All the "just use gzip/rar/winrk/..." fools can stop jabbering now...)

1. Human generated language models.
2. Machine generated language models.

From Matt Mahoney's rationale for the large text compression benchmark:

In fact, language modeling is an active area of research in speech recognition. Often, it is studied independently of the acoustic model. The most widely used measure of the quality of a language model is its perplexity, which is a measure of how well it will compress a corpus of natural language text [6]. Models are also used to correct errors in other NLP systems such as OCR [7] and language translation [5].

It is common practice in speech recognition research to evaluate a language model isolated from the acoustic model by measuring text compression (expressed as word perplexity), and this method has been found experimentally to be equivalent or superior to other proposed language model evaluation methods [6].

See Matt Mahoney's rationale for the large text compression benchmark.

Unless they're vaccinated, don't give them MUMPS; if you do find a nice Doctor (Like Dr. Pascal), 'cuz Pascal was fun for me in College.

If they like noises, Squeak is good, but the cogently verbiaged might prefer SmallTalk in a group. For those speech impaired, knowing there's other people who Lisp would be good.

The mean ones will abuse Snobol in Winter

The A.D.D. kids will probably like the feeling of Euphoria they get from their first

Of course, you could teach them a very nice language with a horrible name, Brainfuck.

Or, you could just look Here for a comparison of popular programming languages.

This is the exact rule (from http://cs.fit.edu/~mmahoney/compression/textrules. html ):

"The decompressor must be a single Windows (x86 32 bit) or Linux (x86 32 or 64 bit) executable file. It is not zipped. The size of the source code (if available) is not used. The decompressor must not depend on any other files not normally part of Windows or Linux such as dictionaries, configuration files, etc.

Which means that you cannot make a 100mb library to go with your file, I guess.

Another thing to note is that the file, enwiki8, isn't actually 100mb, it's 100,000,000 byte, ie 10^8 byte, not 100*2^20 byte. So not really 100mb. To me this is strange, I mean any real CS guy would have gone with binary, right? Only a real newbie would go with the exact decimal number that make very little sense in computer terms.

Points are not awarded for attempting to circumvent the intent of the competition. I expect such attempts would result in future submissions from the same source being ignored.

A (good) sign of the times, I guess.

There are some amazing compression programs out there, trouble is they tend to take a while and consume lots of memory. PAQ gives some impressive results, but the latest benchmark figures are regularly improving. Let's not forget that compression is not good unless it is integrated into a usable tool. 7-zip seems to be the new archiver on the block at the moment. A closely related, but different, set of tools are the archivers, of which there are lots with many older formats still not supported by open source tools

See Matt Mahoney's description of Marcus Hutter's proof that compression is equivalent to general intelligence.

The profound danger of a biased AI here is quite avoidable. The theoretic problem of unbiased AI has been formally solved by Marcus Hutter with AIXI:

Computational AI. There are strong arguments that AIXI is the most intelligent unbiased agent possible in the sense that AIXI behaves optimally in any computable environment.

This is the reason I set up the following definition of the C-Prize:

Let anyone submit a program that produces, with no inputs, one of the major natural language corpora as output.

S = size of uncompressed corpus
P = size of program outputting the uncompressed corpus
R = S/P (the compression ratio).

Award monies in a manner similar to the M-Prize:

Previous record ratio: R0
New record ratio: R1=R0+X
Fund contains: $Z at noon GMT on day of new record
Winner receives: $Z * (X/(R0+X))

Compression program and decompression program are made open source.

Explanation For an idea of why the C-Prize can solve the AI problem, if it is solvable, see Matthew Mahoney's comment on it:

Matt Mahoney
Jun 17, 7:18 pm show options
Newsgroups: comp.compression
From: "Matt Mahoney"
Date: 17 Jun 2005 20:18:59 -0700
Local: Fri, Jun 17 2005 7:18 pm
Subject: Re: The C-Prize

Hutter's AIXI, http://www.idsia.ch/~marcus/ai/paixi.htm makes another argument for the connection between compression and AI that is more general than the Turing test. He proves that the optimal behavior of an agent (an interactive system that receives a reward signal from an unknown environment) is to guess that the environement is most likely computed by the shortest possible program that is consistent with the behavior observed so far. In other words, the most likely outcome for any experiment is the one with the simplest explanation, where "simplest" means the smallest program that could model what you currently know about the universe.

He gives a formal proof, but it basically says that the only possible distribution of the infinite set of programs (or strings) with nonzero probability is one which favors shorter programs over longer ones. Given any string of length n with probability p > 0, there are an infinite set of strings longer than n, but only a finite number of these can have probability higher than p.

-- Matt Mahoney

Matt Mahoney is the author of Text Compression as a Test for Artificial Intelligence which states:

It is shown that optimal text compression is a harder problem thanartificial intelligence as defined by Turing's (1950) imitation game; thus compression ratio on a standard benchmark corpuscould be used as an objective and quantitative alternative test for AI (Mahoney, 1999).

(Mahoney is also a competitor who has some winnings from The Calgary Corpus Compression Challenge.)

Now a big question here is whether it might be possible to create a verifiably unbiased AI without making the compression program open source. In any case I don't think it is wise to trust any AI that hasn't at least gone through a compression competition with other purportedly unbiased AI's compressing an open source corpus.

Now, who might fund something like the C-Prize?

Well, here's a suggestion:

Since:

1. Larry Page is on the board of directors of the X-Prize Foundation.
2. The mission of the X-Prize Foundation:

There are some amazing compression programs out there, trouble is they tend to take a while and consume lots of memory. PAQ gives some impressive results, but the latest benchmark figures are regularly improving. Let's not forget that compression is not good unless it is integrated into a usable tool. 7-zip seems to be the new archiver on the block at the moment. A closely related, but different, set of tools are the archivers, of which there are lots with many older formats still not supported by open source tools

You should've given PAQ a try too. From what I understand PAQ compression uses adaptive switching between multiple compression algorithms on the fly based on which produces the best result for a current block. Be warned that it is pretty slow and memory intensive.

Another one to try is UCL . This is a compression engine behind UPX, executable file compressor. It has a remarkable property of having super-fast decompression.

I would feel better if the state would pass http://blackbox.cs.fit.edu/blog/kaner/archives/000 124.htmlthis software customer bill of rights. I especially like #4 "User has right to see and approve all transfers of information from her computer. Before an application transmits any data from the user's computer, the user should have the ability to see what's being sent. If the message is encrypted, the user should be shown an unencrypted version. On seeing the message, the user should be able to refuse to send it. This may cause the application to cancel a transaction (such as a sale that depends on transmission of a valid credit card number), but transmission of data from the user's machine without the user's knowledge or in spite of the user's refusal should be prosecutable as computer tampering."

1 x "Microsoft Analyst Review and Report"
@ $26 000

1 x "FL Tech will deliver services to define and document all the various aspects of testing for security vulnerabilities in Microsoft software, as directed by Microsoft."
@ $50 000

1 x "Microsoft Analyst Review and Report"
@ $26 000

1 x "FL Tech will deliver services to define and document all the various aspects of testing for security vulnerabilities in Microsoft software, as directed by Microsoft."
@ $50 000

Well, apparently this is the second time Microsoft has come out on top of a research project by Mr. Richard Ford.

http://www.virusbtn.com/magazine/articles/letters/ 2004/01_01.xml

Apparently there was some question to the validity of an earlier project because it was sponsored by Microsoft.

However, I would like to note that both researchers seem very well educated, especially in computer security. And, additionally, they both note that a lot more could be done to lock down the Linux server.

The intro reminded me of the "Everything that can be invented has been invented" comment by Charles H. Duell in 1899 and the rest of it made feel like I was reading a hip-hop cover of Eric Raymond's Cathredral & the Bazaar with a few verses left out.

• 1 hour
• 10 hours
• 100 hours

Each class should have its own championship title of $1 million, with each runner-up winning 1/2 the money of the next higher.

Each contestant must provide 2 systems -- a compressor and a decompressor. DARPA feeds the compressor the corpus and the compressor feeds DARPA the compressed corpus. DARPA then measures the ratio and feeds the decompressor system the compressed corpus, which then returns the original corpus, or is disqualified. Compression and decompression times must add up to no more than the time limit for the competition class.

The rationale for this approach to advancing the state of AI is given by a short paper by Matthew Mahoney titled "Text Compression as a Test for Artificial Intelligence" (1 page poster, compressed Postscript) published in the 1999 AAAI Proceedings. Matt Mahoney shows that text prediction or compression is a stricter test for AI than the Turing test.

So far there have been lots of promises and decades spent. Let's try something different with well-founded objetive metrics tied to serious near-term commercial incentives for evolutionary progress.

Viewing this mapping as a data compressor, we connect to earlier work on Normalized Compression Distance.

This is basically what I was referring to in my response to "Using The Web For Linguistic Research" when I said:

There needs to be an annual prize for the highest compression ratio using random pages from the web as the corpus. This would probably do more for real advancement of artificial intelligence than the Turing competitions.

Intelligence can be seen as the ability to take a sample of some space and generalize it to predict things about the space from which the sample was drawn. The smaller the sample and the more accurate the prediction, the greater the intelligence. This is also a short description of what a compression algorithm does.

Text Compression as a Test for Artificial Intelligence, 1999 AAAI Proceedings. Matt Mahoney shows that text prediction or compression is a stricter test for AI than the Turing test. (1 page poster, compressed Postscript).

Text Compression as a Test for Artificial Intelligence, 1999 AAAI Proceedings. Matt Mahoney shows that text prediction or compression is a stricter test for AI than the Turing test. (1 page poster, compressed Postscript).

http://cs.fit.edu/~mmahoney/compression/
http://www.msoftware.co.nz/WinRK_benchmarks.php

http://cs.fit.edu/~mmahoney/compression/
http://www.msoftware.co.nz/WinRK_benchmarks.php

http://cs.fit.edu/~mmahoney/compression/
http://www.msoftware.co.nz/WinRK_benchmarks.php

Surely they mean SNOBOL.

Also, several new teams are entering, among them Stanford, and Florida Tech. Should be a very interesting Challenge next year!"

Its actually University of Florida and not Florida tech.

As a proud dropout of Florida Tech, I'd just like to point out that the "Florida Tech" link goes to an article about the University of Florida. If Florida Tech is competting, I wish them all the best luck. Here's the correct link to their website: www.fit.edu

SPSS has been around since 1968. Check out the very fascinating history of this program here. BTW, I still have a card deck with the original Fortran source code, from 1971, when I ran it on an IBM 1130 minicomputer. Kermit was born c. 1980, and in contrast is "just a kid." Possibly the truly oldest continuously supported software -- which is open source -- is the Fortran IV Scientific Subroutine Library (I have a copy dating back to 1962, which was based on the Fortran II library, which is presumably older but no longer supported). The library was eventually codified as an ANSI standard in 1977. A capsule history is at fit.edu. You can access an archive of the Fortran source code at Harwell's..

>> Really? Computers have been around for 60 years??

Yes.

Donald Knuth seems to like INTERCAL.
Mentioned on his Recent News page (towards the middle of the page).
He implemented an algorithm in it here.
The same algorithm implemented in various other languages is here.

...in the past thinking like this has resulted in things like Ada.

Yes, this is true. Ada was devised as the result of 4 groups competing against each other, and the best of em (though opinions vary) selected as the Mandated Language for the DOD.

The whole process is described thusly:

The driving concern of the HOLWG was to assure that the design was guided by a responsible principle investigator, and to preclude "design by committee". On the other hand, picking a single contractor to do the job and trusting to luck would have been imprudent. The procurement was through multiple competitive contracts, with the best products to be selected for continuation to full rigorous definition and developmental implementation.

And it was a disaster from the Defence Contractor's viewpoint. Firms fell over themselves trying to get exemptions from using it, ANYTHING was better. Because Ada had

• Objects
• Exceptions
• Generics (templates)
• Multiple-threading/Tasking
• Strong Typing

So yes, had compatibility been mandated in the past, we might all have systems far more reliable and robust. But Microsoft wouldn't have $40 Billion and a number of tame Congresscritters.

Thank you for your comments, but the Ada FAQ (see section 2.6), a timeline of the language's history, and a writeup for a UMich course suggest otherwise. While S. Tucker Taft may have led the Ada 95 revision team and Jean Ichbiah led the Cii Honeywell Bull team designing the submission to the DoD (which became Ada 83), the design of the language is clearly not the work of a small number of people.

Rather, it was designed to meet a specification from a DoD working group and received feedback from hundreds of reports (and dozens of individuals or other groups).

While Robert Dewar's answer to "Was Ada designed by a committee?" is certainly witty, a rose by any other name would smell the same. Taft's answer lists -- for just Ada 95 -- not only the core "Design Team," but five other full- or nearly-full-time teams ("Language Precision Team", "Requirements Team", and three "user/implementor" teams) and the large group of "Distinguished Reviewers."

Given all that, I will retract my description of Ada as being designed by a committee -- it was in fact designed by a beauracracy!

(As for VHDL, I cannot find as detailed information on the web about its history, and don't have my VHDL books with me right now. However, the timeline at the University of Erlangen-Nürnberg's VHDL-online -- with discussions and defining requirements taking up most of a decade -- suggests that it suffered from too many cooks at times.)

While I agree that Ada and VHDL are very expressive and powerful, it's not good to get rose-colored glasses about their history or (especially in VHDL's case) drawbacks.