This might be true, however only if:
1 every attribute changes for every record
2 if they are not related to each other
3 when each column can appear in a WHERE clause.
And IMHO it is still bad design to have 1000+ separate unordered attributes for a single entity. I would like to see a real world example where this is really needed.
When you can make a logical subdivision: PostgreSQL implements array's (also 2D), where you can store vector type data(or "multiple columns") up to 1 Gb, but more importantly it implements user defined composite datatypes: http://www.postgresql.org/docs/8.3/static/sql-createtype.html
Assuming there is some logical subdivision to make in your 1000+ attributes, this can be implemented in a "few" user defined types.
And another method in postgresql would be to use inherited tables, where the attributes that are the same for every record are inherited and the attributes that are specific per project are added to the child table for that project.
And yes, sub-queries can be difficult, but wading through a 1000+ attributes in a single table and maintaining integrity constraints are as well.
A database table is not an Excel sheet with fewer limits! Have some local wizard help you in the design. And if you use postgresql or any other full featured database, you can use views to retrieve your data in a format you need for analysis.
I wish that every organization and government would do this. It is a great tool for research.
In the Netherlands "public" data is still regarded as property and our government charges us for this.
Making feature films with cheap video equipment, supplemented with lots of computer processing, is nothing new. Many scenes of blockbuster films have been made this way, even if the entire film is not. And a few hit movies in recent years have been filmed this way entirely. The difference between actual location, a set, or computer graphics is almost nil in the eye of the audience, so this liberates the film makers from the costs and hassles of staging scenes in costly locations. With computers as cameras you can generate whatever you can imagine.
That part of film magic is evident in any "making-of" movie. What's new is that the new camera/apps are steadily coming becoming like a word processor -- both pros and amateurs use the same one. The great script is not due to a better word processor; it's how the great write uses it. Likewise, a great film is not due to better gear. The same gear needed to make a good film is today generally available to amateurs -- which was not so even a decade ago. Film making gear is approaching a convergence between professional and amateur, so that what counts in artistry and inventiveness.
The newest frontier shaped by this parity seems to be making large-scale films without a lot of extras. Computer generated crowds were first used a decade ago, and reached some public awareness in the Lord of the Rings trilogy. In this memorable scenes batches of smaller crowds were replicated computationally to produce very convincing non-repeating huge crowds. But if you are cheap, desperate and inventive, smaller crowds can be generated from no crowds at all -- just a couple of people.
Here's a clip demonstrating how a World War II D-Day invasion was staged in a few days with four guys and video camera.
Nirvana:~/stuff bill$ chmod 662 reolutions.pdf
Nirvana:~/stuff bill$ ls -hal
-rw-rw--w- 1 bill bill 42M Nov 30 9:58 resolutions.pdf
"Ha, that'll teach them. They can edit, but not read!"
"Hm, on second thought..."
Nirvana:~/stuff bill$ split -b 64k resolutions.pdf
"Memo to myself: Find somebody to hide it on the internet..."
The Blue Gene is a supercomputer designed by IBM, based on their research towards the chess computer Deep Blue that beated Kasparov at his own game. It is not a beowulff cluster (that would by definition consist of consumer hardware).
I don't have time to look it all up again, but a few years ago I was involved in negotiations for the purchase of such a system...
So, from memory:
Each processor (powerpc/cell technology, I think also used in the PS3, but maybe another expert can enlighten me on that one) is a dual core and has six (or was it four?) high speed network connections to its neighbors. 64 processors are mounted on a "motherboard".
In a rack 16 of these boards are installed. The network connections of the processors on the side of the boards are connected to the neighbors on the boards above and below.
Per rack this amounts to 2048 cores. Each rack is connected back to back to another rack, giving a total of 4096 cores in a kind of network matrix.
These dual racks can again be cascaded to make a very large system. (The slant of the racks, see picture, has to do with the cooling of the system). One of the first computers to use such a matrix setup was the transputer in the 80s.
As said, the processors have high speed connections to their direct neighbors, connections to others are slower.
So this machine is very fast at for example signal processing or, more general, any pipeline where the output of one processor can be sent to the next in line for further processing.
Other applications are for example spatial simulations, climate and such, where each processor gets a part of the atmosphere, assuming that effects to other parts will be more local.
A third is biochemical simulations, hence the "Gene" in its name. And when you turn its coolers temporarily of, you might be able to get water hot enough for coffee.
Plan 9 I have no real knowledge on, but it seems to be an operating system that is tailored to enormous amounts of jobs on massive parallel computers.
This link will give you a results page with links to many similar benchmarks of MySQL vs PostgreSQL on their site.
They are very thorough in their tests, are themselves MySQL users, and give insight in which optimizations they used to perform these tests.
A very convincing read.
http://tweakers.net/reviews/674/7
This is a more recent test where the scalability of MySQL and PostgreSQL to an 8-way opteron is reviewed.
Again PostgreSQL 8.2 outperformes MySQL 5 by far.
Interesting is that they discovered a small issue in the stable release of postgresql 8.2 which was subsequently patched by the postgresql developers.
It was still much faster and robust then MySQL which degrades terribly under heavy loads.
The version pg 8.2.1 did ship with these patches.
Another interesting subject is that they revealed issues with Solaris, which were investigated by Sun developers, who helped in configuring the Solaris installation.
http://tweakers.net/reviews/649/7
Tweakers is a dutch community of online tweakers that was deciding on new hardware.
The above link leads you to some stunning performance graphs that show that not only postgresql 8.2 is faster then MySQL 5.0, it also scales MUCH better on heavy loads!
This benchmark was discussed earlier on slashdot to some degree, but I think quite a few of the commenters above have not seen this.
The tweakers community are longterm MySQL users and were new to PostgreSQL.
However they even brought in a MySQL expert to configure it optimally. Still PostgreSQL turned out to be the better dbms
Slashdot Mirror
www.pgadmin.org
This might be true, however only if: 1 every attribute changes for every record 2 if they are not related to each other 3 when each column can appear in a WHERE clause. And IMHO it is still bad design to have 1000+ separate unordered attributes for a single entity. I would like to see a real world example where this is really needed. When you can make a logical subdivision: PostgreSQL implements array's (also 2D), where you can store vector type data(or "multiple columns") up to 1 Gb, but more importantly it implements user defined composite datatypes: http://www.postgresql.org/docs/8.3/static/sql-createtype.html Assuming there is some logical subdivision to make in your 1000+ attributes, this can be implemented in a "few" user defined types. And another method in postgresql would be to use inherited tables, where the attributes that are the same for every record are inherited and the attributes that are specific per project are added to the child table for that project. And yes, sub-queries can be difficult, but wading through a 1000+ attributes in a single table and maintaining integrity constraints are as well.
A database table is not an Excel sheet with fewer limits! Have some local wizard help you in the design. And if you use postgresql or any other full featured database, you can use views to retrieve your data in a format you need for analysis.
I wish that every organization and government would do this. It is a great tool for research. In the Netherlands "public" data is still regarded as property and our government charges us for this.
% make love
no target to make love
text of blog:
Making feature films with cheap video equipment, supplemented with lots of computer processing, is nothing new. Many scenes of blockbuster films have been made this way, even if the entire film is not. And a few hit movies in recent years have been filmed this way entirely. The difference between actual location, a set, or computer graphics is almost nil in the eye of the audience, so this liberates the film makers from the costs and hassles of staging scenes in costly locations. With computers as cameras you can generate whatever you can imagine.
That part of film magic is evident in any "making-of" movie. What's new is that the new camera/apps are steadily coming becoming like a word processor -- both pros and amateurs use the same one. The great script is not due to a better word processor; it's how the great write uses it. Likewise, a great film is not due to better gear. The same gear needed to make a good film is today generally available to amateurs -- which was not so even a decade ago. Film making gear is approaching a convergence between professional and amateur, so that what counts in artistry and inventiveness.
The newest frontier shaped by this parity seems to be making large-scale films without a lot of extras. Computer generated crowds were first used a decade ago, and reached some public awareness in the Lord of the Rings trilogy. In this memorable scenes batches of smaller crowds were replicated computationally to produce very convincing non-repeating huge crowds. But if you are cheap, desperate and inventive, smaller crowds can be generated from no crowds at all -- just a couple of people.
Here's a clip demonstrating how a World War II D-Day invasion was staged in a few days with four guys and video camera.
Youtube link: http://youtube.com/watch?v=WRS9cpOMYv0
Note: not sure if I found the correct version, Richard Hammond presents just three guys. But this is too cool anyway... Just watch it!
You are absolutely right. The subject should have been "success!" to be modded insightfull.
And in a few months time, they'll evaluate and state that the consumers aren't ready yet for DRM-free music.
Do not forget that they can put it in a bank account, or invest with it and live of those profits alone!
Nirvana:~/stuff bill$ chmod 662 reolutions.pdf
Nirvana:~/stuff bill$ ls -hal
-rw-rw--w- 1 bill bill 42M Nov 30 9:58 resolutions.pdf
"Ha, that'll teach them. They can edit, but not read!"
"Hm, on second thought..."
Nirvana:~/stuff bill$ split -b 64k resolutions.pdf
"Memo to myself: Find somebody to hide it on the internet..."
662 was just too much...
The Blue Gene is a supercomputer designed by IBM, based on their research towards the chess computer Deep Blue that beated Kasparov at his own game. It is not a beowulff cluster (that would by definition consist of consumer hardware). I don't have time to look it all up again, but a few years ago I was involved in negotiations for the purchase of such a system...
So, from memory:
Each processor (powerpc/cell technology, I think also used in the PS3, but maybe another expert can enlighten me on that one) is a dual core and has six (or was it four?) high speed network connections to its neighbors. 64 processors are mounted on a "motherboard".
In a rack 16 of these boards are installed. The network connections of the processors on the side of the boards are connected to the neighbors on the boards above and below.
Per rack this amounts to 2048 cores. Each rack is connected back to back to another rack, giving a total of 4096 cores in a kind of network matrix.
These dual racks can again be cascaded to make a very large system. (The slant of the racks, see picture, has to do with the cooling of the system). One of the first computers to use such a matrix setup was the transputer in the 80s.
As said, the processors have high speed connections to their direct neighbors, connections to others are slower.
So this machine is very fast at for example signal processing or, more general, any pipeline where the output of one processor can be sent to the next in line for further processing.
Other applications are for example spatial simulations, climate and such, where each processor gets a part of the atmosphere, assuming that effects to other parts will be more local.
A third is biochemical simulations, hence the "Gene" in its name. And when you turn its coolers temporarily of, you might be able to get water hot enough for coffee.
Plan 9 I have no real knowledge on, but it seems to be an operating system that is tailored to enormous amounts of jobs on massive parallel computers.
http://tweakers.net/search?DB=Reviews&item_catId=1 2&productCatId=&Query=postgresql&x=0&y=0
This link will give you a results page with links to many similar benchmarks of MySQL vs PostgreSQL on their site.
They are very thorough in their tests, are themselves MySQL users, and give insight in which optimizations they used to perform these tests.
A very convincing read.
Apologies for replying to my own posts.
http://tweakers.net/reviews/674/7
This is a more recent test where the scalability of MySQL and PostgreSQL to an 8-way opteron is reviewed.
Again PostgreSQL 8.2 outperformes MySQL 5 by far.
Interesting is that they discovered a small issue in the stable release of postgresql 8.2 which was subsequently patched by the postgresql developers.
It was still much faster and robust then MySQL which degrades terribly under heavy loads.
The version pg 8.2.1 did ship with these patches.
Another interesting subject is that they revealed issues with Solaris, which were investigated by Sun developers, who helped in configuring the Solaris installation.
http://tweakers.net/reviews/649/7 Tweakers is a dutch community of online tweakers that was deciding on new hardware. The above link leads you to some stunning performance graphs that show that not only postgresql 8.2 is faster then MySQL 5.0, it also scales MUCH better on heavy loads! This benchmark was discussed earlier on slashdot to some degree, but I think quite a few of the commenters above have not seen this. The tweakers community are longterm MySQL users and were new to PostgreSQL. However they even brought in a MySQL expert to configure it optimally. Still PostgreSQL turned out to be the better dbms