Actually, tablespaces is already in CVS tip today, the day of the feature freeze and should make it into the final release.
Writing to a raw partition is no real winner for PostgreSQL, and not likely to be implented until someone can SHOW that they'd be faster for an actual test case and then do the work.
That's not the same thing. If the higher ups at Oracle tell the Oracle developers to include a dubios feature that may be bug ridden, the developers DO IT because they were told to.
If IBM tells the apache group to put in dubious and buggy code, the apache group tells them to buzz off.
There is a difference, even if it isn't obvious at first glance.
No, it doesn't. There's a new buffer manager that runs the vacuums at a much lower I/O priority than before, meaning it takes three or five times longer, BUT it doesn't slow the machine down.
Of course, you'd have to have known what you were talking about to have known that before you posted your idiotic reply, but I'm sure thinking isn't your strong suit.
3ware escalades are quite good.
on
SATA vs ATA?
·
· Score: 1
and they support real RAID configurations, like RAID 1+0 or 5 etc...
Actually, installing from Source is actually pretty darned easy for PostgreSQL. or Xine, or gxine_client, or a whole bunch of other projects out there.
The hardware will NOT be free. The cost will simply be rolled into the price of the software. This is simply a marketing ploy to try and lock people into non-open hardware with cheap up front costs that just keep repeating over and over.
It's not gonna work, but I'm sure Sun and Microsoft are gonna try anyway.
Actually, I'd say Dell is as Open Source friendly as they can be, but only on servers. They have their own developer working on drivers for things like RAID controllers and such.
On the desktop, however, they are at best luke warm towards open source. And funnily enough, their recent desktop systems also seem to have a problem running Microsoft's Virtual PC software as well.
It's difficult to partner with PG as there is not a controlling company.
I'm sorry, but I just disagree on this. While there is no one company to partner with financially, there certainly is a single controlling group known as the "PostgreSQL Global Development Group" or PGDG.
Much like the Apache Group, the PGDG is the group that's the head mofos in charge. Their website is developer.postgresql.org
Much like the Apache Group, the PGDG is made up of the key developers, each sponsored in one way or another by some interested party. If HP were wanting to sponsor a developer to work on PostgreSQL, it is quite likely that they could find one or two good ones already working on it in their spare time who would be happy to stay at home and hack on it. Compare the developer mailing lists for both MySQL and PostgreSQL, and you'll find a much larger number of developers working on PostgreSQL than MySQL.
By the way, there is a seperate marketing arm, hosted on advocacy.postgresql.org which is where PostgreSQL
But claiming you need a commercial arm in order to properly partner with an Open Source project is to completely misunderstand how real open source projects get done.
Simple, really. People who want PostgreSQL aren't as likely to be the people who need HP to hold their hands. They either are old hands and build form source, or they can just use the RPM bundled with whatever distro they are using.
Plus, PostgreSQL still presents a higher maintenance load and learning curve for most people. While PostgreSQL has made great strides in being relatively maintenance free for joe average user, it still requires a little more than a passing knowledge of what you're doing to keep it happy (i.e. vacuuming, editing things like random_page_cost, setting up shared_buffers and the kernel to support them, and so on.) This is in no way an indictment of PostgreSQL, as I personally am quite willing to endure some small amount of administration / performance tuning to get a fast and reliable database that stays up and running and can do some very heavy lifting.
When I first started using pgsql, I found it much easier to just compile from source than to try to use RPMs, so I got used to installing from source. This was typical four or five years ago, due to issues with RPM updates not being able to automagically upgrade the data store. MySQL has an advantage in that all versions can upgrade in place over previous versions, so RPMs were commonly used for it, making it far more ubiquitous in the past. PostgreSQL still suffers from the problem of needing to dump / reload between major revisions, and occasionally between minor revisions for certain bugs.
So, the folks who need it don't need HP, and the folks who need HP are the ones most likely to already have Oracle installed elsewhere, and need a small, fast database to store things like web pages. MySQL is pretty good at that. PostgreSQL is also good at it, but it doesn't have the kind of popularity that MySQL enjoys in things like magazines.
But it is interesting how Sun kind of provides NO information about alternate sources for Java. While they may have some small reason to prefer their own flavors, they would seem to have more reasons to support alternates that handled hardware / OS combos they don't support.
This is one area where more open languages like Perl, PHP, and Python have an advantage. All you need to know is how to find the home page and you're pretty much golden. To quote the Perl web site:
Note that CPAN does not build these packages: we just provide the hyperlinks. So please don't ask to build a package for you: you have access to the plaftorm, not us. Also, this page lists operating systems, not hardware platforms: therefore Perl packages for, say, Linux PDAs or SONY Playstation, or XBox, or Tivo, or toasters, or so forth running Linux are not "ports" as such.
I.e. if the hardware runs an OS listed here, Perl should work. The same is true of most other free / OS languages.
I don't think the real problem is the harder certification process Sun has, it's their tendency to think more of their brand than their user's needs.
The download page for every free / OS language I've ever seen had a list of every OS / hardware platform (where applicable) supported and whether / if it had been ported to the latest version for your system yet.
This tendency to hide ugly information to make a product look prettier to pointy haired boss types works against Sun's Jave in the long run, because it's harder to find a java for certain platforms, and makes it more susceptable to loss of users due to migration to other, more easily findable languages for odd / uncommon hardware / OS combos.
While the number of computers sold that aren't i386 can't be more than a few percent, that few percent could well contain some tiny number of enterprise users who need an odd piece of hardare / OS combo to get the job done.
Now, in the Open Source realm, the refrain is, "you have access to the platform, not us" which is actually a backhanded offer of support. You compile it, or try to, and report back on your progress. We'll try to help you figure it out or ask on the mailing lists to see who else has tried and what their results are. No guarantees, but nearly complete disclosure, and, usually, better coverage.
And the thing is, Java doesn't make Sun any money directly, so marketing it like a product is stupid. It's a language, and they need to offer support for the language from a paternalistic kind of point of view. I.e. their job isn't to corner the market on java SDKs and such, but on their app server and directory technologies and such. The attitude towards Java, the language, really should be something like how Perl or PHP does it. Here're the links, we run on just about everything but X, Y, and Z, and if you can't get it to work, let us know. Play no favorites. The more people you can get to port your language for you, as a company, the better.
Instead, like a 3 year old child wanting a toy simply because he can't have it, they cry out "Mine!" instead of taking care of their users.
Funny you should mention that. It seems the escalade IDE RAID controller doesn't suffer from this problem. My guess is that it uses its own cache and turns off the drive's cache to achieve reliability.
So, while I'm not a big fan of IDE and consider most tests done on them to be non-valid due to the problems with fsync, the escalade is the apparent exception to the rule. It's fast AND reliable, which is rather non-IDE like in nature...
I don't think you're understanding what I'm saying. Here's a brief time line based explanation.
The OS writes to the journal what it's going to do. It calls fsync to make sure the journal is on the disk. The disk says "oh yeah, it's there." The file system then writes the actual data, and fsyncs that. The disk, again, tells it that it wrote the data out. The file system then marks that part of the journal as having been written.
However, the disk hasn't actually written out its cache yet. It lied to the OS / file system and said it had, but it hasn't, it's busy doing something else. Poof, the power goes out.
Now, the journal doesn't have our data, we've already cleared it out, and the file system, which is supposed to have been coherent because we fsynced it, is not, and it is now corrupted.
I have reproduced this behavior a dozen or more times on IDE based systems. The only way to stop it is to tell the drive to stop using it's write cache.
Now, SCSI drives don't lie about fsync. At least none of the ones I've tested have done that. so, when the file system asks the disk to fsync and reports back that it has fsynced, the data really is on the drive. And we can now roll forward the journal pointed and proceed with the knowledge our data is coherent on the drive.
You can prove this to yourself. Set up a server on IDE and another on SCSI. Install postgresql, running on a journaled file system like ext3, and then run the follow commands on each:
Now, in the middle of the test on both machines, pull the plug.
When you restart the machines, the SCSI one will come right up, database coherent and no problem. the IDE one will fail, at least every so often, or as in my experience, nearly every time.
It looks like Oracle has gotten the same basic results as the PostgreSQL Global Development Group have. JFS and ext3 are generally the fastest under a database, while XFS and Reiser seem to be pretty slow.
The odd thing here is that most other tests show XFS and Reiser as the kings. But the kind of random access databases do seems to be better handled by JFS / ext3.
The problems with your RAID consistency, were these file system problems, or RAID level problems? If they're RAID level problems that would seem to point at your RAID controller, as no file system should be able to bonk a RAID array on the head, since it doesn't really have that kind of access to it.
Thanks for the correction. By any chance was that ReiserFS partition on an IDE drive with write cache enabled? See some of my other posts in this thread for the problems IDE drives with enabled write cache present.
If you are using IDE drives with the write cache enabled, then NO journaling file system is safe on your system. This is because IDE drives with write cache respond immediately to requests for fsync with true, whether they've written the data out or not.
If your data is important, either turn off the cache on your IDE drive or buy a SCSI drive, which won't lie about fsync. This is a problem for both linux and BSD.
Later model IDE drivers and drives may be able to work properly with cache enabled, but not now. There are ongoing discussions on BOTH kernel hackers lists, BSD and Linux, about what to do, and no solution in sight.
Sorry, but there are a few reasons these results are questionable. As one poster has already pointed out, ext3 by default journals both meta data (i.e. directory structure) AND data. The other three by default only journal meta data.
But more importantly than that is the fact that in linux, that IDE drive is responding immediately to fsync requests before it's actually done them. I.e. when the journaling file systems says "hey, did you get that stuff written out to the disks?" The drive says "yes sir!" Then writes it out whenever it gets a chance after that.
This means that you may well have a corrupted directory structure should the machine lose power in the middle of writing data, because the hard drive LIED to the OS, and the journaling file system flushed out it's journal when it wasn't actually written to the disk.
To get trustworthy results, you need to either 1: turn off the write cache on the IDE drive or 2: Use SCSI, which doesn't lie about such things.
As long as you're writing to an IDE drive with an enabled 8 meg cache, the results are worthless, because you don't actually have the reliability you think you have, and the numbers might change drastically when you start using drives (like SCSI) that actually fsync properly.
Slashdot Mirror
Take a look here:
efeu
Do a google search for slony. It's in early beta right now, but looks very promising.
Tablespaces is already in CVS, so unless some truly hideous, unfixable bug shows up, it should make it into beta come August or so.
Actually, tablespaces is already in CVS tip today, the day of the feature freeze and should make it into the final release.
Writing to a raw partition is no real winner for PostgreSQL, and not likely to be implented until someone can SHOW that they'd be faster for an actual test case and then do the work.
That requires two or three phase commit, which I don't know if it's going to make it into 7.5 or not.
That's not the same thing. If the higher ups at Oracle tell the Oracle developers to include a dubios feature that may be bug ridden, the developers DO IT because they were told to.
If IBM tells the apache group to put in dubious and buggy code, the apache group tells them to buzz off.
There is a difference, even if it isn't obvious at first glance.
No, it doesn't. There's a new buffer manager that runs the vacuums at a much lower I/O priority than before, meaning it takes three or five times longer, BUT it doesn't slow the machine down.
Of course, you'd have to have known what you were talking about to have known that before you posted your idiotic reply, but I'm sure thinking isn't your strong suit.
and they support real RAID configurations, like RAID 1+0 or 5 etc...
http://www.3ware.com/products/serial_ata.asp
Suddenly all those "I'm patenting peanut butter and jelly sandwich" posts are seeming more and more prophetic...
I'm hoping that such insane uses of patents will result in the USPTO and Congress waking the hell up and fixing this mess.
Sorry for feeding the trolls, but...
Actually, installing from Source is actually pretty darned easy for PostgreSQL. or Xine, or gxine_client, or a whole bunch of other projects out there.
Reading leet speak, OTOH, is a pain in the arse
The hardware will NOT be free. The cost will simply be rolled into the price of the software. This is simply a marketing ploy to try and lock people into non-open hardware with cheap up front costs that just keep repeating over and over.
It's not gonna work, but I'm sure Sun and Microsoft are gonna try anyway.
Actually, I'd say Dell is as Open Source friendly as they can be, but only on servers. They have their own developer working on drivers for things like RAID controllers and such.
On the desktop, however, they are at best luke warm towards open source. And funnily enough, their recent desktop systems also seem to have a problem running Microsoft's Virtual PC software as well.
It's difficult to partner with PG as there is not a controlling company.
I'm sorry, but I just disagree on this. While there is no one company to partner with financially, there certainly is a single controlling group known as the "PostgreSQL Global Development Group" or PGDG.
Much like the Apache Group, the PGDG is the group that's the head mofos in charge. Their website is developer.postgresql.org
Much like the Apache Group, the PGDG is made up of the key developers, each sponsored in one way or another by some interested party. If HP were wanting to sponsor a developer to work on PostgreSQL, it is quite likely that they could find one or two good ones already working on it in their spare time who would be happy to stay at home and hack on it. Compare the developer mailing lists for both MySQL and PostgreSQL, and you'll find a much larger number of developers working on PostgreSQL than MySQL.
By the way, there is a seperate marketing arm, hosted on advocacy.postgresql.org which is where PostgreSQL
But claiming you need a commercial arm in order to properly partner with an Open Source project is to completely misunderstand how real open source projects get done.
Simple, really. People who want PostgreSQL aren't as likely to be the people who need HP to hold their hands. They either are old hands and build form source, or they can just use the RPM bundled with whatever distro they are using.
Plus, PostgreSQL still presents a higher maintenance load and learning curve for most people. While PostgreSQL has made great strides in being relatively maintenance free for joe average user, it still requires a little more than a passing knowledge of what you're doing to keep it happy (i.e. vacuuming, editing things like random_page_cost, setting up shared_buffers and the kernel to support them, and so on.) This is in no way an indictment of PostgreSQL, as I personally am quite willing to endure some small amount of administration / performance tuning to get a fast and reliable database that stays up and running and can do some very heavy lifting.
When I first started using pgsql, I found it much easier to just compile from source than to try to use RPMs, so I got used to installing from source. This was typical four or five years ago, due to issues with RPM updates not being able to automagically upgrade the data store. MySQL has an advantage in that all versions can upgrade in place over previous versions, so RPMs were commonly used for it, making it far more ubiquitous in the past. PostgreSQL still suffers from the problem of needing to dump / reload between major revisions, and occasionally between minor revisions for certain bugs.
So, the folks who need it don't need HP, and the folks who need HP are the ones most likely to already have Oracle installed elsewhere, and need a small, fast database to store things like web pages. MySQL is pretty good at that. PostgreSQL is also good at it, but it doesn't have the kind of popularity that MySQL enjoys in things like magazines.
I always wanted to get Tipper in bed and get her to talk dirty. That would be hot... :-) Seriously, hottt...
But it is interesting how Sun kind of provides NO information about alternate sources for Java. While they may have some small reason to prefer their own flavors, they would seem to have more reasons to support alternates that handled hardware / OS combos they don't support.
This is one area where more open languages like Perl, PHP, and Python have an advantage. All you need to know is how to find the home page and you're pretty much golden. To quote the Perl web site:
Note that CPAN does not build these packages: we just provide the hyperlinks. So please don't ask to build a package for you: you have access to the plaftorm, not us. Also, this page lists operating systems, not hardware platforms: therefore Perl packages for, say, Linux PDAs or SONY Playstation, or XBox, or Tivo, or toasters, or so forth running Linux are not "ports" as such.
I.e. if the hardware runs an OS listed here, Perl should work. The same is true of most other free / OS languages.
I don't think the real problem is the harder certification process Sun has, it's their tendency to think more of their brand than their user's needs.
The download page for every free / OS language I've ever seen had a list of every OS / hardware platform (where applicable) supported and whether / if it had been ported to the latest version for your system yet.
This tendency to hide ugly information to make a product look prettier to pointy haired boss types works against Sun's Jave in the long run, because it's harder to find a java for certain platforms, and makes it more susceptable to loss of users due to migration to other, more easily findable languages for odd / uncommon hardware / OS combos.
While the number of computers sold that aren't i386 can't be more than a few percent, that few percent could well contain some tiny number of enterprise users who need an odd piece of hardare / OS combo to get the job done.
Now, in the Open Source realm, the refrain is, "you have access to the platform, not us" which is actually a backhanded offer of support. You compile it, or try to, and report back on your progress. We'll try to help you figure it out or ask on the mailing lists to see who else has tried and what their results are. No guarantees, but nearly complete disclosure, and, usually, better coverage.
And the thing is, Java doesn't make Sun any money directly, so marketing it like a product is stupid. It's a language, and they need to offer support for the language from a paternalistic kind of point of view. I.e. their job isn't to corner the market on java SDKs and such, but on their app server and directory technologies and such. The attitude towards Java, the language, really should be something like how Perl or PHP does it. Here're the links, we run on just about everything but X, Y, and Z, and if you can't get it to work, let us know. Play no favorites. The more people you can get to port your language for you, as a company, the better.
Instead, like a 3 year old child wanting a toy simply because he can't have it, they cry out "Mine!" instead of taking care of their users.
Funny you should mention that. It seems the escalade IDE RAID controller doesn't suffer from this problem. My guess is that it uses its own cache and turns off the drive's cache to achieve reliability.
So, while I'm not a big fan of IDE and consider most tests done on them to be non-valid due to the problems with fsync, the escalade is the apparent exception to the rule. It's fast AND reliable, which is rather non-IDE like in nature...
I don't think you're understanding what I'm saying. Here's a brief time line based explanation.
The OS writes to the journal what it's going to do.
It calls fsync to make sure the journal is on the disk.
The disk says "oh yeah, it's there."
The file system then writes the actual data, and fsyncs that. The disk, again, tells it that it wrote the data out. The file system then marks that part of the journal as having been written.
However, the disk hasn't actually written out its cache yet. It lied to the OS / file system and said it had, but it hasn't, it's busy doing something else. Poof, the power goes out.
Now, the journal doesn't have our data, we've already cleared it out, and the file system, which is supposed to have been coherent because we fsynced it, is not, and it is now corrupted.
I have reproduced this behavior a dozen or more times on IDE based systems. The only way to stop it is to tell the drive to stop using it's write cache.
Now, SCSI drives don't lie about fsync. At least none of the ones I've tested have done that. so, when the file system asks the disk to fsync and reports back that it has fsynced, the data really is on the drive. And we can now roll forward the journal pointed and proceed with the knowledge our data is coherent on the drive.
You can prove this to yourself. Set up a server on IDE and another on SCSI. Install postgresql, running on a journaled file system like ext3, and then run the follow commands on each:
Setup:
pgbench -i -s 10
Test:
pgbench -c 100 -t 100000
Now, in the middle of the test on both machines, pull the plug.
When you restart the machines, the SCSI one will come right up, database coherent and no problem. the IDE one will fail, at least every so often, or as in my experience, nearly every time.
Nice article, thanks for the link.
It looks like Oracle has gotten the same basic results as the PostgreSQL Global Development Group have. JFS and ext3 are generally the fastest under a database, while XFS and Reiser seem to be pretty slow.
The odd thing here is that most other tests show XFS and Reiser as the kings. But the kind of random access databases do seems to be better handled by JFS / ext3.
The problems with your RAID consistency, were these file system problems, or RAID level problems? If they're RAID level problems that would seem to point at your RAID controller, as no file system should be able to bonk a RAID array on the head, since it doesn't really have that kind of access to it.
Thanks for the correction. By any chance was that ReiserFS partition on an IDE drive with write cache enabled? See some of my other posts in this thread for the problems IDE drives with enabled write cache present.
Hey, we're not talking about a better choice based on technology, we're talking about a moral victory...
If you are using IDE drives with the write cache enabled, then NO journaling file system is safe on your system. This is because IDE drives with write cache respond immediately to requests for fsync with true, whether they've written the data out or not.
If your data is important, either turn off the cache on your IDE drive or buy a SCSI drive, which won't lie about fsync. This is a problem for both linux and BSD.
Later model IDE drivers and drives may be able to work properly with cache enabled, but not now. There are ongoing discussions on BOTH kernel hackers lists, BSD and Linux, about what to do, and no solution in sight.
Sorry, but there are a few reasons these results are questionable. As one poster has already pointed out, ext3 by default journals both meta data (i.e. directory structure) AND data. The other three by default only journal meta data.
But more importantly than that is the fact that in linux, that IDE drive is responding immediately to fsync requests before it's actually done them. I.e. when the journaling file systems says "hey, did you get that stuff written out to the disks?" The drive says "yes sir!" Then writes it out whenever it gets a chance after that.
This means that you may well have a corrupted directory structure should the machine lose power in the middle of writing data, because the hard drive LIED to the OS, and the journaling file system flushed out it's journal when it wasn't actually written to the disk.
To get trustworthy results, you need to either 1: turn off the write cache on the IDE drive or 2: Use SCSI, which doesn't lie about such things.
As long as you're writing to an IDE drive with an enabled 8 meg cache, the results are worthless, because you don't actually have the reliability you think you have, and the numbers might change drastically when you start using drives (like SCSI) that actually fsync properly.
Sorry, but you're wrong on the need for a journaled file system. PostgreSQL recommends using one, and cannot guarantee recoverability without it.
While some older versions of ReiserFS from a couple years back or so had some issues with corruption, it would appear those problems have been fixed.
The actual CPU overhead of a good journaling file system is quite low.
But, no matter big the elephant that is IBM, the thousands of ant bites that is the FSF community is just as lethal.
And probably would make for better press.