Slashdot Mirror

Greenplum is another company that markets a commercial database based on PostgreSQL. I know EnterpriseDB contributes back to the core PostgreSQL project a lot, and I assume Greenplum does too, but as you say, neither "owns" PostgreSQL, which is developed in a pretty thoroughly community-based fashion.

I think some of you here are missing the point of the product. It is an appliance of sorts that is meant to compete with the likes of Netezza and Teradata, i.e. massively parallel, share nothing database architectures meant for datawarehousing. Typically, that means long running queries over gigabytes and terabytes of data.

Note the name of Exabyte. The prefix "exa" is greater than the prefix of "tera" in Teradata. They are trying to compete with Teradata.

Sure, it also claims to have good OLTP capability in addition to going for the OLAP market. But, this isn't a product for a web app database, high transaction volume (hotel reservations), or customer database. It is a product to mine all of the data generated by customers you interact with. All the data you generate when all the doubleclick, etc., tags fire on your websites.

The point of having several 1U, 2 processor machines is to make it like a grid. Again, this is for massively parallel database architectures. I am sure that the the storage servers serve the data to an Oracle RAC installation, thus the 1U 2 processor machines as RAC is meant to be run on lots of small machines.

On the note of massively parallel database architectures, check out Greenplum for a company using MySQL to attack this market.

-fragbait

I think some of you here are missing the point of the product. It is an appliance of sorts that is meant to compete with the likes of Netezza and Teradata, i.e. massively parallel, share nothing database architectures meant for datawarehousing. Typically, that means long running queries over gigabytes and terabytes of data.

Note the name of Exabyte. The prefix "exa" is greater than the prefix of "tera" in Teradata. They are trying to compete with Teradata.

Sure, it also claims to have good OLTP capability in addition to going for the OLAP market. But, this isn't a product for a web app database, high transaction volume (hotel reservations), or customer database. It is a product to mine all of the data generated by customers you interact with. All the data you generate when all the doubleclick, etc., tags fire on your websites.

The point of having several 1U, 2 processor machines is to make it like a grid. Again, this is for massively parallel database architectures. I am sure that the the storage servers serve the data to an Oracle RAC installation, thus the 1U 2 processor machines as RAC is meant to be run on lots of small machines.

On the note of massively parallel database architectures, check out Greenplum for a company using MySQL to attack this market.

-fragbait

I think some of you here are missing the point of the product. It is an appliance of sorts that is meant to compete with the likes of Netezza and Teradata, i.e. massively parallel, share nothing database architectures meant for datawarehousing. Typically, that means long running queries over gigabytes and terabytes of data.

Note the name of Exabyte. The prefix "exa" is greater than the prefix of "tera" in Teradata. They are trying to compete with Teradata.

Sure, it also claims to have good OLTP capability in addition to going for the OLAP market. But, this isn't a product for a web app database, high transaction volume (hotel reservations), or customer database. It is a product to mine all of the data generated by customers you interact with. All the data you generate when all the doubleclick, etc., tags fire on your websites.

The point of having several 1U, 2 processor machines is to make it like a grid. Again, this is for massively parallel database architectures. I am sure that the the storage servers serve the data to an Oracle RAC installation, thus the 1U 2 processor machines as RAC is meant to be run on lots of small machines.

On the note of massively parallel database architectures, check out Greenplum for a company using MySQL to attack this market.

-fragbait

The subject of this article, EnterpriseDB, is trying to target this market with GridSQL. As it's new in it's current form, impossible to say how reliable systems built with it will be quite yet. Those looking for reasons behind the IBM investment might consider whether GridSQL might one day talk to DB2 databases as well.

The closest fully open-source PostgreSQL solution to your requirements that's been around a bit is pgpool-II. It think it's still too immature to be considered five-nines quality though, and there are some restrictions you have to observe. A PostgreSQL replication solution that is very robust and proven is slony but it's not a load-balancing solution in the way I suspect you want.

There's also the Greenplum Database, which isn't free or open-source but is rooted in PostgreSQL technology.

Good enterprise-grade clustering with load-balancing is still on the PostgreSQL work in progress list rather than being here right now. I expect the core infrastructure piece needed to really make it work well (support for read-only warm-standby slaves) will make it into PostgreSQL 8.4 and be released around a year from now. I started a comparison page of the replication solutions currently available that's on the PostgreSQL wiki now that is trying to track progress in this area. Much like core PostgreSQL support for enabling replication, it still needs some work .

Yes, there are a few options.

Slony-I is a BSD licensed Master-Slave system; its successor, Slony-II, is still in development.

Various outfits like EnterpriseDB and Greenplum offer commercial Postgres derivatives with multi-master replication and clustering.

There are also open-source, less-popular options like PGPool and PGCluster.

Cheers,
-J

How about an article on that neat analysis framework Fermi (and others) use for the terabytes of data they generate and have to sift through?

http://root.cern.ch/ (large scale repository for data analysis)

And how would you use Postgres for something like that anyway? Maybe something like:

http://www.greenplum.com/ (biggy-sized Postgres based data warehousing)

they will likely migrate their existing SuSE and Red Hat installations to Unbreakable Linux.

Perhaps Oracle customers faced with this sour ultimatum might find it attractive to keep their current platform and try MySQL (more flexible, faster, lower costs) or Greenplum (100x as fast as Oracle) instead...

Things don't always pan out for the faltering Emperor.

Note you still don't have (and will probably never have) an equivalent to RAC
 
Why would open source never have this?
 
but that is something maybe a proprietary company can come in and offer?
 
Is this like RAC?
 
  No, Greenplum's MPP offers some share-nothing clustering capabilities for sections of a dataset. It also offers parallelism within queries hittind different nodes on a cluster. But this is different.

RAC offers true shared-everything clusters. THis means that you have shared block devices, shared processing servers, and the like. It is fairly complex and involves moving coordination out of the system itself and into some sort of distributed lock manager (usually part of a clustered filesystem like OCFS2).

Now, here is the problem. PostgreSQL is a very strong application. I don't think it is trivial to compete against them as a fork in the open source world, and the PostgreSQL community is not interested in doing what would be necessary for shared-everything clusters. THese things would make PostgreSQL less competitive in the majority of the markets that the community is in at the moment by adding unnecessary I/O overhead, and added complexity of the code would slow down development. There are some other sticky issues too (disk-based DLM's are going to be slower than in-memory semaphores, for example).

So, what you are left with is something which won't have the broad interest that would make the product less generally applicable. So it isn't going to happen.

I would compare the situation less with Green Plum (where there would be interest in their control node code) and more with EnterpriseDB. EnterpriseDB's big thing is Oracle compatibility. PostgreSQL doesn't aim to be compatible with Oracle, and so a lot of stuff like handling empty strings as NULL's (or is it the other way around) in Oracle will never be accepted into PostgreSQL. Hence EnterpriseDB has to maintain a set of changes to the parser to do those things that the PostgreSQL development community for good reason doesn't want.

Note you still don't have (and will probably never have) an equivalent to RAC

Why would open source never have this?

but that is something maybe a proprietary company can come in and offer?

Is this like RAC?

Right now DB2 viper is in beta, that allow you to use MDC, range partitioning, or hash partitioning. All in combination. This means that if you really wanted to scale up you could spread the data across a dozen two-way boxes with each one partitioning its local data. This isn't cheap to do (licensing db2 this way is expensive), but if you wanted to get adhoc queries against a 100 million row table returning results in 1-4 seconds - this is the way to do it. ;-)

Greenplum is doing something similar using PostgreSQL as a base:

http://www.greenplum.com/products/bizgresMpp.php

I have some data analysis I'm doing right now under PostgreSQL that I previously did under Oracle. Being able to have several parallel processes working on the same query in Oracle makes things go much faster. Oracle is really efficient at creating subsets of data. Most of what I need to do was previously done with lots of "create table as..." statements. Oracle's performance is much better than PostgreSQL for the volume of data I'm working with.

There are clustered versions of PostgreSQL out there with the same ability to split up a query across multiple CPUs and multiple machines. It is not free but take a look at Bizgres MPP that GreenPlum sells.

GreenPlum is contributing back significant improvements in other areas, keeping this specific parallelization technique in house.

You're right.

>So yeah, mysql and postgresql are neat products, but their lack of scalability for massive tables or analytical queries is a major gap.

Perhaps in a year or so these guys will have something for low-end data warehousing on PostgreSQL: http://www.greenplum.com/

...but some folks believe you've gotta pay money or the app isn't any good.

There are good reasons to pay someone for support, if the people you're paying know their stuff. If you're building enterprise level, mission critical data warehouses, you'll want immediate access to expert help when things go horribly wrong. And Sorbannes/Oxley reinforces that need.

For those seeking paid support, there are several companies working to do interesting things with Pg:

• GreenPlum also working to enhance Pg with the Bizgress project
• EnterpriseDB - working to make Pg interoperable w/ Oracle tools
• Netezza - MPP appliance h/w running a modded version of Pg

There are some other outfits dedicated to Pg support, but I can't recall the particulars...

Meanwhile, MySQL still seems to be having difficulty getting stored procs and real views released...5.0 is starting to make Longhorn's development schedule look like a quarterly maintenance release.

It's also interesting that TFA didn't mention the rise of alternatives ranging from SQLite (which pretty much does everything that folks used MySQL for in the first place, but wo/ any license confusion), to Firebird, to the recently open'd Ingres.