Slashdot Mirror

← Back to Stories (view on slashdot.org)

PostgreSQL 9.2 Out with Greatly Improved Scalability

Posted by Unknown on from the rev-your-engines dept.

The PostgreSQL project announced the release of PostgreSQL 9.2 today. The headliner: "With the addition of linear scalability to 64 cores, index-only scans and reductions in CPU power consumption, PostgreSQL 9.2 has significantly improved scalability and developer flexibility for the most demanding workloads. ... Up to 350,000 read queries per second (more than 4X faster) ... Index-only scans for data warehousing queries (2–20X faster) ... Up to 14,000 data writes per second (5X faster)" Additionally, there's now a JSON type (including the ability to retrieve row results in JSON directly from the database) ala the XML type (although lacking a broad set of utility functions). Minor, but probably a welcome relief to those who need them, 9.2 adds range restricted types. For the gory details, see the what's new page, or the full release notes.

36 of 146 comments (clear)

  1. Re:/. Poll by Eponymous+Hero · · Score: 4, Insightful

    E) stop using oracle and start using postgres

    --
    insensitive clod overlords obligatory xkcd car analogy russian reversals whoosh pedant fanbois ftfy in 3...2...1..PROFIT
  2. That's great and all, but . . . by Mitchell314 · · Score: 2, Funny

    When are they going to come out with the feature where it installs on OS X without requiring a human sacrifice? :P

    --
    I read TFA and all I got was this lousy cookie
    1. Re:That's great and all, but . . . by Anonymous Coward · · Score: 5, Informative

      9.3. Seriously.

      http://rhaas.blogspot.com/2012/06/absurd-shared-memory-limits.html

    2. Re:That's great and all, but . . . by Anonymous Coward · · Score: 3, Informative

      http://postgresapp.com/

    3. Re:That's great and all, but . . . by dragonk · · Score: 5, Informative

      I just posted this to the blog, but I will repeat it here --

      There is a very good reason we OS vendors do not ship with SysV default limits high enough to run a serious PostgreSQL database. There is very little software that uses SysV in any serious way other than PostgreSQL and there is a fixed overhead to increasing those limits. You end up wasting RAM for all the users who do not need the limits to be that high. That said, you are late to the party here, vendors have finally decided that the fixed overheads are low enough relative to modern RAM sizes that the defaults can be raised quite high, DragonFly BSD has shipped with greatly increased limits for a year or so and I believe FreeBSD also.

      There is a serious problem with this patch on BSD kernels. All of the BSD sysv implementations have a shm_use_phys optimization which forces the kernel to wire up memory pages used to back SysV segments. This increases performance by not requiring the allocation of pv entries for these pages and also reduces memory pressure. Most serious users of PostgreSQL on BSD platforms use this well-documented optimization. After switching to 9.3, large and well optimized Pg installations that previously ran well in memory will be forced into swap because of the pv entry overhead.

    4. Re:That's great and all, but . . . by cas2000 · · Score: 4, Funny

      you atheists love to take all the fun out of things, don't you?

      Eliminate the human sacrifice now and next you'll be saying we have to get rid of our Steve Jobs altars.

    5. Re:That's great and all, but . . . by colinrichardday · · Score: 2

      Get rid of your Steve Jobs altars!

    6. Re:That's great and all, but . . . by schmiddy · · Score: 4, Insightful

      There is a serious problem with this patch on BSD kernels. All of the BSD sysv implementations have a shm_use_phys optimization which forces the kernel to wire up memory pages used to back SysV segments. This increases performance by not requiring the allocation of pv entries for these pages and also reduces memory pressure. Most serious users of PostgreSQL on BSD platforms use this well-documented optimization. After switching to 9.3, large and well optimized Pg installations that previously ran well in memory will be forced into swap because of the pv entry overhead.

      I don't see your comment on the blog (maybe it has to be approved?), but the same issue was raised here during review of the patch. The concern was mostly blown off (most PG developers use Linux instead of BSD, that might well be part of it), but if you had some numbers to back up your post, the -hackers list would definitely be interested. Ideally, you could give numbers and a repeatable benchmark showing a deterioration of 9.3-post-patch vs. 9.3-pre-patch on a BSD. If that's too much work, just the numbers from a dumb C program reading/writing shared memory with mmap() vs. SysV would be a good discussion basis.

      --
      http://cltracker.net -- powerful craigslist multi-city search
    7. Re:That's great and all, but . . . by gazbo · · Score: 3, Informative

      Each client connected to the DB has its own child process - the shared memory is a buffer that is shared across postgresql child PIDs with the same parent. That's why the proposed patch would work using an anonymous shared memory segment - because the memory is only passed to children of the same process.

    8. Re:That's great and all, but . . . by gazbo · · Score: 3, Informative
      Well...arguably. This is the exact same argument as Apache vs Nginx, where Apache spawns a child process per client, whereas Nginx has a limited number of worker processes that handle a queue of requests as they become free. Nginx definitely has an advantage in terms of RAM when servicing thousands of (truly) simultaneous requests.

      While Postgresql does use the Apache model, there is middleware available (google 'pgpool' for an example) that amongst other things will queue requests so they can be serviced by a limited number of children. Of course this only matters if there are an awful lot of simultaneous queries (without the corresponding amount of server RAM).

      However; your claim about threads per CPU is oversimplified, and especially wrong with a DB server where processes will most likely be IO bound. With 1 core, for example, there is nothing wrong with having 5 processes parsing and planning a query for a few microseconds, while the 6th is monopolising IO actually retrieving query results. Or the reverse - having 1 CPU-bound process occasionally being interrupted to service 5 IO bound processes, which would negligibly impact the CPU-bound query, while hugely improving latency on the IO bound queries.

    9. Re:That's great and all, but . . . by m.dillon · · Score: 3, Informative

      I don't think this is true any more. Threads are light weight... that's the whole point. They all share the same pmap (same hardware page table). Switching overhead is very low compared to switching between processes.

      The primary benefit of the thread is to allow synchronous operations to be synchronous and not force the programmer to use async operations. Secondarily, people often don't realize that async operations can actually be MORE COSTLY, because it generally means that some other thread, typically a kernel thread, is involved. Async operations do not reduce thread switches, they actually can increase thread switches, particularly when the data in question is already present in system caches and wouldn't block the I/O operation anyway.

      There is no real need to match the number of threads to the number of cpus when the threads are used to support a synchronous programming abstraction. There's no benefit from doing so. For scalability purposes you don't want to create millions of threads (of course), but several hundred or even a thousand just isn't that big a deal.

      In DragonFly (and in most modern unix's) the overhead of a thread is sizeof(struct lwp) = 576 bytes of kernel space, +16K kernel stack, +16K user stack. Everything else is shared. So a thousand threads has maybe ~40MB or so of overhead on a machine that is likely to have 16GB of ram or more. There is absolutely no reason to try to reduce the thread count to the number of cpu cores.

      --

      There are two reasons for using lock memory for a database cache. The biggest and most important is that the database will be accessing the memory while holding locks and the last thing you want to have happen is for a thread to stall on a VM fault paging something in from swap. This is also why a database wants to manage its own cache and NOT mmap() files shared... because it is difficult, even with mincore(), to work out whether the memory accesses will stall or not. You just don't want to be holding locks during these sorts of stalls, it messes up performance across the board on a SMP system.

      Anonymous memory mmap()'s can be mlock()'d, but as I already said, on BSD systems you have the pv_entry overhead which matters a hell of a lot when 60+ forked database server processes are all trying to map a huge amount of shared memory.

      Having a huge cache IS important. It's the primary mechanism by which a database, including postgres, is able to perform well. Not just to fit the hot dataset but also to manage what might stall and what might not stall.

      In terms of being I/O bound, which was another comment someone made here... that is only true in some cases. You will not necessarily be I/O bound even if your hot data exceeds available main memory if you happen to have a SSD (or several) between memory and the hard drive array. Command overhead to a SSD clocks in at around 18uS (verses 4-8mS for a random disk access). SSD caching layers change the equation completely. So now instead of being I/O bound at your ram limit, you have to go all the way past your SSD storage limit before you truly become I/O bound. A small server example of this would be a machine w/16G of ram and a 256G SSD. Whereas without the SSD you can become I/O bound once your hot set exceeds 16G, with the SSD you have to exceed 256G before you truly become I/O bound. SSDs can essentially be thought of as another layer of cache.

      -Matt

  3. Range data types by slack_justyb · · Score: 5, Interesting

    I think everyone has glossed over the single most important feature in the Postgre SQL that they have refined in this release, IMHO. Ranged data types. Let's say you have a meeting schedule DB application. Well currently if you want to restrict a room between two times (start and stop) so that no one else can have the room during that time, you are going to have to write that logic in your application.

    Postgre's range data type allows you to create unique checks on ranges of time. This can in two lines of code, do every single logic check that is needed to ensure no two people schedule the same room at the same time.

    How this is not showing up on anyone's radar is beyond me, or maybe we all just use Outlook or Google Calendar now. However, the range types are not just limited to the application of time, but of anything that requires uniqueness along a linear fashion, as opposed to just checking to see if any other record matches the one that you are trying to insert.

    1. Re:Range data types by nebosuke · · Score: 2

      It seems that you're misunderstanding the definition of a range datatype in this context. The data type of the column in the scheduling example would be defined as a timestamp range, and the constraint on the column would be that no timestamp range value can overlap with any other timestamp value in the table (or in the table for any rows that share a key value, such as user ID). There is no need to alter the column definition to accomodate changes to scheduling data.

    2. Re:Range data types by nebosuke · · Score: 2

      Strictly speaking, at minimum it would require two time fields and two boolean fields, with each boolean field specifying whether or not the interval is inclusive of each corresponding end point. It would also require a lot more than one simple constraint to get the desired behavior provided by the new datatype--and the whole mess would need to be repeated for every single interval with a simple exclusivity constraint. The new range datatype also makes it relatively simple to, e.g., specify a non-zero overlap constraint, which is significantly harder without it.

    3. Re:Range data types by Kergan · · Score: 3, Informative

      Oh, it's simple enough to do with two separate fields and a check constraint. That's how you'd do it i other DB engines, in fact.

      Ensuring there are no overlaps is an entirely different story, however: queries against those two fields cannot make any reasonable use of an index. The ranged type, by contrast, allows you to query the data using a nearest neighbour search and a GiST index.

      Think of a GiST index as indexing the smallest boxes that enclose your shapes of interest. When queried, the DB scans for boxes that overlap your box of interest, and discards rows that don't match the data's actual shape.

    4. Re:Range data types by DragonWriter · · Score: 2

      So you want to do that in the DB now. Will you have to change column definition to change that range?

      No, the range is data, not part of the column definition, I would say "RTFA", but to be fair the link was mislabelled in TFS as being about "range-restricted types", rather than range types.

      But here's the docs on range types. The scheduling use case is the basic example of exclusion constraints on range types (Sect 8.17.10 in the linked doc.)

  4. Postgres-Curious by kwalker · · Score: 4, Interesting

    TL;DR: Is there an advanced PostgreSQL for MySQL Users guide out there somewhere? Something more than basic command-line equivalents? And preferably from the last two major releases of the software?

    Long version
    I've been using MySQL personally and professionally for a number of years now. I have setup read-only slaves, reporting servers, multi-master replication, converted between database types, setup hot backups (Regardless of database engine), recovered crashed databases, and I generally know most of the tricks. However I'm not happy with the rumors I'm hearing about Oracle's handling of the software since their acquisition of MySQL's grandparent company, and I'm open to something else if it's more flexible, powerful, and/or efficient.

    I've always heard glowing, wonderful things online about PostgreSQL, but I know no one who knows anything about it, let alone advanced tricks like replication, performance tuning, or showing all the live database connections and operations at the current time. So for any Postgres fans on Slashdot, is there such a thing as a guide to PostgreSQL for MySQL admins, especially with advanced topics like replication, tuning, monitoring, and profiling?

    --
    ... And so it comes to this.
    1. Re:Postgres-Curious by Art3x · · Score: 4, Informative

      PostgreSQL replication is new (revision 9.1) so there may be little out there (Yes, there was replication, but with additional software, like Slony).

      I'm in the weird position of having used PostgreSQL mainly --- for seven years, writing dozens of applications --- but never MySQL. I've also used --- out of necessity only --- Microsoft SQL, Oracle, and Ingres, and PostgreSQL is much better. Just from a programming point of view, the syntax is, in my mind, simpler yet more powerful --- more ANSI-SQL-compliant, too, I've heard.

      Anyway, the point is, I've never used anything I like more. I adore PostgreSQL. It's so powerful. So many useful datatypes, functions, syntax. Not to mention it's ACIDity.

      To your question, though --- are there any good books to help a MySQLite move to PostgreSQL? Not that I've come across. But then again, I haven't found any good PostgreSQL books --- or even, for that matter, very well-written SQL books, period. They all are stupefyingly boring --- but I got what I could out of them.

      Actually, PostgreSQL's documentation is not that bad. In particular, try sections I, II, V, VI, and III, in that order. Skip anything that bores you at first. You can always come back. Honestly, there can't be that much of a learning curve for you, coming from MySQL.

    2. Re:Postgres-Curious by poet · · Score: 2

      9.0 was the first version with replication, not 9.1 and we have had things like warm standby since 8.1.

      --
      Get your PostgreSQL here: http://www.commandprompt.com/
    3. Re:Postgres-Curious by gullevek · · Score: 2

      There are two PostgreSQL books I used a lot in the past: PostgreSQL 9.0 High Performance by Gregory Smith (Packt) and PostgreSQL Second Edition by Douglas Douglas (O'Reilly).

      There is an extended list of books listed on the PostgreSQL homepage: http://www.postgresql.org/docs/books/

      Problem with all books is, they get outdated too quickly. While a lot of the basic info is still true for the books above, the O'Reilly book is very much based on 8.4 with is pretty ancient already. Perhaps getting an ebook is less a waste of paper.

      --
      "Freiheit ist immer auch die Freiheit des Andersdenkenden" - Rosa Luxemburg, 1871 - 1919
    4. Re:Postgres-Curious by rycamor · · Score: 4, Informative

      Unfortunately, I haven't found a really good guide of the type you are looking for. I can give you my experiences, going from MySQL to PostgreSQL, back to MySQL to support it at a large company, and then back to PostgreSQL. Generally, these days there is really *nothing* that I can find about MySQL that can't be done better in PostgreSQL. I mean it. At least for awhile MySQL could boast of native replication, but Postgres biw has that and it is arguably much more robust than MySQL's solution (had the misfortune to support MySQL replication for 2 years). Ditto with full-text indexing, and just about any other MySQL feature.

      Main differences:

      1. PostgreSQL is much more "correct" in how it handles data and has very little (essentially no) unpredictable or showstoppingly odd behavior of the sort you find in MySQL all the time. Your main problem in migrating an app to PostgreSQL will be all those corner cases that MySQL just "accepts" when it really shouldn't, such as entering '0000-00-00' into a date field, or allowing every month to have days 0-31. In other words, PostgreSQL forces you to be a lot more careful with your data. Annoying, perhaps, if you are developing a non-mission-critical system like a web CMS or some such, but absolutely a lifesaver if you deal with data where large numbers of dollars and cents (or lives) depend on correct handling.

      MySQL has provided for a fair amount of cleanup for those who enable ANSI standard behavior, but it is still nowhere close to PostgreSQL's level of data integrity enforcement.

      2. MySQL has different table types, each of which support different features. For example, you cannot have full-text indexing in InnoDB (transactional) tables. PostgreSQL has complete internal consistency in this regard.

      3. MySQL has an almost entirely useless error log. PostgreSQL's can be ratcheted up to an excruciating level of detail, depending on what you want to troubleshoot. Ditto with error messages themselves.

      4. MANY MANY more choices in datatypes and functions to manipulate them. Definitely a higher learning curve, but worth it for expressive capability.

      5. Don't get me started on performance. Yes, if you have a few flat tables, MySQL will be faster. Once you start doing anything complicated, you are in for a world of pain. Did you know that MySQL re-compiles every stored procedure in a database on every new connection? PHP websites with per-page-load connections can really suffer.

      6. Don't get the idea that PostgreSQL is more complex to work with. If you want simple, you can stick with the simple parts, but if you want to delve into complex database designs and methodologies, PostgreSQL pretty much opens up the world to you.

      - Glad to be back in the PostgreSQL world...

    5. Re:Postgres-Curious by fuzzytv · · Score: 3, Informative

      Well, recommending a PL/SQL book as a source for learning SQL is a bit silly IMHO. Moreover, I find the books from Oracle rather bad - there are better sources to learn PL/SQL (e.g. the one from Feuerstein is a much better book).

      And in fact there's a great book about administering PostgreSQL from Hannu Krosing - it's called "PostgreSQL 9 Admin Cookbook" [http://www.packtpub.com/postgresql-9-admin-cookbook/book]. It's a great set of recipes for admins for common tasks, not an exhaustive documentation (that's what http://www.postgresql.org/docs/9.1/interactive/index.html is for), but if you want to learn how real pros admin the database, this is the right choice. And yes, I'd recommend it to newbies coming from MySQL.

      It might seem that the PostgreSQL community considered MySQL to be a toy database in the past, but it definitely was not a generally shared view. And this definitely changed recently - there's no reason not to join the community mailing lists / IRC channel and start a post with "I'm using a MySQL right now and I don't understand why PostgreSQL does SOMETHING."

  5. Re:LOL by Tough+Love · · Score: 2, Insightful

    Because we love to bash our keyboards into so much plastic scrap whenever we come across one of its many standards-defiant idiosyncracies?

    You mean, idiosyncracies different from Oracle's idiosyncracies, Microsoft's idiosyncracies and IBM's idiosyncracies?

    By the way, care to be specific? Oh yeah, posting anon. Right.

    --
    When all you have is a hammer, every problem starts to look like a thumb.
  6. MariaDB and Percona by kbahey · · Score: 3, Insightful

    Oracle is not that big a of concern.

    There is MariaDB which is data-compatible with MySQL, and has some nice additions (like microsecond performance data), and there is also Percona Server.

    If Oracle messes up, like they did with OpenOffice, there will be another version that they cannot touch, like LibreOffice.

    --
    2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.
  7. Re:meh by Anonymous Coward · · Score: 2, Insightful

    Wake me when MemSQL supports data-warehousing.

  8. How PostgreSQL stacks up to Oracle ? by Taco+Cowboy · · Score: 3, Interesting

    I've been searching for a comparison chart of various SQLs but all I can find are very very old articles

    There's a database project that I'm working on and I'm choosing which SQL to be employed

    MySQL is obviously not up to par

    I don't know how good PostgreSQL is - so, is there a comparison chart or something that can facilitate us, the one who are going to make purchasing decision, to make one choice over the other?

    Thank you !

    --
    Muchas Gracias, Señor Edward Snowden !
    1. Re:How PostgreSQL stacks up to Oracle ? by rycamor · · Score: 5, Informative

      Generally there is very little in the sense of logical data manipulation capabilities in which Oracle exceeds PostgreSQL (usually the opposite, actually). The main advantage Oracle has is in the extreme high end of scalability and replication, and that benefit is offset by massive complexity in setup and configuration. Even there, PostgreSQL is closing fast these days, with built-in streaming replication, table partitioning, and all sorts of high-end goodies.

      I do all sorts of PostgreSQL consulting, and you would be surprised at the number of large companies and government organizations considering migration from Oracle to PostgreSQL.

      And if you *really* need PostgreSQL to go into high gear, just pay for the commercial Postgres Plus Advanced Server from EnterpriseDB and you will get a few heavy-duty add-ons, including an Oracle compatiblity layer.

      Also, IMHO one of the really cool things about PostgreSQL is the number of very geeky tools it puts at your disposal, such as a rich library of datatypes and additional features, along with the ability to create your own user-defined datatypes.

    2. Re:How PostgreSQL stacks up to Oracle ? by F.Ultra · · Score: 2

      Why is MySQL _obviously_ not up to par? Yes I'm really curios.

    3. Re:How PostgreSQL stacks up to Oracle ? by serviscope_minor · · Score: 3, Informative

      and you would be surprised at the number of large companies and government organizations considering migration from Oracle to PostgreSQL.

      Not really.

      I've had no experience with the database end of things, but I've been on the receiving end of some other Oracle "products" at two places I've been. Once you've been Oracled, there is a strong incentive never to go anywhere near them again, no matter how they look on paper.

      When it comes for utter distain and hatred for their customers, Oracle make Sony look like rank ametures.

      As far as Oracle are concerned, the customer is a fool whose sole purpose is to be screwed over for as much cash as possible.

      --
      SJW n. One who posts facts.
    4. Re:How PostgreSQL stacks up to Oracle ? by Lennie · · Score: 2

      MySQL has some nice replication built in I believe, I've never used them.

      Other than that, I would thread lightly with MySQL:

      "Why not MySQL"

      http://www.youtube.com/watch?v=1PoFIohBSM4

      --
      New things are always on the horizon
    5. Re:How PostgreSQL stacks up to Oracle ? by Lennie · · Score: 2

      Lots of people such, but it is just hard to trust your data to MySQL. Just a moment ago I posted a link above to this video which illustrates it:

      http://www.youtube.com/watch?v=1PoFIohBSM4

      --
      New things are always on the horizon
  9. JSON by Art3x · · Score: 2

    To me, JSON very interesting. I don't know how exactly I'll use it, but it combines all that's great about PostgreSQL with some of what was interesting about CouchDB and other projects like it.

    Mainly, one-to-many relationships may be easier. Usually, they are two separate select statements. For example, one to get the article, another to get the comments. Then you patch it all together in PHP, or whatever middle language you're using. With JSON support, that could be a single SELECT, crammed up in JSON, which you then uncram with a single json_decode function call in PHP, which would yield nice nested arrays.

  10. While Postgres is good for many things... by FlyingGuy · · Score: 2

    Until the fix the TX number issue ( the infamous rollover ) then they are pretty much out of the running in DB's that have VERY high insert levels since the vacuum process cannot hope to keep up with tables that have 100's of millions of rows.

    I am an Oracle professional but I do keep track of Postgres and like it, but the 32 bit TX t is a bit of an Achilles heel.

    --
    Hey KID! Yeah you, get the fuck off my lawn!
    1. Re:While Postgres is good for many things... by Anonymous Coward · · Score: 2

      you can't vacuum your table every 2 billion transactions? did you know autovacuum exists?

      There is no table with "100's of millions of rows" that can't be vacuumed every 2 BILLION transactions.

      http://www.postgresql.org/docs/current/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND

    2. Re:While Postgres is good for many things... by rtaylor · · Score: 2

      I don't see why not.

      You had the IO to create those 17 trillion tuples in the first place; so vacuum will use that same IO capacity to maintain it.

      The low billions of tuples isn't much of an issue despite being on spinning disk with very little in memory.

      --
      Rod Taylor
  11. Range types -- not range-restricted -- are major by DragonWriter · · Score: 2

    Minor, but probably a welcome relief to those who need them, 9.1 adds range restricted types.

    First, its 9.2, not 9.1.

    Second, (as shown in the link) these are range types, not range-restricted types. Range-restricted types (as known from, e.g., Ada) are something that (via domains with check constraints) PostgreSQL has supported for a very long time.

    Range types, combined with 9.2s support for exclusion constraints, are a pretty major new feature that give 9.2 a great facility in dealing with (among other things) temporal data and enforcing common logical constraints on such data in the database as simple-to-express constraints rather than through triggers.