Slashdot Mirror

← Back to Stories (view on slashdot.org)

Web Analytics Databases Get Even Larger

Posted by CmdrTaco on from the who-watches-the-oh-never-mind dept.

CurtMonash writes "Web analytics databases are getting even larger. eBay now has a 6 1/2 petabyte warehouse running on Greenplum — user data — to go with its more established 2 1/2 petabyte Teradata system. Between the two databases, the metrics are enormous — 17 trillion rows, 150 billion new rows per day, millions of queries per day, and so on. Meanwhile, Facebook has 2 1/2 petabytes managed by Hadoop, not running on a conventional DBMS at all, Yahoo has over a petabyte (on a homegrown system), and Fox/MySpace has two different multi-hundred terabyte systems (Greenplum and Aster Data nCluster). eBay and Fox are the two Greenplum customers I wrote in about last August, when they both seemed to be headed to the petabyte range in a hurry. These are basically all web log/clickstream databases, except that network event data is even more voluminous than the pure clickstream stuff."

3 of 62 comments (clear)

  1. Recursive queries too by coryking · · Score: 3, Interesting

    These little puppies, i.e. recursive queries, look pretty cool too. Sounds like a good tool for threaded comment systems or finding related items in a table:


    Recursive queries are typically used to deal with hierarchical or tree-structured data. A useful example is this query to find all the direct and indirect sub-parts of a product, given only a table that shows immediate inclusions:

    WITH RECURSIVE included_parts(sub_part, part, quantity) AS (
            SELECT sub_part, part, quantity FROM parts WHERE part = 'our_product'
        UNION ALL
            SELECT p.sub_part, p.part, p.quantity
            FROM included_parts pr, parts p
            WHERE p.part = pr.sub_part
        )
    SELECT sub_part, SUM(quantity) as total_quantity
    FROM included_parts
    GROUP BY sub_part

    ... It will take a while to wrap my brain around this new concept though. That doesn't look like a normal query I'm used to reading!

    They'll get replication some day soon. But there is a lot of cool, very useful stuff with every new release. I usually feel like kid in a candy store wondering what's new that I can exploit.

  2. Re:Another win for PostgreSQL... by TooMuchToDo · · Score: 2, Interesting

    I have to say, I love postgresql. We use it to store hundreds of gigabytes of metadata for our 17 petabyte disk/tape storage system at my day gig.

  3. Google? by wiedzmin · · Score: 2, Interesting

    Who cares about eBay and MySpace... tell me about the major players! What is Google running?

    --
    Bow before me, for I am root.