Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Biggest MySQL Cluster, Ever?

Posted by Cliff on from the an-RDBMS-that-needs-a-diet dept.

ExcerLee asks: "Our team is going to build a MySQL (load-balancing and fail-over) cluster for the Cluster World Expo in the end of June. This cluster is going to have at least 10 nodes, and will use dual-opteron systems from Polywell Computers and SuSE Enterprise Linux for AMD64 from SuSE. While we are working on this hefty cluster, we wonder if this is the biggest MySQL cluster has ever been built. I googled with terms like 'biggest mysql cluster', but didn't find much." If you've run a MySQL cluster before, how large was it and how well did it perform? Krow: I have been told of much larger clusters then this; Slashdot DBs total 6 machines with 18 processors, and LiveJournal has one of the most complicated clusters I have ever seen set up.

55 comments

  1. Why would they think that? by Hanashi · · Score: 4, Insightful

    IT environments are so complex today, why would anyone think only 10 of anything is "the biggest"?

    --
    Check out my eclectic infosec blog at InfoSecPotpou
    1. Re:Why would they think that? by moosesocks · · Score: 2, Informative

      10 of these ought to be more powerful than anything else that's been built. 1060 processors, 5760gb of RAM.

      It would definitely be the world's largest MySQL cluster. Of course, I dobut that anybody would even WANT a cluster that big. Once you get a database that size, you would probably want to custom-tailor your own solution, or not use a traditional database at all. The only databases big enough to warrant this type of power are probably those of the government (ie. the IRS); that being said, the government is probably using an ancient antiquated system which works just fine for their purpose.

      --
      -- If you try to fail and succeed, which have you done? - Uli's moose
    2. Re:Why would they think that? by afidel · · Score: 1

      Actually that sounds kind of like what Walmart would want for their datamining database =) 5.7TB of ram sure would make those massive joins faster. BTW for stuff of that size I believe most people are using Terrdata's database, I know the 3 largest live databases in the world including Walmarts are all hosted using Terrdata.

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    3. Re:Why would they think that? by sql*kitten · · Score: 1

      10 of these ought to be more powerful than anything else that's been built. 1060 processors, 5760gb of RAM

      You're not even close. This cluster has 48 nodes with 128 processors apiece, but each node can take up to 512. Each node also has up to a terabyte of RAM.

    4. Re:Why would they think that? by ExcerLee · · Score: 1

      Because MySql itself doesn't support real clustering and there is any solutions that supports real cluster for MySql yet. Master/Slave data replication should not be considered as clustering. The solution we are working on supports simultaneous replication for data writes. I haven't seen anybody else had this done before.

    5. Re:Why would they think that? by Hanashi · · Score: 1

      I see. In that case, I guess the answer to your question depends on the definition of "cluster". Regardless of the definition, though, your work sounds very useful. It's even cool, which is better. 8-)

      --
      Check out my eclectic infosec blog at InfoSecPotpou
  2. Wow! Correct use of the Cliff Comma Subject! by JUSTONEMORELATTE · · Score: 4, Funny

    Mark this day! Cliff's trademark comma actually landed in an appropriate place!
    Assuming, of course, that he was trying to sound like Fat Comic Book Guy from The Simpsons.

    --

  3. One Question.... by clonebarkins · · Score: 3, Funny

    ...what are you going to put in the db?

    --

    "The evil of the world is made possible by nothing but the sanction you give it." -- Ayn Rand

    1. Re:One Question.... by PerlGuru · · Score: 3, Funny

      *everything*

    2. Re:One Question.... by clonebarkins · · Score: 3, Funny
      *everything*

      You forgot the evil "Mwahahahahahaha" after it!

      --

      "The evil of the world is made possible by nothing but the sanction you give it." -- Ayn Rand

  4. BEOWULF by Anonymous Coward · · Score: 2, Funny

    Imagine a BEOWULF cluster of these CLUSTERS.
    Someone had to say that stupid shit. :-(

    1. Re:BEOWULF by ciaran_o_riordan · · Score: 5, Interesting

      I know you were only joking but I thought I'd mention that this is impossible.

      BEOWULF support has to be compiled into an application for it to be used. MySQL doesn't have BEOWULF support, and I don't see a patch for it on the web.

      OpenMosix performs kernel level clustering. You apply the OpenMosix patch to Linux, compile, reboot, and Linux will now migrate processes automatically to other OpenMosix enabled computers on your network.

      Beowulf is more efficient, but it would probably take a month or so to program MySQL to use it. OpenMosix takes about 30 minutes to set up and it then works with every application you run.

      I thought this was interesting.

      Ciaran O'Riordan

      --
      Expert in software patents or patent law? Contribute to the ESP wiki!
    2. Re:BEOWULF by The+Bungi · · Score: 2, Funny
      You mean something called 'Beowulf' that has something to do with clusters actually exists??

      All these years...

    3. Re:BEOWULF by h3 · · Score: 3, Interesting

      OpenMosix takes about 30 minutes to set up and it then works with every application you run.

      Well, not *every* application. Some don't migrate, depending on I/O, threading, etc. But a lot do, and it really is easy to setup.

      We use it at work on a bioinformatics cluster and the best part is that it'll transparently balance user jobs without them having to do or learn anything new.

      Ob topic: we happen to run MySQL on one of our cluster nodes, but it doesn't migrate ;).

      -h3

    4. Re:BEOWULF by Anonymous Coward · · Score: 0

      you're an idiot. beowulf doesn't require any patches, dumbshit. it's just the design which allows multiple machines to work off an NFS root. PVM, LAM/MPI, etc are not required to have a "Beowulf".

    5. Re:BEOWULF by steveit_is · · Score: 1

      OpenMosix will not migrate any process which uses shared memory unless you use the (sorry MAASK)unstable 'migshm' patch from http://mcaserta.com/maask/. MySQL uses shared memory. I tried it.

    6. Re:BEOWULF by stonecypher · · Score: 1

      > I thought this was interesting

      Score: (5, Interesting)

      These are not the droids you are looking for.

      (Pity I posted tihs 8 days late, huh?)

      --
      StoneCypher is Full of BS
  5. Documentation by wixu · · Score: 4, Insightful

    Are you planing to write some documentation of this work and showing it to internet?

  6. Why powerful servers? by samjam · · Score: 2, Interesting

    It would say more if you used less powerful machines and still did well.

    If you show how well it can run on stock hardware (as opposed to buying more) it may impress more people.

    That means 800MHz to 1GHz maybe?

    Just a thought.

    Sam

    --
    blog.sam.liddicott.com
  7. comic book guy sez: by Tumbleweed · · Score: 1

    "Biggest...cluster...ever."

    "There is no emoticon adequate to express what I'm feeling right now."

    1. Re:comic book guy sez: by Bingo+Foo · · Score: 1

      You mean "evar."

      --
      taken! (by Davidleeroth) Thanks Bingo Foo!
  8. livejournal? by orangesquid · · Score: 1

    I know LJ has a very complex system, but is it documented somewhere? I know there are several database clusters, and lots of cache servers, and several webservers and such...

    Who can tell me how it all cums together?

    --
    --TheOrangeSquid Is it any wonder things seem so awry? We swim in a sea of confusion and don't have to think to survive
    1. Re:livejournal? by aeinome · · Score: 5, Informative

      Here's an XML file detailing all of the clusters. It's a little hard to read, but it works.

      --
      When you don't have a leg to stand on, don't even get up.
    2. Re:livejournal? by Phil+Ulrich · · Score: 1

      And yet, with all of that equipment, it's amazing how dog slow LJ really still is. Just goes to show, nothing - absolutely nothing - will keep up with intrepid, well-meaning, angsty teenage goths who want to whine about their lack of a sex life and make friend with other angsty teenage goths.

      --
      Prepare to be burninated!
    3. Re:livejournal? by semaj · · Score: 1

      There's also an article floating around written by Brad Fitzpatrick (LiveJournal creator) a while ago now.

      If a journal site was using 6 databases in 2001, forgive me if I'm not impressed by the "Cluster World Expo" using four more, two years later.

      --
      Meep meep
    4. Re:livejournal? by Anonymous Coward · · Score: 0

      I guess my lack of membership in that demographic would explain why I wouldn't know so much about LiveJournal. Conversely, would your membership in that demographic seem to explain your intimate knowledge of its details?

  9. Wow! Amazing! by FroMan · · Score: 4, Funny

    That is truly amazing. Oh, not the cluster, that doesn't sound so impressive I guess. I really don't know, I don't work with clustered databases.

    The truly amazing part is that the postgresql folks have not arrived yet.

    --
    Norris/Palin 2012
    Fact: We deserve leaders who can kick your ass and field dress your carcass.
    1. Re:Wow! Amazing! by cperciva · · Score: 4, Funny

      The truly amazing part is that the postgresql folks have not arrived yet.

      Not at all. Postgresql is a database. This story has absolutely no connection to databases.

      --
      Tarsnap: Online backups for the truly paranoid
    2. Re:Wow! Amazing! by Sxooter · · Score: 1

      Plus, you wouldn't cluster postgresql, you'd just run it on a Sun E15k.

      --

      --- It is not the things we do which we regret the most, but the things which we don't do.
  10. Cliff's commas by A+nonymous+Coward · · Score: 1

    See here

    --
    Infuriate left and right
  11. They didn't [Was: Re:Why would they think that?] by Mr.+Sketch · · Score: 1

    IT environments are so complex today, why would anyone think only 10 of anything is "the biggest"?

    The submitter never claimed that their cluster was the biggest, but while they were building a good sized cluster, they were just curious what the biggest one ever built was.

    --
    Things you think are in the Constitution, but are not.
  12. Re:They didn't [Was: Re:Why would they think that? by Hanashi · · Score: 2, Informative
    I'm sorry, but I don't think you are correct. See the following excerpt:

    While we are working on this hefty cluster, we wonder if this is the biggest MySQL cluster has ever been built.

    They're clearly asking if theirs is the biggest ever built.

    --
    Check out my eclectic infosec blog at InfoSecPotpou
  13. mobile.de by willamowius · · Score: 2, Interesting

    mobile.de (German car marketplace) has over 100 MySQL database servers in their cluster and is performing very well.

  14. Re:Wow! Correct use of the Cliff Comma Subject! by babbage · · Score: 1

    Thus proving that even the blind can hit an occasional bullseye -- it's not accuracy, it's random luck ;-)

    --
    DO NOT LEAVE IT IS NOT REAL
  15. I'll bet the biggest is Google... by ubiquitin · · Score: 2, Informative

    Google has clusters of 100+ CPU's, and they run MySQL. I haven't seen any public confirmation that they are clustering MySQL, but it would make a lot of sense for them to do so. They would be smart to use a relatively flat db to store all incoming search requests for analysis.

    --
    http://tinyurl.com/4ny52
    1. Re:I'll bet the biggest is Google... by satterth · · Score: 1
      Do they run MySQL or Oracle?

      I can't find any contrete info.

      --
      Being called a dork on Slashdot must be like being called the retard in special ed.
    2. Re:I'll bet the biggest is Google... by bobbozzo · · Score: 1
      Do they run MySQL or Oracle?

      Well, apparently both, according to the same page you linked.

      The Oracle dev position is for corporate financials, whereas the MySQL is for Technical Web Applications, whatever that means (maybe data mining).

      They also have another position with MySQL experience mentioned on the same page.

      --
      Nothing to see here; Move along.
    3. Re:I'll bet the biggest is Google... by philj · · Score: 4, Informative

      How google works. Enjoy.

    4. Re:I'll bet the biggest is Google... by satterth · · Score: 1

      Yeah, i guess two jobs that require Oracle experience and two jobs that require MySQL experience doesn't really say much to what database they are using for their google data. It still could be just about anything though.

      --
      Being called a dork on Slashdot must be like being called the retard in special ed.
    5. Re:I'll bet the biggest is Google... by CodeArt · · Score: 1

      Great! I want more articles about Google setup!

  16. In Soviet Russia by buckinm · · Score: 1

    Beowulf imagines a cluster of YOU!

    --
    This isn't any ordinary darkness. It's advanced darkness.
  17. How does the load balancing work? by tcopeland · · Score: 1

    Is there one master that handles all the writes and replicates them out to slaves? And only the reads get distributed? Or is there some sort of n-way replication going on there?

    Tom

    --
    The Army reading list
    1. Re:How does the load balancing work? by teemu.s · · Score: 1

      I think one master who replicates the written records wont make sense - at least from the high-availability-point-of-view. If the master dies the whole database cluster would be useless or would need manual interaction (- to promote a new cluster master).

      multimaster replication would solve that issue ..

    2. Re:How does the load balancing work? by tcopeland · · Score: 1

      > If the master dies the whole database
      > cluster would be useless

      Yup, that sounds reasonable.... does anyone know how the replication mentioned here works?

      Tom

      --
      The Army reading list
  18. Wow, all those CPUs! by Moonwick · · Score: 1

    So, um, the important question... what kind of IO are you putting into these nodes?

    CPUs are great and all, but if you're sucking data through a soda straw (i.e., anything less than a good hardware RAID) those CPUs aren't going to have much to do...

    --
    Only on slashdot can a posting be rated "Score -1, Insightful".
    1. Re:Wow, all those CPUs! by budgenator · · Score: 1

      Our team is going to build a MySQL (load-balancing and fail-over) cluster for the Cluster World Expo in the end of June.

      I think what they are doing is more of an look we got all the pieces here. Maybe what he realy wants to know is, is this the biggest ever set up as a convention demo, not the biggest production system.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
  19. Re:Wow! Correct use of the Cliff Comma Subject! by Anonymous Coward · · Score: 0

    Hey, even a broken clock is right twice a day!

  20. this is a joke right by Anonymous Coward · · Score: 0

    6 nodes in the financial world is one tiny development setup for a small division. Finanacial databases often have hundreds of nodes. There are only a few people out there with the specialized knowledge and skill to setup one of these types of clusters, but they do exist in production systems all over the world.

  21. Since MySQL doesn't actually support 'clustering' by Anonymous Coward · · Score: 1, Insightful

    in any meaningful sense of the word, you could say the biggest MySQL cluster ever built has been... 1 machine.

    You can take a flatfile text database, put it on a bunch of computers,run round-robin DNS, and label it a 'large database cluster' if you want, but I don;t think that is a very useful definition of 'cluster'

    You could periodically run rsync to update the slave node's files from a single master file too, but this is routinely done with things like FTP mirrors of Linux distros. Is rpmfind.net a 'cluster'?

    Now, if MySQL have support for multi-master replication, or have made it possible for multiple processes to simultaneously read/write to the same database, then they might have something.

    Seriously, if you can run multiple MySQL master processes on an OpenMOSIX cluster without worrying about data corruption or lost transactions when a node fails, then you can say you have a cluster.

    If you can use a simple IPVS-based cluster, log a transaction to any node of a cluster, and have that transaction replicate transparently to all other nodes of the cluster, without running the risk of serving 'stale' data from the other nodes, then you can say you have a cluster.

    At this point, I don't think MySQL can do either, and probably won't in the near future.

    I'm all for a truly cluster-enabled databse solution in the OSS space, but MySQL isn't it.

  22. Re:Wow! Correct use of the Cliff Comma Subject! by kulpinator · · Score: 1

    Though for the sentence to be entirely correct, the comma after the quote 'biggest mysql cluster' should be placed within the single quotes, IIRC...

    No one is perfect. :-)

    Against stupidity the gods themselves contend in vain.

    --
    Karma: Positive (mostly due to rash moderations)
  23. Re:Since MySQL doesn't actually support 'clusterin by ExcerLee · · Score: 1

    You made a very good point about what is a cluster, especially on what is MySql cluster. But before you confirm your closing statement, please check this out, our secret weapon - the active cluster solution from Emic Networks. (http://www.emicnetworks.com/products/products_eac _mysql.html) I believe this is the only solution available right now to build a MySql cluster.