Slashdot Mirror

← Back to Stories (view on slashdot.org)

Zvents Releases Open Source Cluster Database Based on Google

Posted by ScuttleMonkey on from the surprised-it-took-this-long dept.

An anonymous reader writes "Local search engine company, Zvents, has released an open source distributed data storage system based on Google's released design specs. 'The new software, Hypertable, is designed to scale to 1000 nodes, all commodity PCs [...] The Google database design on which Hypertable is based, Bigtable, attracted a lot of developer buzz and a "Best Paper" award from the USENIX Association for "Bigtable: A Distributed Storage System for Structured Data" a 2006 publication from nine Google researchers including Fay Chang, Jeffrey Dean, and Sanjay Ghemawat. Google's Bigtable uses the company's in-house Google File System for storage.'"

87 comments

  1. Looks promising by stoolpigeon · · Score: 0

    I'll check back when they get out of alpha.

    --
    It's hard to believe that's how Micronians are made. Why don't we see it right now by having you both kiss one another?
    1. Re:Looks promising by alxbtk · · Score: 1

      "Imagine a Beowulf cluster of those"

    2. Re:Looks promising by calebt3 · · Score: 1

      A Beowulf cluster of Beowulf clusters?

    3. Re:Looks promising by Anonymous Coward · · Score: 0

      The question is whether each Beowulf cluster of Beowulf clusters is, in itself, another cluster of clusters.

      segfault

    4. Re:Looks promising by MadnessASAP · · Score: 1

      It's no use going on like that young man, It's Beowulf clusters all the way down.

      --
      I may agree with what you say, but I will defend to the death your right to face the consequences of saying it.
    5. Re:Looks promising by borgboy · · Score: 1

      Imagine a Beowulf cluster of these, imagining you. In Soviet Russia. In Hot. Grits-Filled. PANTS. That is all.

      --
      meh.
  2. Kitten Nipples by milsoRgen · · Score: 2, Insightful

    ..designed to scale to 1000 nodes, all commodity PCs... I'm just curious if anyone has had any experiance with these types of systems using commodity PCs, how is performance and does how well does it scale as you increase the amount of nodes?
    --
    I'm sick of following my dreams. I'm just going to ask where they're goin' and hook up with 'em later.
    1. Re:Kitten Nipples by Anonymous Coward · · Score: 0

      I'm just curious, did you RTFA? Apparently not.

    2. Re:Kitten Nipples by milsoRgen · · Score: 1

      Yes, as a matter of fact I did read it. But I'm kind of curious as to people's first hand knowledge...

      --
      I'm sick of following my dreams. I'm just going to ask where they're goin' and hook up with 'em later.
    3. Re:Kitten Nipples by Idiot+with+a+gun · · Score: 2, Insightful

      IIRC, Paypal and Google both use commodity PCs in clusters like this. Google uses something similar to above (duh), and Paypal uses a 3 tiered, multi-PC setup (Database, caching layer, and application side layers, respectively).

    4. Re:Kitten Nipples by Vectronic · · Score: 1

      I don't really have any first hand knowledge (outside of network rendering at a pretty small scale) but the concept is deffinetly sound, its the same reason why software uses "threads" and why processors now have more than one core...

      As for scaling, it would scale at the same rate as Non-Commodity Computers... if you have 999 computers all of equal performance, and then you add another one, you could expect a 0.1% change over-all...however its largely based on what sort of controllers you use, the same as how when Hyper-Threading first came out, many OS's (and far more software) couldnt take very good advantage of the second (pseudo) processor, it would still try and send to the first processor, and if it couldnt, wait until it could...

      As for this specific implimentation? I Dunno, if you plan on using this storage system, jump aboard, it is "Open Source"

      Commody PC's are often (if not always?) cheaper as far as performance goes, and although they are generally less reliable, because they are cheap, its...well... cheap... I mean its far cheaper to replace a $500 system every 2 years than a $10,000 every 5 years...

    5. Re:Kitten Nipples by TheRealMindChild · · Score: 1

      I was contracted to make the firebird db able to work with OpenSSI. Quite frankly, it worked beautifully, and it didn't require that much work. The issue I was faced with was that the storage had to be remote, which wasn't necessarily a problem per se, because nothing ever failed while I was around. Now if the power went out on the storage server and a few nodes at random, I really have no idea the havoc it would have caused... I was told my job was done and they didn't have a need for any sort of fault tolerance work. Meh.

      --

      "When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
    6. Re:Kitten Nipples by allenw · · Score: 2, Informative

      So, Hypertable runs on top of Hadoop. We don't use Hypertable (or HBase) so I can't commen on those. I can share some of our experiences with Hadoop though. I think it is safe to say that it scales quite well for the vast majority of people who need it. Let's deep dive for a bit...

      Hadoop keeps all of its file system metadata in memory on a machine called the name node. This includes information about block placement and which files are allocated which blocks. Therefore, the big crunch we've seen is the total amount of memory available to the JVM's heap. With a 16G machine (with ~14.5G heap) for the name node and ~2000 machines acting as data nodes, we're scaling to somewhere between 12-18 million or so files [it's been a while since I've looked... :) ]. Our capacity should be around 5PB or so, but keep in mind that large chunks of that are taken up by block replication and that, for the most part, that number is tied into the physical size of the drives in use. ... and, yes, this is all on commodity machines that you can get from pretty much anyone.

      We're working on making it scale better, of course. But we've come a long way in a really short time. [We've doubled capacity in less than... six months? Something like that.]

    7. Re:Kitten Nipples by debatem1 · · Score: 1

      I'm not an expert here, but on HPC clusters below 1k nodes I've never had any major problems with channel bonded gigabit and PVFS, however, the simulations we're running are not very I/O intensive.

    8. Re:Kitten Nipples by unsortedNU · · Score: 1

      Anyone who thinks that machine with 16G or RAM is _commodity PC_ has to see a doctor. Most important question for scaling would be could anyone keep base requirements withing _commodity_ limits in case of data sets grows? So if 16G of RAM _now_ is sufficient what will happen when the data sets doubles? could you just add 16G nodes or you'll ask for 32G or 64G nodes? Could you keep data your engine works with of a certain size? There is a HUGE difference between adding 4 16G nodes and single 64G. Is the memory the only requirements?

    9. Re:Kitten Nipples by Anonymous Coward · · Score: 0

      Stop using _so many_ underscores. Just write _normal text_. Read his comment again. The 16GB machine is the name node, and presumably the thousand other machines are normal nodes.

    10. Re:Kitten Nipples by Anonymous Coward · · Score: 0

      ... because one anonymous internet user's anecdotal evidence is better than another anonymous internet user's anecdotal evidence?

  3. You mean, like really, this time by dotancohen · · Score: 1

    Really, this time, a full fucking beowulf cluster (that runs linux!) is available to /.ers. No. Fucking. Way.

    Alright, I know it's only storage and not processing power, but that was inevitable.

    --
    It is dangerous to be right when the government is wrong.
    1. Re:You mean, like really, this time by drinkypoo · · Score: 2, Informative

      Really, this time, a full fucking beowulf cluster (that runs linux!) is available to /.ers. No. Fucking. Way.

      What?

      There is no particular piece of software that defines a cluster as a Beowulf. Commonly used parallel processing libraries include MPI (Message Passing Interface) and PVM (Parallel Virtual Machine). Both of these permit the programmer to divide a task among a group of networked computers, and recollect the results of processing.

      "Beowulf (computing)." Wikipedia, The Free Encyclopedia. 28 Jan 2008, 12:25 UTC. Wikimedia Foundation, Inc. 9 Feb 2008 <http://en.wikipedia.org/w/index.php?title=Beowulf_%28computing%29&oldid=187453674>.

      Wikipedia lists no less than eight Linux distributions designed specifically for building Beowulf clusters.

      Using OpenMosix, a single-system-image cluster can be created by booting cluster nodes with LiveCDs and with very little configuration. It's even been done with Xboxes, although they have very poor performance per watt consumed by modern standards.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    2. Re:You mean, like really, this time by dotancohen · · Score: 1

      Wikipedia lists no less than eight Linux distributions designed specifically for building Beowulf clusters. Actually, I'm aware of that. You could say that I overreacted.
      --
      It is dangerous to be right when the government is wrong.
  4. Links for anyone interested by Xeo2 · · Score: 1, Redundant

    Project page: http://www.hypertable.org/
    Zvents: http://www.zvents.com/

    --
    ___ alwaysBETA.com - Hey, you've got nothing better to do.
  5. how useful is DHT? by convolvatron · · Score: 3, Insightful

    i've been interested in this question for the last few years. how much do people value the ability to use a relational language and transactional consistency, or for most of these uses are these things just historical artifacts?

    1. Re:how useful is DHT? by Anonymous Coward · · Score: 0

      If you mean SQL, booooooooooo!!! Die SQL, DIE! Too bad nobody can agree on any sort of better standard.

      Anywho, transactional consistency is awesome. SQL is the lose.

    2. Re:how useful is DHT? by moderatorrater · · Score: 4, Interesting

      It's useful for ridiculously large data sets, like the entire internet. I know that medium sized stores (overstock, etc) use a relational database, and anything with less data than that is probably going to use a relational database. However, for extremely large data sets and certain repetitive, non-dependent loops (such as, say, looping through every website for a search), this can be useful. At least for now, relational databases are more useful overall, but tools like this have their place, and as data sets grow faster than real computational power, they'll be used more and more.

    3. Re:how useful is DHT? by ShieldW0lf · · Score: 4, Insightful

      i've been interested in this question for the last few years. how much do people value the ability to use a relational language and transactional consistency, or for most of these uses are these things just historical artifacts?

      In the 7 years I've been working in the industry, I've never delivered a single project that I would trust to a non-ACID database. Ever. And I doubt I ever will. If you want something that will generate some marketing material at high speed, and if it fails, who cares, well, use MySQL. If you want to do something that can handle a million pithy comments and if some of them get lost in the shuffle, who cares, well, that's fine too. Use whatever serves fast. If you're running Google, and it doesn't matter if a node drops out because there is no "right" answer to get wrong in the first place as long as you spit out a bunch of links, well, these sorts of non-resilient systems are fine.

      Personally, I've never done projects like that. In my projects, if the data isn't perfect always and forever, it's worse than if it had never been written. It's very existence is a liability, because people will rely on it when they shouldn't, for things that can't get by with "close".

      So yes. Transactional consistency and a solid relational model are pretty much mandatory, and not going anywhere soon. The idea that they might be replaced by technology such as this is laughable.

      --
      -1 Uncomfortable Truth
    4. Re:how useful is DHT? by Elite_Warrior · · Score: 1

      for most of these uses are these things just historical artifacts? they are not .There are still some places you can find their use .
    5. Re:how useful is DHT? by nguy · · Score: 2, Informative

      So yes. Transactional consistency and a solid relational model are pretty much mandatory, and not going anywhere soon. The idea that they might be replaced by technology such as this is laughable.

      Relational databases don't implement the relational model correctly anyway. As for transactional consistency, you can get that on top of many different kinds of stores (including file systems); relational databases have no monopoly on that.

    6. Re:how useful is DHT? by rubycodez · · Score: 0, Offtopic

      corporations constantly put bullshit data into those acid-compliant databases and then believe them forever as if they were true.

      already, we have the Dick-Shrub using such databases to terrorize the populace with expansion planned.

    7. Re:how useful is DHT? by convolvatron · · Score: 1

      oh i agree completely. check out datalog

    8. Re:how useful is DHT? by Anonymous Coward · · Score: 0

      Yeah.

      Your bank (sorry, sir, that's your balance, can't tell you where the money went tho), your phone company (sorry you missed that phone call, but you don't seem to be a customer), or an airline (truly sorry about your honeymoon, but there are no records about you being on this flight).

      But you're right, who needs ACID for *truly important things* like slashdot or flick*r!

      "Historical artifact", my ass. That's just the kind of things idiots assume are non-essential because they've always had it.

    9. Re:how useful is DHT? by vicaya · · Score: 1

      Hypertable is not a DHT. DHT is mostly useful for large amount of relatively small key value pairs. Hypertable like Bigtable use a metadata table to track the tablets (ranges in Hypertable term) Table automatically splits when they grow. A master server assigns the splitted tablets/ranges to appropriate tablet/range servers. Hypertable can be made to support transactions, as it has builtin versioning of data. As a result, Hypertable/Bigtable is more versatile than DHT.

    10. Re:how useful is DHT? by rbanffy · · Score: 1

      I have been using ZODB for a couple years now and one thing that bothers me with systems that store objects directly instead of "dehydrated" representations of them is that when the underlying code for the object changes significantly all sort of weird things occur

      I kind of like dehydrating/serializing objects to a simpler representation when persisting them. This uncomfortable step is nice because it shoehorns the data into a brand new instance.

      But that may be just me.

      --
      http://www.dieblinkenlights.com
    11. Re:how useful is DHT? by Anonymous Coward · · Score: 0

      You can get whatever you want out of whatever you want. I can get transactional consistency out of a single-tape Turing machine or punch cards. The point is what's cost effective, reliable, and performs well for general purpose tasks? A tried and true ACID complete RDBMS, that's what.

    12. Re:how useful is DHT? by YU+Nicks+NE+Way · · Score: 1

      In my thirty-plus years in the industry, I have seen a disk drive which could support transactional storage. The notion that you're going to write data in a manner which is more reliable than the underlying store is laughable. Even if you check the integrity of the underlying record, how do you know that your integrity check actually tested against the data you'll return next time? You don't; all you know is that the odds that you get back something else are negligibly small -- not zero, but low enough that you can neglect them, just like the word says.

      ACID and transactional reliability are useful features of a system, but they are not magic pixie dust which prevents data loss or corruption, and they are not the only way to minimize data loss and corruption. It may be, for instance, that you'll get better reliability out of your system through simple replication -- which has the automatic advantage of making your data easy to geo-replicate.

    13. Re:how useful is DHT? by ShieldW0lf · · Score: 1

      Ok, I'll bite... what are you selling?

      --
      -1 Uncomfortable Truth
    14. Re:how useful is DHT? by nullchar · · Score: 1

      I have been using ZODB for a couple years now and one thing that bothers me with systems that store objects directly instead of "dehydrated" representations of them is that when the underlying code for the object changes significantly all sort of weird things occur

      But that may be just me. It is not just you. What you described is THE problem for all Object Oriented Database Management Systems.

      That's why ORM (object-relational mapping) is so popular. People want a way to just use objects and not have to manually "dehydrate" data to disk. Unfortunately, most ORM isn't smart enough to execute the underlying SQL in an optimal way (depending of course on the relationship between your entities/objects).
  6. Pairing with RDBMS by Dr.Who · · Score: 1

    The article talks about adapting MySQL to be a front end. I wonder if someone is working on adapting PostgreSQL to be a front end too.

  7. Column Orientated DBMS by inKubus · · Score: 5, Informative

    This is a classic column-orientated DBMS, ala Sybase. You use these for data warehousing since they are optimized for read queries and not transactions. Stuff like Google search queries. It also allows you to quickly build cubes of data across a timeline, since you have data in columns instead of rows.

    IE:

    a,b,c,d,e; 1,2,3,4,5,6; a,b,c,d,e;

    instead of:

    a, 1, a;
    b, 2, b;
    c, 3, c;
    d, 4, d;
    e, 5, e;

    A cube using the time dimension would look like:

    01:01:01; a,b,c,d,e; 1,2,3,4,5; a,b,c,d,e;
    01:01:02; a,b,c,d,e; 1,2,6,4,5; a,b,c,d,e;

    It's pretty difficult to do the same thing with row-based DBMS. However, you can see that doing an insert is going to be costly.. This looks like a pretty good try, I know there were some other projects going to try to replicate what BigTable does. And after hearing that IBM story the other day about one computer running the entire internet, I started thinking about Google.

    More interesting is their distributed file system, which is what makes this really work well.

    --
    Cool! Amazing Toys.
    1. Re:Column Orientated DBMS by SystematicPsycho · · Score: 2, Informative

      You can do all those things in Orace -

      http://download.oracle.com/docs/cd/B19306_01/server.102/b14223/dimen.htm#i1006266

      Distributed filesystem - Oracle RAC (Real Application Clusters) fits the bill.

      --
      Analytic & algebraic topology of locally Euclidean meterization of infinitely differentiable Riemmanian manifold
    2. Re:Column Orientated DBMS by The+Clockwork+Troll · · Score: 1

      "orientated" is not considered proper usage.

      The correct word is "Asiantated".

      --

      There are no karma whores, only moderation johns
    3. Re:Column Orientated DBMS by kestasjk · · Score: 1

      Data warehousing and clustering is also available in MS SQL Server. Mod SQL Server +1 Underrated

      --
      // MD_Update(&m,buf,j);
    4. Re:Column Orientated DBMS by vicaya · · Score: 1

      Sorry, it's column-oriented but not really "classic". It supports ID part of ACID (which is possible but not implemented) very well, due to its builtin data versioning. It's optimized for both reads (random or sorted sequential scan) and very high random writes as you don't have to lock anything. The scalability and fault tolerance part is not classic either. BTW, google search queries do NOT use Bigtable. It's used for storing all the crawl data and input/output for their Map-reduce framework to build search indices that serve the real queries. It's also used for serving processed data to Google Reader, Map and Analytics etc.

    5. Re:Column Orientated DBMS by Duncan3 · · Score: 1

      Pointing out to the kids at Google their "new" tech is 30 years old and not even close to interesting to any researcher, becasue it's 30 years old, isn't going to get you anywhere around here.

      --
      - Adam L. Beberg - The Cosm Project - http://www.mithral.com/
    6. Re:Column Orientated DBMS by Anonymous Coward · · Score: 0

      BTW, google search queries do NOT use Bigtable. It's used for storing all the crawl data and input/output for their Map-reduce framework to build search indices that serve the real queries. It's also used for serving processed data to Google Reader, Map and Analytics etc.
      Source?
    7. Re:Column Orientated DBMS by vicaya · · Score: 1

      RTFP?

    8. Re:Column Orientated DBMS by jdanton1 · · Score: 1

      But it costs like 25k per processor for RAC though..

    9. Re:Column Orientated DBMS by vicaya · · Score: 1

      The Google system group guys are definitely not kids. Neither are Hypertable developers, who have actually built and deployed web scale search engines themselves. How many people on the slashdot can say that? We're well aware of the literatures in the area (RTFP to find out) and continuously learning from peers. Both Bigtable and Hypertable built upon previous solutions to solve real world web scale data problems. Many algorithms used in Bigtable/Hypertable appeared in literature only in the late 90s+, claiming the technology is 30 years old is like claiming a new car technology is hundreds of years old because horse carriages existed back then.

      The locality group in Hypertable/Bigtable allows you to control/tradeoff the I/O performance between column and row layout without changing the query interface. There are some expensive commercial solutions that claim auto tuning in that regard. However, it is simply waste of resources in many cases.

      The scale of actual Bigtable deployment is unprecedented.

  8. Yay! by Jurily · · Score: 1

    Can we do a distributed search engine with it? Google@home would be sooo cool.

    1. Re:Yay! by robo_mojo · · Score: 1

      You want to donate your network to google?

    2. Re:Yay! by Vectronic · · Score: 1

      Yeah, what a wonderful idea, I mean whatcouldpossiblygowrong if Google could access the hard drive of everyone who signed up to it?

      "Please wait while the Index is updated"

      "Please wait while we Upload new entries"

      "Please wait for the FBI to knock on your door"

    3. Re:Yay! by robo_mojo · · Score: 1

      whatcouldpossiblygowrong if Google could access the hard drive of everyone who signed up to it?
      They do that already, it is called Google Desktop.
    4. Re:Yay! by Vectronic · · Score: 1

      Yeah but thats not part of a network, other people can't search the contents of your computer (at least thats not an advertised "Feature"...lol)

  9. I don't think so... by warrax_666 · · Score: 2, Funny

    A cube using the time dimension looks more like this.

    --
    HAND.
    1. Re:I don't think so... by Anonymous Coward · · Score: 0

      I'm speechless! But I do like the big letters.

    2. Re:I don't think so... by networkBoy · · Score: 1

      W. T. F. ???

      --
      whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
    3. Re:I don't think so... by ushering05401 · · Score: 1

      Does anyone else change their homepage to links like this just to scare their girlfriends?

    4. Re:I don't think so... by sabrex15 · · Score: 1

      I concur, WTF?!?!?!

    5. Re:I don't think so... by moonbender · · Score: 1

      Even better: change her browser's homepage. *cackles*

      --
      Switch back to Slashdot's D1 system.
    6. Re:I don't think so... by inKubus · · Score: 1

      Does anyone else change their homepage to links like this just to scare their girlfriends?

      You must be new here.

      --
      Cool! Amazing Toys.
    7. Re:I don't think so... by xiesiou · · Score: 1

      yeah. he thinks we believe he has a girlfriend.

    8. Re:I don't think so... by neonmonk · · Score: 1

      People who are new to the internet are funn-eh.

    9. Re:I don't think so... by Jester998 · · Score: 1

      1995 called, and Geocities wants their webpage design back. :p

  10. Google 'Forms' by webword · · Score: 2, Informative

    I think Google Forms is more interesting. (Based on Google Spreadsheets.)

    --
    How to Download YouTube Videos
    1. Re:Google 'Forms' by vicaya · · Score: 1

      Do you know Google Forms are built upon Bigtable?

  11. Don't forget HBase by bryanduxbury · · Score: 1

    There's another open source BigTable clone called HBase . It's written in Java, and also runs on top of Hadoop Distributed Filesystem like Hypertable. It has the advantage of being a subproject of Hadoop. For anyone interested in using this kind of database, give HBase a shot. We can definitely use the additional testing. (Full disclosure - I am an HBase developer.)

    1. Re:Don't forget HBase by vicaya · · Score: 1

      Hypertable can run on a variety of DFS' that support global namespace. Currently it can run on HDFS, KFS (GFS clone from Kosmix) and any DFS with a POSIX compliant mounting point, include GlusterFS, Lustre and Parallel NFS, GPFS etc. An S3 DfsBroker can be made easily as well.

      Besides DFS flexibility and not-java, Hypertable supports access group (locality group in Bigtable) unlike HBase, where you have to resort to column family hacks for read performance tuning. Hypertable also have more block compression options (quicklz for fastest compression, lzo for fastest decompression and bmz (similar to the BMDiff and Zippy mentioned in Bigtable paper) for fast multiversion data compression/decompression). More compression options can be added trivially.

      Communication/IO layer of Hypertable is fully asynchronous to achieve throughput not yet possible by HBase.

      (Full disclosure - I'm a Hypertable developer :)

  12. Wheel: reinvented by stonecypher · · Score: 2, Insightful

    Mnesia has been able to handle things far in excess of the numbers cited, and with far better control of placement, for more than a decade. So has KDB. Also Coral8. This wouldn't even be on the map if people didn't start drooling the second they heard "based on Google." When they find out it's unstable and in alpha?

    Yawn.

    --
    StoneCypher is Full of BS
    1. Re:Wheel: reinvented by vicaya · · Score: 1

      Sorry, you couldn't be more wrong. Mnesia, KDB and Coral8 and Hypertable/Bigtable are completely different beasts for different purposes. Mnesia is mostly a DHT for key-value pair lookups while hypertabe/bigtable support efficient primary key sorted range scans. For concurrent read/write/update, Mnesia requires explicit locking. Hypertable/bigtable doesn't need explicit locking for that, consistency and isolation is achieved through data versioning. The most interesting feature here is time/history versioning for all the data, and efficient compression for such data.

      Hypertable/bigtable is mostly for online analytics and storage of many versions of the entire web, Mnesia was built to support real-time lookups and data management for telecom apps tightly coupled with Erlang.

      If you say Mnesia is a wheel, Hypertable/Bigtable would be a floating system for a hovercraft or maglev train.

    2. Re:Wheel: reinvented by Anonymous Coward · · Score: 0

      When they find out it's unstable and in alpha what?

      BTW, Bigtable is not in alpha, it is currently used to power many important Google services with millions of users.

    3. Re:Wheel: reinvented by Anonymous Coward · · Score: 0

      As far as I know,mnesia is not a sql database. It is only accessible through Erlang language.
      moreover, fair competition usually is a plus.

    4. Re:Wheel: reinvented by Anonymous Coward · · Score: 0

      The last time I checked Mnesia was not able to handle HUGE datasets. And I mean HUGE. In the range of multiple Terrabytes.

    5. Re:Wheel: reinvented by Anonymous Coward · · Score: 0

      So, what, they turned you down?

    6. Re:Wheel: reinvented by stonecypher · · Score: 1

      Then you've never checked. Mnesia is in real world use on tables in the petabytes.

      --
      StoneCypher is Full of BS
    7. Re:Wheel: reinvented by stonecypher · · Score: 2, Informative

      Mnesia is mostly a DHT for key-value pair lookups while hypertabe/bigtable support efficient primary key sorted range scans.
      Pretty much every database on earth has key sorted ranges. Please be less of a noob. Go look up ondex_match_object.

      For concurrent read/write/update, Mnesia requires explicit locking
      No, it doesn't. It offers explicit locking, because it's been proven for decades that without it, you cannot have hard realtime queries, something that mnesia wanted to offer. You don't have to use that stuff at all; it's just there in case you need it.

      Hypertable/bigtable doesn't need explicit locking for that, consistency and isolation is achieved through data versioning.
      Every distributed ACID database, by definition, uses a local sixth form implementation under the hood; there is no other way to provide query durability. All given examples do this.

      The most interesting feature here is time/history versioning for all the data, and efficient compression for such data.
      Yeah, I remember the first time I read about that too, back when I was learning Oracle in the early 90s. Just because it's interesting to you doesn't mean it isn't implemented in other databases already.

      Hypertable/bigtable is mostly for online analytics and storage of many versions of the entire web, Mnesia was built to support real-time lookups and data management for telecom apps tightly coupled with Erlang.
      The real purpose of mnesia is distributed durability, so you're aware. They actually discuss it in the documentation which you obviously have never read. That said, I don't really care what the software is for: Mnesia can handle bigger datasets over more nodes in realtime, offers the user better control of which data is on which node, and has a much more flexible locational querying system. There's nothing you can do in Bigtable that you can't do in Mnesia, and the reverse is most certainly not true.

      So, unless SQL is particularly important to you, this is a useless project. There's a reason Google's moving to Erlang so fast - they're discovering that a lot of the tools they've half-assed reinvented in Python already exist in Erlang in far more flexible fashions. This is nothing more than another map/reduce fiasco - a first generation solution to a problem that the internet adores because it's never seen any solution to the problem, but something which has been far better addressed in real industry for thirty or so years. If google would just quit stealing people from Microsoft, who makes application and system software, and start stealing people from AT&T and Ericsson, who make hard realtime system software, they'd find they wouldn't have to spend so much time poorly re-walking what's already been pathed.

      If Google would just buy Bluetail already, things would start changing for the better, fast.

      If you say Mnesia is a wheel, Hypertable/Bigtable would be a floating system for a hovercraft or maglev train.
      Metaphors are only useful when they elucidate something specific. Mnesia is radically more powerful than hypertable; I suggest you spend less time at the altar and more at the library. Or, to put it in terms that apparently you will understand, you just tried to rub in my face how much more powerful your Geo is than my Technodrome.

      You have done such a spectacularly poor job of making your case that all I can imagine as your reason to say something like that is:
      1. You think mnesia doesn't have indices
      2. You think Mnesia is manually locked
      3. You think Mnesia isn't versionned
      4. You think Bigtable can handle more physical storage than Mnesia
      Of those, not only are you wrong on every count, but only the last is in any way something that someone who knows even the basics about distributed databases would even begin to consider. Doesn't support indices? Are you nuts? You really think there's a database that can't sort its contents?

      Unbelievable.
      --
      StoneCypher is Full of BS
    8. Re:Wheel: reinvented by vicaya · · Score: 1
      Thanks for the playing :) Lets see:

      There's a reason Google's moving to Erlang so fast - they're discovering that a lot of the tools they've half-assed reinvented in Python already exist in Erlang in far more flexible fashions. This is nothing more than another map/reduce fiasco - a first generation solution to a problem that the internet adores because it's never seen any solution to the problem, but something which has been far better addressed in real industry for thirty or so years. If google would just quit stealing people from Microsoft, who makes application and system software, and start stealing people from AT&T and Ericsson, who make hard realtime system software, they'd find they wouldn't have to spend so much time poorly re-walking what's already been pathed

      Profound ignorance! Google is evaluating ejabberd/erlang for messaging services/framework. And you claim they're moving to Erlang!? Do you know what percentage of code of Google is written in python? Your criticism of map-reduce is so laughable, it's not worth arguing.

      Mnesia can handle bigger datasets over more nodes in realtime, offers the user better control of which data is on which node, and has a much more flexible locational querying system. There's nothing you can do in Bigtable that you can't do in Mnesia, and the reverse is most certainly not true.
      According the Mnesia FAQ, Mnesia is mostly an in-memory DB and ill suited for large on disk data. The largest table size is 4GB. The largest number of records in a table I've seen so far is on par with 100 million records, which is not even on the same magnitude of Hypertable/Bigtable, which is designed to routinely hold trillions of records and petabytes of data in ONE table. Fragmented table in Mnesia is an afterthought and a ugly hack.

      You have done such a spectacularly poor job of making your case that all I can imagine as your reason to say something like that is:

      1. 1. You think mnesia doesn't have indices
      2. 2. You think Mnesia is manually locked
      3. 3. You think Mnesia isn't versionned
      4. 4. You think Bigtable can handle more physical storage than Mnesia
      Sorry, none of the assumptions (that I had these notions) you mentioned you is true. By versioning I meant explicit version control besides the purpose of MVCC. Hypertable/Bigtable allows you to explicitly keep n number of versions data easily for historical analysis with a simple clause in the schema. By scanning, I meant efficient on disk scanning for data too large to fit in to memory. Indices is useless for that (think about the seeks). Do you know Bigtable/Hypertable doesn't typically use separate indices? Do you know what locality group is for? Can Mnesia compress data to 1/10th of the original size at 100MB/s per core? Erlang is too slow to implement any compression algorithm that worth its salt. Foreign language interface marshaling overhead per invocation in Erlang is unbelievable.

      Of those, not only are you wrong on every count, but only the last is in any way something that someone who knows even the basics about distributed databases would even begin to consider. Doesn't support indices? Are you nuts? You really think there's a database that can't sort its contents?
      Did I say Mnesia doesn't support indices? You're so blind that you're projecting ignorance.

      Unbelievable

      Indeed, I was not even criticizing Mnesia, it's a fine distributed in-memory oriented db great for its purpose, but driving an imaginary technodrome while laughing at a real bullet train is one of the funniest things I've read so far on slashdot.

    9. Re:Wheel: reinvented by vicaya · · Score: 1

      Citations needed.

      All my cited data can be found at http://research.google.com/pubs/papers.html

      Tell me how to store petabytes in a 4GB Table because Erlang dets use 32 bit file offset?

      Storing petabytes small key value pairs on a DHT a la Mnesia is trivial. Sustained ordered on disk scanning at hundreds MB/s per core is not.

    10. Re:Wheel: reinvented by Anonymous Coward · · Score: 0

      Citations needed.
      That's cute. First tell him something you made up about limits that don't exist in mnesia, then point out a stack of bland research papers at google and say you cited your data. Incidentally, that pile of documents doesn't back you up, and pointing at a list like that then saying you cited things is nonsense. Grow up.

      Tell me how to store petabytes in a 4GB Table because Erlang dets use 32 bit file offset?
      Oh, yes, if DETS has a limit, surely it carries up into things built on DETS, right? Oh, wait, I forgot, DETS doesn't actually use a 32 bit file offset. Way to invent things. Pity you're full of crap.

      Sustained ordered on disk scanning at hundreds MB/s per core is not.
      Citations needed. And don't point at a directory full of google papers, jerk.
    11. Re:Wheel: reinvented by Stu+Charlton · · Score: 1

      If Google would just buy Bluetail already, things would start changing for the better, fast.

      I had thought Bluetail was bought many years ago and absorbed into Nortel ...

      --
      -Stu
  13. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion

  14. Is this the correct place to ask... by Seismologist · · Score: 1

    ... if it runs Linux?

    --
    ~ In Trust, We Trust ~
    1. Re:Is this the correct place to ask... by youthoftoday · · Score: 1

      Yes.

      But it's not the correct place to get an answer.

      --
      -1 not first post
  15. Hbase -- Apache's BigTable by otisg · · Score: 1

    Over at ASF a bunch of smart people are building Hadoop and Hbase. The latter is the open-source version of the BigTable, similar to Hypertable, but written in Java (not C++) and being super actively developed in the open and under the ASF umbrella.

    --
    Simpy
    1. Re:Hbase -- Apache's BigTable by Anonymous Coward · · Score: 0

      written in Java (not C++) and being super actively developed in the open and under the ASF
      You're saying that it works great on everything, but it's not fast. Is that what you are telling us?
      a/c

    2. Re:Hbase -- Apache's BigTable by Reverend528 · · Score: 1

      You're saying that it works great on everything, but it's not fast. Is that what you are telling us?
      No, he's saying that it's a bitch to install, works mediocre on most hardware, and is not fast.
      --
      Badass Resumes
  16. Been there, done that; not exciting by Jaxoreth · · Score: 1

    Can we do a distributed search engine with it? Google@home would be sooo cool. I'm afraid that's been done before, and it didn't work out so well, and may have always been a bad idea in the first place.
    --
    In general, it is safe and legal to kill your children. -- POSIX Programmer's Guide
  17. Interesting, but does it.... by Anonymous Coward · · Score: 0

    Looks interesting, but since it's based on Google, does it also add all your data to an NSA database?

  18. No. by Stu+Charlton · · Score: 1

    I was an Oracle DBA in the past.

    Oracle Dimensions are a logical overlay, they have no impact on how the data is physically organized in segments.

    Neither does Oracle RAC -- it uses the same underlying storage format as regular Oracle.

    You *could* do column-orientation in Oracle with a data cartridge, but that would likely be third party.

    I could see Oracle offering this natively in a future release, maybe 11g r2...

    --
    -Stu