Slashdot Mirror
Google File System Evolves, Hadoop To Follow
Christophe Bisciglia, Google's former infrastructure guru and current member of the Cloudera start-up team, has commented on Google's latest iteration on their GFS file system and deemed its features well within the evolutionary capabilities of open-source competitor Hadoop. "Details on Google's GFS2 are slim. After all, it's Google. But based on what he's read, Bisciglia calls the update 'the next logical iteration' of the original GFS, and he sees Hadoop eventually following in the (rather sketchy) footsteps left by his former employer. 'A lot of the things Google is talking about are very logical directions for Hadoop to go,' Bisciglia tells The Reg. 'One of the things I've been very happy to see repeatedly demonstrated is that Hadoop has been able to implement [new Google GFS and MapReduce] features in approximately the same order. This shows that the fundamentals of Hadoop are solid, that the fundamentals are based on the same principles that allowed Google's systems to scale over the years.'"
I wish they would stop taking names from Star Wars.
The quoted text seems to be coming from this register story entitled "Google File System II stalked by open-source elephant", not the one linked in the summary. Also, I can't follow the Firehose link below the story to see if this was changed from the original submission.
My work here is dung.
Kill M$, take the fat.
"The difference between genius and stupidity is that genius has it's limits" - Albert Einstein
I thought GFS was a file system only meant to be internally by Google. And if thats the case, then how is it competing with anything?
After years of not using a signature, I am going to make one to say the following: Fuck Beta
Reading up on the different file systems Hadoop and GFS , it appears these are used primarily for "cloud computing".
Is that correct?
deemed its features well within the evolutionary capabilities of open-source competitor Hadoop.
I didn't know that file systems were living beings that could evolve. I thought they were inanimate and were designed by humans? Should I be afraid? Is it sentient yet?
this nation, under God, shall have a new birth of freedom. -- Lincoln, Gettysburg Address
It Rebelled.
It Evolved.
There are many Copies.
And it has a Plan.
Did you mount a military-grade, variable-focus MASER on an unlicensed artificial intelligence?
Yes, now that somebody else has shown the way, Hadoop will do its best to imitate them, like the good open source project it is!
If you want to be buzz-word compliant, then yes, kind of.
More to the point, GFS and HDFS are distributed file-systems that are designed to run on potentially very large clusters of commodity hardware. The potential applications are quite diverse. Hadoop itself involves more than just the file-system, but HDFS is really at the core of any application you would want to build with it. This list gives you a good idea of who uses Hadoop and for what purpose.
I've been on the market for a distributed, clustered file system for some time. Unfortunately, Hardoop is not really what I'm looking for. What I'm looking for:
1) Redundancy - no single point of failure.
2) Suitable for standard-sized file I/O.
3) Performance that doesn't completely suck ass.
4) Graceful re-integration when bringing a cluster portion back online.
5) Accessible through standard interfaces. (EG: Posix F/S)
6) Doesn't require a PHD in the technology to administer.
7) Doesn't require insane quantities of cash to build.
8) Stable.
There are clustered file systems that have some of these qualities. None that I've found so far have *all* of these qualities.
Hardoop fails on #1, #2, and #6. It has a single nameserver commanding the cluster, so if it goes down, well... (shrug) It also does poorly for "normal" sized files, somehow having a 10 GB file is the norm for Google. And setting a multiple node cluster up is definitely non-trivial.
Of all that I've reviewed, GlusterFS did the best but even in that case, I ran into severe over-serialization that brought my 6-node cluster to its knees. I tried three times to roll it out, and had to roll back all three times. I fiddled with the brick setup and caches for days before finally throwing in the towel.
Now I get by with rsyncing program files, and a homegrown data distribution setup using network sockets and xinetd. Not optimal to be sure, but so far it's scaled linearly and provides decent performance, at the price of a PHD in said technology. I guess you could compare our technology to MogileFS, only our scheme
A) uses DNS records to coordinate the cluster so that it scales up,
B) has a richer "where is the file" schema than the simple flat keys used by Mogile, and
C) has the ability to execute programs against files for performance. (EG: grep for searching text files, tar/gzip for compress/uncompress, virus scans, etc)
D) has the ability to "hang open" for activities like logging.
So far, this has held up well with about 500,000 file operations and millions of log entries per business day with an average file size of about 1-3 megabytes and every sign that growth can continue by simply stacking on more hardware. No, I'm not talking about massive throughput, but I *am* talking about the need for high availability systems that scale nicely without bottlenecks and exorbitant expense. Yes, it works pretty well, but we've had to invest significant programming time to do this.
Guess it's like the old engineering saw: Convenient, Cheap, Quality: pick any two!
I have no problem with your religion until you decide it's reason to deprive others of the truth.
pc = politically correct
If they keep naming things with coffee references (including Java), what would happen if it's discovered that coffee causes cancer or shrunken balls or what not?
Don't have to wait - in the UK one of the more egregious papers regularly publishes a scare story about cancer. So much so that there are sites dedicated to Daily Mail Oncology Ontology.
Curiously coffee falls into both the good and bad camps.
actually it's not that curious - never let consistency spoil a good rant
...it is being intelligently designed.
"If you want to be buzz-word compliant, then yes, kind of."
I see you're trying to be pedantic douchebag compliant.
Congratulations, you succeeded.