Slashdot Mirror
Google's 4000 Node Linux Cluster
Check out the Red Hat press release running at LWN, or the news article at techweb about Google's 4000 Node Linux Box. Both articles are basically Red Hat commercials, but there's some interesting bits like the fact that they have a terebyte index of 300 million Web pages, and that they might expand their cluster to 6000 nodes in the future.
You can say whatever you want. That's what the anonymous button is for. Say something you fear punishment from anonymously, karma whore with your name so keep your karma!!!
What are they doing for Linux? Exploiting it.
That's me.
Building a graph is quite straightforward
If you've taken a algorithms course (and passed) you, too, could probably write an Oracle of Bacon.
I believe that this was done on a single computer. Pretty sure it wasn't a cluster of 4,000 ;)
The Oracle takes up about 10% of the CPU time on a single Sun Ultra 5/300. (I didn't pick the machine. The Oracle also runs on my Linux 2xP2/350 at home.) It takes around 80 MB of memory -- 25 for the actors and movies and the rest for a cache of recent queries. Each query consumes 0.6 seconds of CPU time, or 0.02 seconds if it comes from the cache. 90-95% of queries get served from the cache, so the Oracle should withstand 10+ queries per second, sustained.
The task is trivially parallelizable across big clusters (UVA has a 256-node cluster that would do the trick), but the need for that has never arisen... :)
--Patrick
Because I am a hacker of the old school...I don't throw away hardware...my old house router was a 386sx 16 with two 40 Meg MFM drives. I was running Slackware, of course. I don't there are any other full-featured distros that will fit in that small of space.
ttyl
Farrell
CAN-CON 2019 - Ottawa's only book oriented Science Fiction Convention! October 18-20, Sheraton Hotel, Ottawa, Canada h
To start with, I would guess price. If you are running 3K systems, the cost of buying in batches of 100+ of x86 clones is pretty cheap, and parts are interchangable. If you had bought DEC, oops, COMPAQ Alphas, the design of the case, etc, changes between generations of the product, and you loose hardware interchangablity. Same goes for SUNs, or PPC. ON the other hand, I can *still* put an old MFM hard drive controller in dual PIII 800MHz system at boot off of it. Try getting that level of compatiblity between 15 year old equipment on any other platform that is current today!
ttyl
Farrell
CAN-CON 2019 - Ottawa's only book oriented Science Fiction Convention! October 18-20, Sheraton Hotel, Ottawa, Canada h
Hello? Where did you buy your brain? We are talking about 4000 PCs here. Do the maths.
If a PC fails, on average, more frequently than once every 10 years and 11 months, then you are going to be replacing one machine each day, and three on Mondays, unless you also work Saturdays and Sundays.
Talk about job satisfaction.
I'd be fascianted to read about the problems the google team has had to overcome in managing a their thousands of boxes - perhaps an ask slashdot article?
If your server has time left over to get good SETI@Home stats, then you probably spent too much money on it. ;)
I guess there's something to be said for "headroom."
(In practice, the ratio would probably be closer to 12-15 Intel boxes per Sun 6500, I would guess, as a PIII doing it kind of integer work would likely outperform a SPARC II)
There is more to a computer than the CPU. I have never needed to test or specify a system for a specific use, I thought that Suns had better memory system bandwidth, something that would seem to me to be a bottleneck here than computation speed.
For the price, x86 is probably better, though I sure hope they selected solid quality components, I would hate to be on a crew trying to maintain 4000 computers. Other things to consider are power use and the need for climate control or at minimum A/C. So, in your support, there are many factors that need to be taken into account when considering these things, maybe x86 won out on a serious shootout.
I think RAIS is the term you're looking for - "Servers" and, moderation at "Funny"? Heck, it's the way of the future. The best thing about WebObjects is you get RAIS for free with your application. I wondered for a while why they don't make redundant macs, before I realized they weren't necessary.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Nothing more exciting that seeing rows and rows of rack mounted PCs.
Actually, computing the Kevin Bacon problem from IMDB info is not that computing intensive. Building a graph is quite straightforward, and traversing it (at least for specified names, as opposed to all names, which I guess might take a while) is also reasonably straightforward. Check out the Oracle of Bacon- when a friend of mine was at UVa, they did the conversion to a graph and set up this web interface to it. In particular, you might be interested to note the Bacon Numbers, which indicate that of the 390,027 actors who can be linked to Kevin Bacon, 390,023 can be linked to in 7 steps or less. The other 4 can be linked to in eight steps. I believe that this was done on a single computer. Pretty sure it wasn't a cluster of 4,000 ;) ~luge(ahh, the continuing quest to confuse the moderators... is this OT or "interesting?" Only time will tell :)
IAAL,BIANLY
Actually, I was looking through some articles linked to from their site, and the original claim (from the inventors of the game, I think) was that no actor has a Bacon number > 4. The UVa guys a) disproved for the set of all actors (as I mentioned) and b) actually proved for American actors- i.e., everyone who has a Bacon number > 4 is from IMDB's big foreign film section. BTW, there are actually actors who satisfy the seven degree rule- Christo pher Lee and Anthony Quinn, for example.
~luge(I'm asking the friend who owns oracleofbacon.org what kind of hardware they used... no answer yet)
IAAL,BIANLY
Offtopic, but that is possible. When you post to an article where you have moderated posts, your moderations are canceled. You won't get any points back, though.
Looking at information on the web about the new Compaq Wildfire series, surly they would be a sure contender..
You can look at the specs at:
Benchmark performance of GS320
Which says Suns EV10000 (64 processor) is not as fast as the 32 processor GS320....
And the price of the GS320 is estimated arround
$600,000 from:
The Register
So one would have thought this system to be a real contender!!! Considering the PCs are $1000 each and if we use 6000 of them, we would be able to afford 10 of these GS320 beasts (with a total of 320 alpha processors )...
--
A large number of machines in a cluster does not directly imply that the wiring and maintenence is going to be a mess. Perhaps if _you_ did it, but not if I did. I've worked on several large installation are there are right ways to do this stuff. It's not brain surgery.
There are very clean, and tested, methods to install large cabling installations, to handle large power requirements, etc... Certainly, the setup is complex, but that's the biz.
Think telco...go check out one of UUnet, Globalcenter, or Exodus' datacenters when you have a chance.
-Buffy
Read the original Google paper. It includes some description of Google's architecture.
I am not really sure why the leading Distribution company out there really needed this publicity.
... statement at the bottom? Come on, we all have our reasons for running Linux. It just seems like fluff or worse geek bragging.
First, I thought as another user that it was obvious that Google was using Linux. Also, the whole clustering capability has been known for awhile.
Do we need every site to have a THIS SITE RUNS ON
I can hear the slashdotters now:
"You can't be a real geek site you run RedHat and not Debian!"
"Well little boy you aren't a real man till you have gotten Slackware working on a 486 33mhz machine with a bad BIOS."
"You are all full of crap because I run BSD and it has REAL security."
Yeah, yeah, yeah. Whatever. They can run a cluster and that is really neat and I love to Quake from their server and I bet my projects would compile really quick and wouldn't be neat if..... I think you know the rest.
ACK
Actually many of the larger sun systems now have the ability to 'partition' the processor boards in the system into seperate 'virtual machines' so that if one goes down the rest of the system can keep running while you replace that one processor board and then bring it up.
Use to have a link but lost it...
--"Karma is justice without the satisfaction"
God forbid that Sun should have competent people getting in the way of product shipping dates...
Evil fake Bruce (.) strikes again.
-- Remember: Wherever you go, there you are!
Of course there are several manufacturers of x86 hardware, so a comparason between a high-quality Sun box and a low-end x86 box is meaningless. How about a comparason of an x86 server, like VA or Compaq, that has much higher quality components against a Sun. Apples-to-apples.
Administration doesn't have to be too difficult either, there are several tools to help in managing large numbers of UNIX type systems, like PIKT and rdist, that can replicate files and configurations througout a mass of machines.
Maybe it would be more efficient to have a few very large boxes (E10K, S80, S/390, SGI O2K) but I don't think they started with the megabucks to burn, this gives them a cheap, scalable paradigm (solution, I meant solution!) that seems to work for them.
-- Remember: Wherever you go, there you are!
Who is "The Islamic Faith", and can I hire him as a wedding singer? This statement makes about as much sense as yours, the Islamic faith isn't just one person, it would be like stating that all Americans are Timothy McVeigh.
-- Remember: Wherever you go, there you are!
Does anyone know the specs of the machines that were used in the cluster?
How about creating a low powered Tiny PC cluster for handling all the pages?
Jumpstart the tartan drive.
Gee, man, thanks for the support! :)
Even if the actual search is slower than other engines, it's a user interface design that makes the overall searching much quicker. The only thing that I would like to see changed on Google is to be able to display 100 or more results from the front page. Then again, that would take away from the "streamlined interface" that I just got done praising, so I'll just shut up now.
-sk
- 100 results per page, most other engines only do 20 and/or changing the number of resutls per page is hard to find/do.
- Cached results. When I was looking for paintings by an artist, Google found several hits on past eBay auctions. The auctions were no longer on eBay, so I tried Google's cached page, and found pictures of stunning paintings.
By not overloading my bandwidth with crap ads and layout, letting me see tons of results on a page, and getting me information that been removed from the web Google has built tremendous user loyalty. Other web companies might want to note how Google has become so popular and built such loyalty. They're doing it right.Anyone know what database is used by either Google or Fast? Did they stay open source (Postgres) or is the database commercial (Oracle, Sybase...).
There's no indication in those that Google is using Beowulf technology. Beowulf is only one way to cluster, although the others don't generally have such an enchanting name.
Any OS can get bloated, it's just a matter of what you consider excessive software. There's no question that the 300 little tools that come with Unix systems are useful for scripting, but to someone that only wants to run a web browser they're bloat. However, Unix is modular enough that you can run without many pieces -- it's the monolithic systems where bloat becomes really painful.
If you read the instructions, Google tells you that you put phrases in quotation marks. Or maybe you should use Metacrawler, as it has a "phrase" button.
Maintenance of a large number of machines
comes down to managing differences between
them. If they're the same then handling
200 is no worse than handling 2000.
E10000's are far more tricky beasts than a
simple linux box, especially if you're wanting
to do domaining (the only reason you would choose
an e10000 over an e6500).
-- Don't believe everything you read, hear or think
What does this article really offer new? It's been known from right the beginning that they run a Linux cluster. See here:
I find it more interesting that infact they use python .
signatures pending - ansa@kos.to - (dont mail there)
I think Google is making an enormous mistake by using Linux as their OS-of-choice. Something more robust, such as Java or even Python,
Sure, Java already does many OS-like things. Python doesn't, tho. Strangely enough, google is widely known for using Python already....
Sure they do. Try searching for "linux server" and you get back a text-only ad for DigitalNation at the top.
FWIW, moderators, this guy is a fake. I'm not sure if it's someone trying to impersonate and defame a real individual, or if it's just someone trying to see how misinformation can get moderated up if it's written in a certain fashion, but either way, there's a lesson here: don't mod something "informative" because of its tone.
--
Michael Sims-michael at slashdot.org
I have been a big fan of Google since they first came out and have been consistently impressed, not only for the quality of results but for their clean interface. I hope they keep it that way.
My favorite searches: Out of curiosity, I typed in Onion, and the top site was "The Onion- America's finest news source". My all time favorite happened the other day when I was searching for information of the statistics program "R". I typed in R, and the R FAQ was number 4 on the list! Rock on guys!
"He looks like he got in a fight with the 70's and got his ass kicked"
-Sherman Alexie
Is it the first stage in a 12-step recovery plan for VBlusers?
Molly.
Was this article posted in an effort to see the /. effect on the cluster?
/. ten seconds ago. There! Look at that little bump on CPU utilization. Wow!"
"Well, it just got posted to
My $.01
LOAD "SIG",8,1
LOADING...
READY.
RUN
Or even better, use a program like cfengine (http://www.iu.hioslo.no/cfengine/) and automate all your sysadmin tasks. We use a similar system at work written by our developers, and maintaining our 150+ servers spread out in 3 countries is *easy*. Need to upgrade Apache? Do it in one place, and it'll be distributed out during the night or on demand. Need to apply a patch of some kind? Yet again, it's done in one place and pushed out to the servers.
There's no end to the possibilities!
needlejaw@angst:~$
ugh, it is more than that, you are obviously talking about what you read from the paper, why don't you go talk to the google guys directly. A lot more than the web crawlers and servers that feeds the crawlers are implemented in python. ...and when you do talk to them, don't ask how many modules is implemented in python, but rather ask how much work python plays a role.
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
I was agreeing with you till you mentioned "Arab terrorists", I would have agreed with "terrorists", but your having to add Arab their is very wrong. I am not Arab by any means, but I will like those with moderation mode to flag you down, We need to protect people from your likes, not just the NSA. People who sterotype and make very harmful statements.
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
Hey, that was interesting. :)
:-p
But has anyone consider that, those 6000 Linux boxes are actually a VM in an big s/390??
Just a tough.
Ok, I'm starting my own cluster now !! Let me see, a 386/4M, 2 486/16M, Pentium 75/32M, Pentium II 400/128M and 1 AMD 750/256M. Sure i can beat them. I have my own cluster!! What can i crunch now? I will test SETI@Home. May be i can find an alien life form sending email to Bill Gates:
BORG:Resistence is Futile!
BG: Hey! That's my MODO I'm gonna sue you!
Regards
assert(expired(knowldege)); core dump
For as far as I know, techweb has a (I think very nice) script running that puts links for all technical terms in papers anyway.
Nothing wrong with that, I'd say. If you know it, fine, if you don't and accidentilly end up on one of their pages, it's only of help. It's not like you have to actively do anything with them.
Mad.
Coz eternity my friend, is a long *ing time.
When you buy a Sun the damned thing just doesn't fall down unless you have a system mangler who keeps dicking around with it. And if a single Sun could not address the problem, then maybe it's time to buy some real iron, like a maxed out S/390. When you have a terabyte of data to process, you have to start paying a little more attention to things like I/O.
I can assure, as one with experience, that Suns most certainly *DO* fall over. The E10K is a nice box with quite a few redundant feature. The ability to remove a system board on the fly ranks pretty high up there. Assuming you lose a processor or memory and assuming (this is a big one) that your system doesn't fall over immediatly, you can most likely replace that processor hot. However... it's been my experience that failling processors or memory bring the box (or domain) down more often than not. And let me assure, sun processors *DO* fail periodically. In a large shop, you can exect to replace at least one or two processors per year.
Not to start a flame war over languages, but I'd imagine that they just like Python better for large projects. Personally, perl has many cool features, but it certainly can be difficult for some people (such as myself) to make it scale to large projects. Other people seem to have no such problem, so I'm sure that it is just a matter of personal taste.
-- Superlame http://catpro.dragonfire.net/joshua/
I won't be happy until they have
10^100 nodes...
--
http://www.aikiweb.com - AikiWeb Aikido Information
None the less, I see your point. I just wasn't refering to such a small setup...
And you have to pay per installation of Solaris, and per support query. You can't have a team of qualified Solaris people ready to nail down any bizarre bug in the OS. You can with Linux. Also, it is cheaper to replace PCs than it is to replace SPARC servers. The electricity cost will be high but SPARC servers aren't exactly environmentally conscious either what with them having multiple PSUs/processors etc. As was the initial investment but the rest of the costs are negligible and it works. The Solaris server we had at uni. was constantly needing reboots and various tools had to be removed as they just didn't work properly.
Did you read what was on that page??1 /com1921.htm
http://www7.scu.edu.au/programme/fullpapers/192
Goto Section 4.1 and read it! Here is what you missed:
"Most of Google is implemented in C or C++ for efficiency and can run in either Solaris or Linux."
Only the web crawlers are implemented in python. (See Section 4.3)
I think google just took over the search engine industry. theysa sooo smart.
-= Briareos =-
So if you're worried about keeping what you're saying private, why are you making it available via the web? Fnord.
Of course, there are such things as newsgroup and mailing list archives. And I've had a few of my e-mail addresses spammed because of a mailing list archives.
But even when I'm writing to a so-called closed mailing list fnord, I still think of it as public communications. If you don't intend for the whole world to hear, you'd best keep your mouth shut.
But what you're saying is like Lady Godiva wondering why everyone is staring at her.
Oh, and just in case anyone is paying attention, I've ordered sattelite images of Bill Clinton's new house in New York. I just have figure out the right amount of C4 to use. And yes, I'll be downloading Metallica songs off of Napster and reading Mein Kampf while I do it. Fnord.
Any sufficiently advanced civilization is indistinguishable from Gods.
Have you ever been to a hosting facility? Everything is provided for you (cooling, UPS, generator, security, cabling, etc.).
Most hosting facilities can handle several thousand machines easily and most large sites (Yahoo, Netscape, etc.) have systems in at least three facilities. Yahoo has about 1000 machines at the old Frontier Global Center in Santa Clara, CA (they're all Dell mini-towers).
Most of the big sites use this exact same strategy and very few people use Sun E10k's for HTTP servers (DB's yes...). Looking around most hosting facilities in Silicon Valley (I've been to quite a few...), the most common machines are pizza-box style Sun's, followed by Compaq's and then everyone else.
Besides, 4000 1U rackmount linux boxes would only take up about 100 racks, which is about 4000 sq/ft. Just to give you an idea, the new PSInet facility going up in South San Francisco is 50,000 sq/ft, so I would say that Google's strategy is no big deal, just a generic way to do it that everyone else is using...
Chris.
-- I don't have a cool sig.
As a reference, Yahoo uses FreeBSD running on Dell mini-towers. For load balacing on www.yahoo.com, they use straight round-robin DNS, hosted by Akami.
I don't know about their apps, or what alogrithms the DSN is running. However, if you will have lots of traffic, you should have someone else do DNS for you (Akami, TotalDNS, UltraDNS) and you should think of the ant model rather than the elephant model for both your content hosting and applications.
Don't forget about content replication and deployment, which should all be automated.
HTH,
Chris.
-- I don't have a cool sig.
Now if I could just build this kinda processing power in my head maybe it wouldn't take me so long to learn sh ... er new stuff ;)
spam, spam, spam, spam, e-mail, news and spam.
did you notice on the techweb article, they had links to the definitions of MIPS and TB? what's their deal? i mean, if they're a technical web site, they need not explain such novice terms, do they? if you are an IT professional and don't know your basic acronyms, i suggest you spend a day with the Acronym Finder. Sorry to bring this up, it just irked me. i could see them putting this on like yahoo news or something, but techweb?
anyone got photos of the hardware? that would be something cool to see ...
And a googleplex is 10^google. Useless info huh?
I know I'm not going to get this answered due to the very nature of the question, but I'd still love to know quite how you manage to keep pumping endless amounts of shit out about everything under the sun, and remain in character no matter what people post. And why the Steven W0sten imposter imposters? Damn fine troll anyway :)
Others might respect your trolling, but the only thing that matters in the end is high-karma--and you ain't got it.
I dunno, the trolls are pretty funny and karma is just another game really. Depending on my mood I'll play one or the other. But anyway, this Steven Wosten (and his alter ego, Steven W0sten) really do get people going. And he never comes out of persona either, which is unusual for a troll.
Plural subject requires plural predicate... and I would have pluralised 'dog' too, but that's a style issue.
Sorry, but you're wrong. The original post was an imperative, asking everyone and their dog to flame him, not a statement about what happens, so the original poster was correct :)
Neither one of these articles provided any interesting details about the Cluster. What language are they using? How efficient is the message passing (is it PVM or MPI)? What kind of benchmarks does this thing have? Is it pushing GFLOPS or TFLOPS? Does it beat out everyone in The Top 500 List. I just think some details about how great this system is would be nice, instead of just a RedHat advertisement! I know RedHat must be proud that their distribution is being used, but I would like some more concrete information than some sugarcoated marketspeak.
sid=lug
/^([Ss]ame [Bb]at (time, |channel.)){2}$/
As far as power density, most 1U systems pull 150 watts for dual 600-800Mhz CPUs. That gives you a total of 100 CPU's per rack, IMHO more power than an E10K, since we're dealing with an application whose individual threads probably don't utilize more than one or two CPUs at a time.
Steve
Please.. if you are a moderator and you feel that you should moderate something up, do so only if you understand what someone is writing.
Maybe I am too stupid to understand how a web backend can "break up apps like Real Audio or Napster" or I don't understand the significance of having "a low-lever packet assembly/disassembly at the POP" but if you ask me, this guy just fooled a bunch of you that thought his ramblings were Informative
Think before you moderate!
Now that the post is at +5, I sure hope it is all true and I am the fool in thinking that this is a fake..
Can we stretch this offtopic thread any further?
Will the NetAdmin ever call me with the mailserver password so I can do some work instead of wasting my time on /.?
Insanity is the last line of defence for the master diplomat. But you have to lay the groundwork early.
Plural subject requires plural predicate... and I would have pluralised 'dog' too, but that's a style issue.
Now it's my turn to receive a pedantic pummeling...Flame on, I'm gone.
"Don't do it Bart, don't do it!"
"Oh I gots to, I gots to."
Insanity is the last line of defence for the master diplomat. But you have to lay the groundwork early.
It's possible that the maintenance of the x86 farm is still cheaper than the loans for a big Sun server. If it reallly made a big difference cost-wise, they could have made it a Linux-on-Sparc cluster and still have the same PR effect for Linux.
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
Don't know about all of their bills but I think Virgin use a Google search engine on their home page. They paid for this but it's not branded as Google - just the technology underneath.
There's this as well....
Affiliate Programs that people can pay for to put a Google search utility on their site.
Google, one of the fastest growing search engines on the Web, operates its search engine and all of its computing functions on a cluster of more than 4,000 PCs running Red Hat.
That would sort of imply that it is indeed used as people's workstations.
J
Damn.. beat me to it...
i was going to put in the obligitory BW cluster of a BW cluster post... oh well..
just a thought, but if Google "chaches"(read: copies) pages of commercial & copeyrighted sites, wouldn't someone somewhere try to claim plagurism?
--DV
--DV
In this day it is safer to be a ninja than a samurai
But it is a cluster of 4000 PCs which means if one goes down the whole system keeps working. If you have one big Sun and it goes down you have no redundancy and no backup. Reliability and up time for websites is make or break.
take a triptonica to subthunk
I was only talking about clustering and redundancy. I'm not sure who rattled your cage but hey-ho...
> When you have a terabyte of data to process, you have to start paying a little more attention to things like I/O.
If you stripe a terabyte of data across 4000 disks and ask them all to search for it you will find it faster than if you have a single huge disk doing the searching.
>a wise business practice that anyone over 30 would recommend.
So why did they do it? Are you saying that google's tech support are all idiots?
>by all means invent some statistics that show that your MIPS/$ is better
Considering you started your argument with some statistics I'm surprised you say this. Did you read posts #178, which lists the prices, and #46, which is from a Solaris administrator?
>Oh, and BTW, your regex is suboptimal
It's supposed to be like because it is obfuscated. Have you not had your morning coffee or something?
take a triptonica to subthunk
I'd love to see a picture of their machine room! (What do they use in terms of networking kit?)
Anyone know how Google can afford a 4000 node cluster? I mean, there are like no ads, no subscriptions, etc.
But what if the cluster was wired with fiber and was used as a giant rendering farm for real-time ray-traced Quake (RTRTQ)?
My other
hehe
why am i here?
It is OT you dam troll :)
Actucally that is intresting, either way though, I still disproved the theory (with a little help), for example the theory clearly states "All actors are linked to Kevin Bacon is 7 steps or less." which this has been disproved by the "other 4" actors that can be linked in eight steps.
Now I only have to disprove a couple more things before my life is complete this first one was easy, here are some more things I need to disprove
1 ) 7 Steps of Kevin Bacon -- Complete
2 ) N != NP
3 ) Christ
4 ) Hilter
5 ) Lee Harvey Oswald was a patsy
6 ) 1 + 1 == 2
7 ) American aren't under Mind Control
8 ) God
9 ) Your parents where right
10) Ingornace == Bliss
The fisrt one seemed easy enough. I wonder if a cluster could help me with disproving any of the above.
btw, mod the above post up
"`Ford, you're turning into a penguin. Stop it.'" -THHGTTG
I know at places I have worked that many of our powerful computing systems have seti@home running. I know of a Sun Enterprise 450 that gets great stats. :)
Yeah 4000 pc's take a lot less current than a few E10k servers.
Only the State obtains its revenue by coercion. - Murray Rothbard
"bloat laden"? Which OSes are bloated, please list them all. Thanks
.... let me think ....
Errr
I can think of several. The real give-away is that they usually have the words "Microsoft Windows" somewhere in their name.
Stephen Hawking has written another book. It's about time as well.
The specialized hardware required is no more than any standard server room. Most decent server rooms have a minimum of a raised floor, several rack cabinets and climate control. The space required for these systems is not as much as it may seem.
Say you're using a 34U 19" rack to hold these servers. If they're using 1U servers for each node (very probably) then each rack holds 34 nodes. That's 118 cabinets. Each cabinet is approx 2'x3' and if we assume 2x's the cabinet size for total space (walkways and such) then that's only ~1400 square feet or the size of a large apartment. That's not abysmally huge for a server room. If you consider that they might be using 1Ux4 servlets (4 nodes per U) then we're down to 25% of that estimate.
Things could be much better or worse than I've depicted but the point is it's very doable in a reasonable amount of space with standard equipment.
As for your suggestion that they move to a high capacity single server or mainframe that's rather ludicrous. Assume they're paying $2500/node (a very large assumption) that's $10,000,000. From what I've read about mainframes that won't buy you much.
Have you bothered to check the list of the top 500 most powerfull super-computers? Several Linux clusters are on that list (and very high up the list at that) and these clusters were built for FAR less money than most (if not all) of their peers.
A well designed cluster system is very easy to maintain. You maintain one (or several) NFS root servers and every system remote boots. To change the configuration of one node you change it at one point. To install a new node you simply install the new hardware, make a couple of simple configuration changes and you're set. When you've done 4000 of them already another node probably takes less than 15-30 minutes to install.
-Zane
This sig is worse than my last.
A beowulf cluster of those?
You know you want to give me -1 Troll
Uh. (a) Where are you getting that a PIII would outperform an Ultrasparc? The MHz rating? (b) It is extremely lame to compare a distributed system with a parallel system as if the number of processes is going to be the only factor that affects performance. You are forgetting about the fact that memory and disk information passing has to happen over a network rather than within a high speed bus like you find in a parallel computer. This causes a lot of slowdown. The reason to go distributed is cost, but I would personally guess that an E6500 with 20 processors would be equivalent to something more on the order of 40 or 50 Intel boxes that are in a cluster. There are ways to reduce the impact of the clustering, but it will never be better than a parallel computer.
A quick check of this "Steve Woston"'s User Info page should give us all the info we need:
:) Sure, this guy is trollin, and making moderators look like fools, but at least he's original!
"I have worked on many ground-breaking products within the industry, including Spork II, Star Trek: The Fillament Legacy, and Pitfall for Nokia 5125"
Hehe, Pitfall for Nokia 5125
Many of the larger Sun systems? Try one. The E10K. And the term you are looking for is "Dynamic System Domains", not "virtual machines".
so that if one goes down the rest of the system can keep running while you replace that one processor board and then bring it up
True, unless the part that fails happens to be your backplane. Then the entire system must be powered down for replacement.
Use to have a link but lost it...
Try this one.
I was just wondering? No point at all, but thanks for your answer.
I like traffic lights
I believe they switched to BSD after they had been playing around with it as an experimental machine. They saw how stable it was (and cheap!) and switched over.
Well, technically supporting "OR" is redundant because you can always make two separate searches. You'd need two windows to combine results though and that is kind of annoying.
My monkey uses google too.
My *mother* uses Google. What more do you want? ;)
Not to be a bother, but 200 times 200 is 40,000.
Now that's what I call a Beowulf cluster. :)
Jouni
--
Jouni Mannonen : 3D Evangelist @ SurRender3D.com
Jouni Mannonen | Game Designer, Consultant
Why not just make moderation based on absolute levels. ie. If Moderator B sees post #42 at (2), then have the choices to moderate it to (1) and (3). They choose (3), but in the meantime Moderator A has changed it to (3) already. They submit, and because the change has already been affected, nothing happens. They retain moderation points, and possibly a notice appears at the top of the next page informing them that /. didn't need to use their points.
You'd better ask Steve Woston. He knows.
That said, I think it's time I changed my
And besides, what does this have to do with clusers?
That said, I think it's time I changed my
Hmmmm. No. I don't think clusters are suited to the type of application in question. Quake wouldn't really benefit from massive parrallel processing. Yes, from more processing power - as long as it's on a local bus and not distributed over a network into seperate nodes. The main factors in Quake speed would be Random Access Memory and the quality of graphics adapter used in the comptuer running the game.
I think Beowulf is an excellent technology for many applications though, especially back-end services which need the extra oomph and can be distrubted cleanly, as well as, of course, some academic processes, such as analysis, etc.A good example of how Linux can be made into a model of distributed service handling is the impressive HA Linux. (High Availability Linux). The team is working closely with the Linux Virtual Server project, and the technology looks impressive. In a few years it could even compete with Sun's high end technology.
-
Sun Certified Programmer for the Java Platform
- Sun Certified System Administrator for Solaris
Cedric Balthazar Rotherwood
Sun Certified Programmer for the Java Platform +
System Admin. for Solaris
Um...that's nice, I guess...
Ah, forget it. Ahem, "HOLY OFFTOPIC BULLSHIT, BATMAN!!!"
Thank you for your attention.
I went to
But hey, there's always distributed.net...
I went to
Umm, yeah. I've seen a dead sun before too. While the situation was sad, I wasn't inspired to write any poems.
Hello? Sure Sun boxes are more reliable, but x86 boxes just don't fail everyday.
Many execs and IT leaders are still afraid of Linux, while at the same time relying on services like Google to perform their work.
This article is the perfect eye-opener for these "going 100MPH with blinders on" types!
I am the lead programmer for j-j-j-julius, not head of the IS department (although I could be). Please speak with him about our website being out.
You're a little late on number 6. 1+1 has been shown to be equal to 2, but the proof took 211 pages in Principia Mathematica . You can also find the proof for 2+2=4 here. 2+2=4 is obviously related to 1+1=2, with a few extra steps.
...phil
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
--
I doubt if they would "patch it back." In case you haven't noticed, most companies haven't given a rat's ass about impressing the OSS crowd until very recently. Something tells me they really are running FreeBSD.
You could still be right though... this only means they are running their frontend web servers on BSD. As to what powers their database is anyone's guess, since you'll never see that server from the outside world if they have half a brain. So, they could be running NT for their backend. This would be the totally wrong thing to do though. Usually companies use NT to serve the HTML because it has better applications for interactivity available than Unix, and a *nix for the real meaty, hardcore database queries, etc. I believe this is what Ebay does but I could be mistaken.
--
I think there is a world market for maybe five personal web logs.
This presented some unique problems, tho. Using 300 nodes meant that, potentially, you could have 300 connections to EACH CLIENT. We needed to make a transparent single point of entry and use a 10.X.X.X ->legal NAT translation. Problem with that, of course, is that NAT often breaks apps like Real Audio or Napster or anything that embeds source/destination within the packet to be router through the routing level of the requestor.
Using NAT as a front-end to a server farm returning straight HTML documents won't cause any problems.
Go away!
I forwarded a copy of the above post to Rob Malda, and he sent me a concise reply describing his view of how it's supposed to work. I think it's worthwhile to share his insights with the whole crew. With his permission, here's what he had to say:
So there you have it. Of course, that does raise the question of why we have the Overrated and Underrated moderation categories, but otherwise, I think I see his point.
--Joe--
Program Intellivision!
At this time, it appears to have been rated back down to a 3. I think what happens is that moderators scan / read through posts, selecting particular posts to be moderated up or down. When they finally get to the end of the page, they click [Moderate]. When several moderators are actively viewing a story, you end up with multiple moderations pending for the same article. So, what should've received a +1 might get +2 or more of multiple moderators agreed that it deserved +1.
The problem is that the moderators don't get to see the other moderations being performed in parallel to their own moderation. Perhaps there's a solution. Slashdot could ask for confirmation in cases of "moderator collision."
For example, consider the following sequence of events:
Currently, Slashdot will apply both moderations immediately. This results in article #42 receiving +2, when it may only deserve +1. It's neither Moderator's fault -- they've moderated past each other. Alternately, I propose that Slashdot, in this case, only apply the unique moderation immediately, and then ask for confirmation on Moderator B's moderation of #42. This is because Moderator B had no way of knowing that Moderator A moderated #42 up while he was still reading the posts. Let's assume all moderations are applied, and continue the example:
At this point, Slashdot will apply the moderation. Under my proposal, this would not change, as Moderator C did already see that #69 was moderated up before he selected it for moderation.
What I'm guessing would be necessary is an additional bit of state which says "This was the score that the post was viewed with at the time the Moderator selected it for moderation." If the article's current score is different than the score it was viewed with, ask for confirmation that the moderation be applied for that specific moderation. A series of radio buttons could be displayed for the affected articles: "Apply Moderation? [_] Yes [X] No".
Thoughts?
--Joe--
Program Intellivision!
"PCs will work OK in any heat and humidity that people will"
I pictured myself framing a house in the Texas summer heat, and repairing a barbed wire fence in a snowstorm.
-fb Everything not expressly forbidden is now mandatory.
" the guy who kicked your fucken ass "
If I had moderator points, I'd deal you down accordingly. Since I don't, I'll mention this:
I think that "fucken" is becoming a word. I'm glad it is, because it rhymes with "Turducken". I also think it would work in a subjunctive mood usage context.
-fb Everything not expressly forbidden is now mandatory.
for just a 2 or 4 node cluster, you buy a high-quality PC from VA or some other reputable shop that supports Linux well. once things start to grow, you use those for database, load balancing monitors and things like that, and you grab el cheapo clones for the gruntwork of running httpds.
I knew had to be some heavy-duty equipment back there -- nothing else but a 4,000 node Beowulf cluster could power the awsome "Mentalplex" search engine. It's unfortunate that the search also requires the combined mental powers of 4,000 users. Which might be why I can't seem to get the Mentalplex to find anything but pr0n and mp3s. :).
.....oooh, swirly..."
"Must... Concentrate!
Allthough I use Google mostly, it is not the fastest engine around, that has got to be Fast (http://www.alltheweb.com)!
It is so damn fast, that it just keeps amazing me.
If you haven't tried it - you should!
Here is an example on a seach for "linux":
"3810249 documents found - 0.0051 seconds search time".
I would like to put one of these in my basement and finally disprove the "7 steps to Kevin Bacon" theory everyone seems to buy into.
It's usually 6 steps to kevin bacon, and it's an NP complete problem. If you do find a way to solve it in polynomial time, please share your algorithm. You'll probably get a Nobel prize.
--Shoeboy
(former microserf)
There are ways to reduce the impact of the clustering, but it will never be better than a parallel computer.
That's complete bunk. Whether a centralized multiprocessor machine or a massively-parallel distributed cluster would be faster depends completely on the task at hand. Specifically: How parallel is the task?
If the task can be broken up into many completely self-contained pieces, then a cluster will generally win. You can buy lots of low-end hardware cheaper then you can buy even very good high-end hardware.
If the task contains contention points or data access is very random, then you're better off with a single multiprocessor machine. An example of a contention point would be the locks in a database. An example of random data access would be logins to Slashdot.
Finally, it is worth pointing out that, after a certain point, most large machines have to move to a NUMA design, at which point you start to resemble a massively parallel cluster anyway.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Take a look at your user history. All your posts eventually get looked at by moderators not smoking crack and get modded down to 0 or -1. At best you are entertaining yourself for a few minutes with a single temporarily, high-modded post at a time.
Others might respect your trolling, but the only thing that matters in the end is high-karma--and you ain't got it.
BTW, don't bother responding with a "what are you talking about, I'm not a troll" response: I don't intend to read it.
--
Have Exchange users? Want to run Linux? Can't afford OpenMail?
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
Still an Open Source victory!
Alltheweb.com is running "Apache/1.3.6 (Unix) PHP/3.0.11" on FreeBSD...
When I first saw the "powered by Dell Poweredge" sticker on their page, I briefly worried that it was going to be an NT site. Nope!
Napster-to-go says "Fill and refill your compatible MP3 player", which is a lie. It's not MP3. It's WMA with DRM.
It's all well and good that there's PR out there about this.
As someone who is building a large portal with Redhat, it'd be nice to have some kind of technical reference as to how they've built it. What are they using to handle the clustering? Are they using the Piranha stuff that comes with Redhat 6.2, or are they using hardware, or maybe something they've written themselves? Are they using sessions, and if so how are they handling them?
Are any parts of the cluster sharing processing power, or they all just individual boxes clustered to appear as one?
I think it's great that they're getting press, I'm just hoping that one of these days there will be something published on how it all went down.
No, I wasn't trying to say that google makes a good poster boy for open-source, but it is a great example of large organizations embracing the fruits of open source labor. Linux has gone through a lot of media exposure in recent months due to it's current 'fashionable' status, what is actually need to maintain linux's spot in the media world is examples of linux doing real world jobs. Large companies like google making public statements that they use a massive linux installation to solve a problem because it's the best tool for job are not going to hurt.
I would be interested to hear more about "The troubles that go on at Google behind the scenes are bound to become public knowledge very very soon.", without further information I'd like to think that linux would not get a 'black eye' over any problems within google, but you seem to know more about this than me.
Only 2 of the modules for the entire system are implemented in python, specifically the web crawlers and the server that feeds the crwlers url's. the rest of the system is implemented in c or c++.
raging.com
(seems to come up with slightly diff hits than Altavista itself, but works plenty good for me!)
a Beowul....Oh, wait a minute, never mind ;)
John Saul Montoya (Yeah, Wosten, thatJohnny Montoya, the guy who kicked your fucken ass over the KKW second-stage funding. Don't fuck with Wall Street).
-- the most controversial site on the Web
BTW, have you looked at the http://www.hotsheet.com/ portal? It's a portal, yeah, but it's really "clean" looking and has a ton of useful links. That's why they host my email. (no, I'm not affiliated)
----
Did you say that with a straight face?
Assuming you depreciate a machine over three years (and that's really stretching things in the Real World), you're replacing a machine every just over every six and a half hours. Plus all the effort gets skewed down the the end of the three years. It would almost be economical to throw the door-key away and start afresh.
When you buy a Sun the damned thing just doesn't fall down unless you have a system mangler who keeps dicking around with it. And if a single Sun could not address the problem, then maybe it's time to buy some real iron, like a maxed out S/390. When you have a terabyte of data to process, you have to start paying a little more attention to things like I/O.
4000 PCs cannot be a viable economic replacement. That amount of hardware would require as highly a specialised environment as that of a mainframe (cooling and electricity), and certainly much more real estate. And they have really shitty I/O. If Google has money and space to piss away, well good for them, but it's hardly a wise business practice that anyone over 30 would recommend.
If you want to play with Linux, by all means invent some statistics that show that your MIPS/$ is better than the competition. Statistics can say anything you want them to. I, however, would like to know how they derived such figures. Ignorant readers of the article might otherwise be mislead into pursuing foolish choices in computing platforms.
Oh, and BTW, your regex is suboptimal, the split is entirely redundant and you shouldn't use double-quoted strings in Perl if you're not interpolating anything.
An interesting quote is this:
The Sun solution would be much more expensive because it wouldn't be only one Sun. It would require many, many, Sun 6500's or 10000's. Since their application distributes quite nicely, the price/performance of Intel boxes running Linux would be very hard to beat.
;) (In practice, the ratio would probably be closer to 12-15 Intel boxes per Sun 6500, I would guess, as a PIII doing it kind of integer work would likely outperform a SPARC II)
Try substituting Sun 6500's with 20 CPU's for each set of 20 Intel boxes and see what that does to the pricing.
Geeky modern art T-shirts
But in this case I think Google is on the right track. MIPS/$ ratio is definately in the favor of the PC. And with sooo many PC's if one goes down it really wouldn't make a huge difference. If it were just a 2 or 4 node cluster then I would lean towards a RISC based architechture for reliability. But in this case the cost is just to staggering to imagine a Sun cluster for this.
Koodoos to Google, my new search engine of choice! Long live Linux!
Google does offer phrase searches, and a few other advanced features. Just click on the Search Tips link from the main page. I'm not sure I'd classify their implementation as "intuitive," but it's no worse than learning, say, REXX. You are correct though, in that full Boolean searching is not available -- as stated on the Tips page, Google does not support the logical or operator at all.
Slightly disreputable, albeit gregarious
Vintage computer games and RPG books available. Email me if you're interested.
It's nice to see some good Linux publicity happening, Google is fast becoming the most respected search engine around, their clean and uncluttered interface is drawing people away from the more traditional search engines where it seems you have to download more portal c$&p every day. It seems poetic the google is becoming an ambassador for linux by showing up their bloat laden competitors in the search engine market, while linux does the same in the OS market.
I can just see it now. A manager at Google walking over to a developer's PC and seeing this sticker and saying,"Why not?"
:)
Now all that's needed is for thinkgeek to claim responsibility for this action.
So this "super computer" will be used for Total World Domination? Oh, can we use it atleast to take over some small thrid world countries? I promise to have it back by six tonight.
The Google crew must have some killer Seti@home stats.
I would like to put one of these in my basement and finally disprove the "7 steps to Kevin Bacon" theory everyone seems to buy into.
"`Ford, you're turning into a penguin. Stop it.'" -THHGTTG
redundant array of inexpensive processors
ok then your [sic] infringing on my copyright! Could you as [sic] me next time before STEALING my comments for your own?
But damn, that takes a staff of 200 people to manage the security/connectivity/accounts/space and other duties just for the cluster.
The Power bill has to be outrageous!
The Cabling/switching/routing mess has to be totally unmanageable
What happens when you reach a buck in the hardware or have to patch the system or replace a kernel because of a hack that came about? It is costly and hellish to work on 4-6,000 pcs
I would have thought it to be wiser to setup Sun E10000's or something like that.. having 4 32 proc e 100000's in a cluster is a hell of alot easier to manage and cheaper. Sure your upfront bill may be more, but only have to worry about 8-16 power connections (redudancy) is alot easier then 6,000 power cords/strips/racks/floor space/cooling/maintenance.
Sure it is one hell of a beast to be proud of, but one hellova costly beast to work with.
Just my 2 cents
just my own 10cents, The google guys use python over perl, hrmmm, i wonder why. :D by the way their paper is a good read. http://www7.scu.edu.au/programme/fullpapers/1921/c om1921.htm
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
I would have thought it to be wiser to setup Sun E10000's or something like that.. having 4 32 proc e 100000's in a cluster is a hell of alot easier to manage and cheaper.
Last I checked (this was about a year or so ago) a fully loaded (64/64) E10K ran around $12M and the base (2psr) system was running around $800,000. Even if that's off by a factor of 3 or 4, you're still talking $3-$4M a piece... at three of them, you're looking at between $12-$48M. On the other hand, the typical white box PC will run between $800-$1500. That amounts to $3.68M-$6.9M for 4600 nodes. This doesn't include the network infrastructure or administration costs, however, as someone who has administered large clusters (largest was an 80 node SP/2), it actually becomes easier to administer that many nodes in a cluster than it would that many servers. Keep in mind that there most certainly are groupings of nodes where they are kept identical except for IP.
Another significant expense is that hardware support costs associated with such systems. If you have 4600 nodes, it's trivial to simply keep (MANY) spare systems floating around. Also, you can disable a node with negligible impact. Even if you're subdomaining an E10K, there are (a small few) single points of failure on the platform (regardless of what Suns documentation says). If you're not subdomaining it, you're simply talking a 32way SMP box (might as well just use a 6500 for that configuration). If you were to lose the backplane for whatever reason, you've lost a singificant portion of your compute resources.