at that node count interconnect speed and latency becomes crucial factor and from what i know about google they have 3-4 data centers and servers connected via commodity interconnect (1000/100).
from my understanding of linpack their setup just isn't suitable for such workload.
It's true - individuals have reported receiving up to 6 invitations (Source: www.wired.com/news/infostructure/ 0,1377,63786,00.html?tw=wn_12culthead ).
At least one of people I invited did not open a Gmail account (the invitation was either forwarded or declined). I have two unused invitations (I won't use them 'cause I don't know a deserving individual to give it to) and I've invited 4 people so far. If we assume there's about 1m active accounts (say 3-4 racks of mail servers), there's probably been at least 10m invitations given away.)
>Except that the Third World would likely be excluded from the nuclear war, on the grounds of having no nuclear capability and thus not being a threat, and their infrastructure (what there is of it) would thus be left intact.
What infrastructure? Without the first two worlds, the 3rd would die off pretty soon.
> before we're sure that there isn't any life on it?
How _sure_ you need to be that there's no intelligent life on Mars? I think it's quite obvious already.
I read about some animal-rights nutcases protesting against any interference with existing life forms (!= intelligent life) to which a researcher commented "Let's get real - bacteria don't have no rights".
>For those who still think email is secure I got news for you: Your email is already exposed in plain text on just about any server it is sent to.
So what - I don't care about any single email being exposed to someone's eyes. But I do worry if someone has access to ALL of my email all the time - I even wouldn't care if they would read it, but I do not like to be analyzed and profiled via content of my email.
>No human will ever read your email.
This is really a stupid argument - OF COURSE they won't because they can't possibly get enough humans to read gazillons of messages in millions of accounts! It is impossible, it'd be too expensive and it wouldn't work because humans can't analyze huge amounts of data.
When will people realize that it is FAR WORSE that AI bots "read" (analyze, parse, whatever) your email because: a) they don't "forget" b) they can easily "read" all emails of all accounts c) it is cost effective (essentially free) to use them for any and all purpose they see fit.
I'm not claiming they are doing bad things with it, but they certainly have all the tools they need. So those who mind would probably prefer to keep their email on less creepy a place (one that doesn't analyze your contacts, URLs in your email and content of your messages to show you ads and whatnot).
The customer asks IBM which distro to use, they tell them use Whatever Linux and the customer says OK if you say so. I don't think that IBM tested SuSE and Red Hat and chose "the better" one. It's probably just a DB cluster and couple of WebSphere servers - not exactly space science.
Actually I just thought of something else - many IBM's Linux wins are people who used to buy IBM's UNIX servers. Now they don't want UNIX any more, so IBM tells them we'll give you Linux. They charge them a lot for services and the customer still manages to save some money.
But what will happen the next time such customers want to upgrade? If they don't change the OS, they won't need special compatibility and other assurances from IBM any more. Any hardware will do and any service provider too.
By then, IBM should get their utility stuff right or else their margins will get squeezed by HP and others (barriers to entry will be very low).
I wondr if such "enforcement" is good enough. Probably they could prove lack of due dilligence in establishing true identities of contributors. Extensive/sufficint self-policing, on the other hand, would cause serious overhead in managing such projects.
The internet. I never bought a book or attended a workshop or anything like that. I do not claim to be an expert, but I would do my own research and then ask one-two very specific questions on a specialized forum. The way the question is asked makes it clear the poster hasn't done his homework and expects others to do it for him.
Okay maybe I was too harsh in the way I said it, but really, this question belongs to a newsgroup/forum for beginners. I believe most Slashdotters prefer to see a "how I created a high performance render farm" rather than "what is a high performance render farm" kind of article.
Existing answers to this kind of question abound on the Internet, if you do a simple search you'll find the 1st result ("Need help for building and setup a render farm" at highend2d.com) to be exactly the same like this question: search Google for: render cluster maya nodes small
I think you can use this CFS to create a directory tree with over 200TB of data (/home/lun1,/home/lun2,.../home/lun255). You can't "tie" them together like with LVM but you do get huge throughput as opposed to a single-host bottleneck with a volume manager and/or clustered NAS filers with the hot spot problem.
>Get systems with massive amounts of I/O that will cope with all the data you're trying to throw at them.
Render nodes get input via simple render scripts, output frames get written to the file server one by one every X seconds as they get rendred. Textures are shared but it's never "massive" and never "thrown at them" (compute nodes). The I/O loading is concentrated on the file server.
> Don't bother with Intel/Linux, with dodgy hardware and the frequently-changing Linux code.
So HP, IBM and other Intel/Linux servers that rendered all those movies are "dodgy hardware"?
>Get SGI, get fibre channel, and (possibly) get gigabit ethernet.
I don't think 8 nodes can write and read data quickly enough to saturate a gigabit link to NFS server since while frames are being rendered NFS' I/O is very low. With that number of nodes perhaps they could muddle thru without external storage (maybe even internal SCSI would do). GbE is more important (and cheaper) than FC storage so I'd say GbE is a must.
n1 n2 n3 n4 n5 n6 n7 n8
[gigabit lan]
nfs1 (optionally with direct-attach FC disk array)
First, that is a very small cluster with embarrassingly parallel compute tasks (rendering of individual frames), so you should be able to find the answers on your own using Google.
Secondly, from your questions it is obvious you're no rendering farm guru so why did the task of planning/researching the configuration get assigned to you? You should rather work with the users to collect and write down the goals (what is the purpose of this cluster, how do users use it, what are their expectations, how it's going to be managed, etc.) and find a small and focused SI who will propose you a better solution than you can come up on your own even with help of Slashdotters.
It's impossible to give a good answer to your questions since you've provided very little info, but in addition to the render nodes, one NFS node for file server and a single gigabit network will do.
Don't skimp on the software and don't use F/OSS just because you "prefer" it (Why do you prefer it? It doesn't sound like you're going to dive into the source and improve it or something.) Instead of focusing on the coolness, focus on creating a user-friendly and maintenance friendly system. Do geeky stuff at home.
One of those hosting-oriented sites did a tour of a facility that hosts one of these DNS root servers (if I remmber well, it was very well protected - something like old military facility - and it looked to me it would take a nuke or an ICBM to bring it down.)
>The upshot of this is that it makes Linux all that more attractive since there's no per processor licensing bullshit to deal with,
Actually licensing for such commercial apps doesn't change - it remains per physical CPU, no matter how many cores the CPU has.
>It's pretty easy to predict that what will happen is Microsoft will be forced to amend or drop its per processor licensing scheme,
As above - if that does happen, the technology itself won't be the reason.
>Then again, they might have enough clout to hold up AMD and Intel's release of the multi-core CPUs...
What? Xeon MP has been out for years and with Xeon MP-based servers, such as IBM x440, when you type "top" you see twice as many CPUs. These of course are supported in Windows as well - use Task Manager to see it in action.
I believe they do have the infra and I think the problem is it's Network Appliance-based and hence outrageously expensive!
1GB of NetApp NFS vs. 1GB of Linux NFS
1GB of Linux NFS is essentially priced same as HDD (per-GB cost - take a 200GB NFS server and cost of the box becomes miniscule as it breaks down to 1/200th per GB).
>Offtopic, but why does google engender a warm fuzzy feeling of trust whereas yahoo, hotmail and the rest "feel" like corporates out to make a quick buck? It's a totally false feeling, but it's happens...
Wait till they go public. Today while typing a personal (naughty) message to friends on Gmail, I did feel uneasy knowing that I'll be "profiled" by AI bots, so I toned down the language:-) Really creepy!
Originally I wanted to use Gmail for my personal stuff ('cause it "never" goes out of date) and Yahoo for memberships, resumes, etc., but I'm thinking about reversing my decision and doing the opposite.
Well, for all we know, Yahoo might be doing the same, but still, at least I don't know about it.
It's fun to read stories like these. Reminds me of waits with my first PC (DX-25) - I used to do desktop publishing back then, it'd take it 20 minutes to repaginate margin change (that was before I upgraded to 4MB RAM, though)
There are several types of customers who do that, I believe - especially big customers (where it makes a big difference) and customers with static workloads (high performance, etc.) - you do it once, you do it right, and in the end it does help.
Admittedly it'd be hard to deal with live data (databases, open files, etc.) using your system, but it's a good idea. I like the fact that you provide both Windows and Linux versions. I might give it a try! Thanks.
Incidentally currently I'm a (primarily) Windows user and I do patch (actually it's "install updates") when Windows tells me they're ready (if I estimate I need the particular update).
Claiming that Windows users "don't care" just because they're Windows users is incorrect, to say the least. How can people mod that as insightful? Generalization like that should be discouraged as it is not constructive, but some actually reward it... Quite puzzling to me..
WTF? What kind of insight does the parent post provide?
>I wonder why there's no comparison of other desktop/newbie oriented distros, such as Xandros and Lycoris?
Because these distributions are marginal and have miniscule market share which renders them unimportant, that's why. Besides you quoted them yourself - "the best Linux products" - which part of that you don't understand? (Don't tell me people want to browse a document with 100 distributions compared one next to each other so that they can make a "choice" - that'd be laughable).
>Seems like that's the real competition to Linspire.
No it's not at all, because we know even though these two are free, they can't get more than 1% market share (combined).
The real competition to Linspire is Windows because Windows uses (most of them anyway) do pay for software considerably more than Linux users. Or do you really think Linspire plots to steal the big bucks that Mandrake is making selling their CDs?
Slashdot Mirror
i find that hard to believe.
at that node count interconnect speed and latency becomes crucial factor and from what i know about google they have 3-4 data centers and servers connected via commodity interconnect (1000/100).
from my understanding of linpack their setup just isn't suitable for such workload.
i'd go and do that if someone pays me to
> They had refused to consider that people might run a business on something that they could download free from the Internet.
well the article writer refused to consider that most enterprise hw and sw does not support freely downloadable Linux any more so...
Why is the parent post modded -1?
It's true - individuals have reported receiving up to 6 invitations (Source:
www.wired.com/news/infostructure/ 0,1377,63786,00.html?tw=wn_12culthead
).
At least one of people I invited did not open a Gmail account (the invitation was either forwarded or declined).
I have two unused invitations (I won't use them 'cause I don't know a deserving individual to give it to) and I've invited 4 people so far.
If we assume there's about 1m active accounts (say 3-4 racks of mail servers), there's probably been at least 10m invitations given away.)
>Except that the Third World would likely be excluded from the nuclear war, on the grounds of having no nuclear capability and thus not being a threat, and their infrastructure (what there is of it) would thus be left intact.
What infrastructure?
Without the first two worlds, the 3rd would die off pretty soon.
> before we're sure that there isn't any life on it?
How _sure_ you need to be that there's no intelligent life on Mars? I think it's quite obvious already.
I read about some animal-rights nutcases protesting against any interference with existing life forms (!= intelligent life) to which a researcher commented "Let's get real - bacteria don't have no rights".
>To find out if this is true, borrow the disc from some sucker who actually bought it, and can't use it. By all means do not buy it yourself.
I don't understand - you can't use a borrowed original disc and yet you suggest that people don't buy it?
So how can one listen to it if he doesn't buy it and he can't use a borrowed CD?
>>I don't know, but I certainly wouldn't complain to see Novell take back a sizeable bite of the business that was stolen from them.
>It was not stolen from them, they gave it away
Uhm, they had stolen that market share from someone else before Microsoft stole or took it from them.
What comes around, goes around.
>For those who still think email is secure I got news for you: Your email is already exposed in plain text on just about any server it is sent to.
So what - I don't care about any single email being exposed to someone's eyes. But I do worry if someone has access to ALL of my email all the time - I even wouldn't care if they would read it, but I do not like to be analyzed and profiled via content of my email.
>No human will ever read your email.
This is really a stupid argument - OF COURSE they won't because they can't possibly get enough humans to read gazillons of messages in millions of accounts!
It is impossible, it'd be too expensive and it wouldn't work because humans can't analyze huge amounts of data.
When will people realize that it is FAR WORSE that AI bots "read" (analyze, parse, whatever) your email because:
a) they don't "forget"
b) they can easily "read" all emails of all accounts
c) it is cost effective (essentially free) to use them for any and all purpose they see fit.
I'm not claiming they are doing bad things with it, but they certainly have all the tools they need.
So those who mind would probably prefer to keep their email on less creepy a place (one that doesn't analyze your contacts, URLs in your email and content of your messages to show you ads and whatnot).
Why? That's irrelevant.
The customer asks IBM which distro to use, they tell them use Whatever Linux and the customer says OK if you say so. I don't think that IBM tested SuSE and Red Hat and chose "the better" one. It's probably just a DB cluster and couple of WebSphere servers - not exactly space science.
Actually I just thought of something else - many IBM's Linux wins are people who used to buy IBM's UNIX servers. Now they don't want UNIX any more, so IBM tells them we'll give you Linux.
They charge them a lot for services and the customer still manages to save some money.
But what will happen the next time such customers want to upgrade? If they don't change the OS, they won't need special compatibility and other assurances from IBM any more. Any hardware will do and any service provider too.
By then, IBM should get their utility stuff right or else their margins will get squeezed by HP and others (barriers to entry will be very low).
I wondr if such "enforcement" is good enough.
Probably they could prove lack of due dilligence in establishing true identities of contributors. Extensive/sufficint self-policing, on the other hand, would cause serious overhead in managing such projects.
The internet. I never bought a book or attended a workshop or anything like that.
I do not claim to be an expert, but I would do my own research and then ask one-two very specific questions on a specialized forum.
The way the question is asked makes it clear the poster hasn't done his homework and expects others to do it for him.
Okay maybe I was too harsh in the way I said it, but really, this question belongs to a newsgroup/forum for beginners.
I believe most Slashdotters prefer to see a "how I created a high performance render farm" rather than "what is a high performance render farm" kind of article.
Existing answers to this kind of question abound on the Internet, if you do a simple search you'll find the 1st result ("Need help for building and setup a render farm" at highend2d.com) to be exactly the same like this question: search Google for:
render cluster maya nodes small
Try a cluster file system.
"Filers" create "hot spots" whereby often-accessed directories/files create IO bottlenecks.
I think you can use this CFS to create a directory tree with over 200TB of data (/home/lun1, /home/lun2, .../home/lun255). You can't "tie" them together like with LVM but you do get huge throughput as opposed to a single-host bottleneck with a volume manager and/or clustered NAS filers with the hot spot problem.
>Get systems with massive amounts of I/O that will cope with all the data you're trying to throw at them.
Render nodes get input via simple render scripts, output frames get written to the file server one by one every X seconds as they get rendred. Textures are shared but it's never "massive" and never "thrown at them" (compute nodes).
The I/O loading is concentrated on the file server.
> Don't bother with Intel/Linux, with dodgy hardware and the frequently-changing Linux code.
So HP, IBM and other Intel/Linux servers that rendered all those movies are "dodgy hardware"?
>Get SGI, get fibre channel, and (possibly) get gigabit ethernet.
I don't think 8 nodes can write and read data quickly enough to saturate a gigabit link to NFS server since while frames are being rendered NFS' I/O is very low.
With that number of nodes perhaps they could muddle thru without external storage (maybe even internal SCSI would do). GbE is more important (and cheaper) than FC storage so I'd say GbE is a must.
n1 n2 n3 n4 n5 n6 n7 n8
[gigabit lan]
nfs1
(optionally with direct-attach FC disk array)
First, that is a very small cluster with embarrassingly parallel compute tasks (rendering of individual frames), so you should be able to find the answers on your own using Google.
Secondly, from your questions it is obvious you're no rendering farm guru so why did the task of planning/researching the configuration get assigned to you? You should rather work with the users to collect and write down the goals (what is the purpose of this cluster, how do users use it, what are their expectations, how it's going to be managed, etc.) and find a small and focused SI who will propose you a better solution than you can come up on your own even with help of Slashdotters.
It's impossible to give a good answer to your questions since you've provided very little info, but in addition to the render nodes, one NFS node for file server and a single gigabit network will do.
Don't skimp on the software and don't use F/OSS just because you "prefer" it (Why do you prefer it? It doesn't sound like you're going to dive into the source and improve it or something.)
Instead of focusing on the coolness, focus on creating a user-friendly and maintenance friendly system. Do geeky stuff at home.
One of those hosting-oriented sites did a tour of a facility that hosts one of these DNS root servers (if I remmber well, it was very well protected - something like old military facility - and it looked to me it would take a nuke or an ICBM to bring it down.)
>The upshot of this is that it makes Linux all that more attractive since there's no per processor licensing bullshit to deal with,
Actually licensing for such commercial apps doesn't change - it remains per physical CPU, no matter how many cores the CPU has.
>It's pretty easy to predict that what will happen is Microsoft will be forced to amend or drop its per processor licensing scheme,
As above - if that does happen, the technology itself won't be the reason.
>Then again, they might have enough clout to hold up AMD and Intel's release of the multi-core CPUs...
What? Xeon MP has been out for years and with Xeon MP-based servers, such as IBM x440, when you type "top" you see twice as many CPUs. These of course are supported in Windows as well - use Task Manager to see it in action.
I believe they do have the infra and I think the problem is it's Network Appliance-based and hence outrageously expensive!
1GB of NetApp NFS vs. 1GB of Linux NFS
1GB of Linux NFS is essentially priced same as HDD (per-GB cost - take a 200GB NFS server and cost of the box becomes miniscule as it breaks down to 1/200th per GB).
>Offtopic, but why does google engender a warm fuzzy feeling of trust whereas yahoo, hotmail and the rest "feel" like corporates out to make a quick buck? It's a totally false feeling, but it's happens...
:-) Really creepy!
Wait till they go public.
Today while typing a personal (naughty) message to friends on Gmail, I did feel uneasy knowing that I'll be "profiled" by AI bots, so I toned down the language
Originally I wanted to use Gmail for my personal stuff ('cause it "never" goes out of date) and Yahoo for memberships, resumes, etc., but I'm thinking about reversing my decision and doing the opposite.
Well, for all we know, Yahoo might be doing the same, but still, at least I don't know about it.
It's fun to read stories like these.
:-)
Reminds me of waits with my first PC (DX-25) - I used to do desktop publishing back then, it'd take it 20 minutes to repaginate margin change (that was before I upgraded to 4MB RAM, though)
Good old times
>1) Your computer gets wet
= articles/s ubmersion/submersion.html
They could submerge it (and still make it look nice) instead:
http://www.octools.com/index.cgi?caller
>2) The chernobyl effect. Assuming it's survived this long, the coolant's now gone, and the computer keeps getting hotter. Uh oh.
It should be a closed system; I presume there isn't a water hose leading to the guy's kitchen?
There are several types of customers who do that, I believe - especially big customers (where it makes a big difference) and customers with static workloads (high performance, etc.) - you do it once, you do it right, and in the end it does help.
Admittedly it'd be hard to deal with live data (databases, open files, etc.) using your system, but it's a good idea. I like the fact that you provide both Windows and Linux versions. I might give it a try! Thanks.
>And this is a search problem,
A detection problem, actually.
They can _find_ spam (and all other) messages, the problem is how to tell which ones are not legitimate while keeping false positives at minimum.
>Windows users don't tend to care.
Or "Windows users tend not to care?"
Incidentally currently I'm a (primarily) Windows user and I do patch (actually it's "install updates") when Windows tells me they're ready (if I estimate I need the particular update).
Claiming that Windows users "don't care" just because they're Windows users is incorrect, to say the least.
How can people mod that as insightful? Generalization like that should be discouraged as it is not constructive, but some actually reward it... Quite puzzling to me..
WTF? What kind of insight does the parent post provide?
>I wonder why there's no comparison of other desktop/newbie oriented distros, such as Xandros and Lycoris?
Because these distributions are marginal and have miniscule market share which renders them unimportant, that's why.
Besides you quoted them yourself - "the best Linux products" - which part of that you don't understand?
(Don't tell me people want to browse a document with 100 distributions compared one next to each other so that they can make a "choice" - that'd be laughable).
>Seems like that's the real competition to Linspire.
No it's not at all, because we know even though these two are free, they can't get more than 1% market share (combined).
The real competition to Linspire is Windows because Windows uses (most of them anyway) do pay for software considerably more than Linux users.
Or do you really think Linspire plots to steal the big bucks that Mandrake is making selling their CDs?