Slashdot Mirror
Distributed Computing Economics
machaut writes "In a ClusterComputing.org article, Jim Gray, director of Microsoft's Bay Area Research Lab, provides an interesting economic analysis for building distributed systems. When do you choose a grid over a cluster or a supercomputer?
When does it pay off to move a task to the data vs moving the data to the task? He takes current hardware and networking costs into account to answer those questions."
.. have already figured it out - let other willing users pay the power bill, bandwidth cost, etc. and crunch the data in their spare time. Seems to be working well for seti@home, etc.
Of course, if you are working with sensitive data (military stuff, major trade secrets, etc.) your security/privacy needs will outweigh the costs involved with doing it all in house.
Don't blame me, I voted for Kodos
interesting thought, but what is the difference between this and the age old concept of the cost/benefit relationship...? im not trolling, it seems that it is jsut that concept with a tech twist..
xao
http://TheHillforum.hopto.org
All the work must(should) be double checked to make sure everything is correct
Cats: All your base are belong to us.
Captain: Take off every sig !!
From the article :
...
1 Mbps WAN link $100/month
From this we conclude that one dollar equates to:
$1=
1 GB sent over the WAN
Oh yeah ? 1 Mpbs for a month == 2678400Gb per month == 334800GB per month. 334800GB / 100 == 3.348TB
From this I conclude that one dollar equates to 3.348 terabytes sent over the WAN.
Gosh that was scary. I can restart xMule now
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
Just last year we were discussing data transfer over the time it would take to overnight some data in a package, worked out that it was faster and wouldn't clog up our line to burn the DVDs and send them through an international package service vs send it over the T1s. I think with all but the largest businesses this is probably still true for larger (Gigabytes) amounts of data. Network costs are too high to be putting data far from where it is to be used. Whether CEOs realize it or not, this has a great effect on the ways businesses with multiple locations structure their company and work together.
That shows how this analysis is done from the perspective of the party performing the investigation, as opposed to society as a whole. For instance, Seti@Home's costs in terms of user electrical, maintenance, etc. isn't considered here...
Stop by my site where I write about ERP systems & more
This is an old maxim of design of any multi-tiered system. The reason is this: computation is largely about selecting and filtering data, before sending the results on to further tiers. This selection and filtering process requires many times more bandwidth towards the data source than it does towards the client layers.
This only stops being true when there is no significant data, i.e. when the computation creates the data, as in the author's examples of render farms.
Ceci n'est pas une signature
It's a nice piece of analysis. Someone could have done it 8 years ago when Java came out; the facts are not significantly different (The values are different of course but the ratios involved are pretty similar. I did some thinking along these lines back then, and then in 2000 when considering working for a "hot P2P company" that an old acquaintance of mine was running.)
My thinking went something like this: There are only a few, "niche" applications which need more compute power and which people pay for (distributed rendering, CFD, FEA, maybe a couple others). Maybe you could build that into a 10-30 million dollar business if you overcame a zillion obstacles but it didn't look like a billion or multi-billion dollar business. The applications for which people buy beefy servers, and which have a monetary payback, are mostly database applications. For those, you need to move the entire database near to the number-crunching PC, and that's not really feasible due to the cost of transporting Gigabytes of data or the unlikelihood that the PC's hard disk can store all the giga/terabytes of information potentially relevant for the computation. Not to mention the security problem.
And Jim Gray's analysis lays out in more precise economic terms why it doesn't make sense. I like how he even calculated the relative merits of a Beowulf-like cluster of PCs versus P2P which I never really analyzed (I lumped them together as basically similar.)
That said, has anybody even made a stab at designing or implementing a relational database with a P2P architecture? I know that there's Oracle Cluster Server, but I'm thinking of something more low-end and more distributed.
--LP
The recurrent theme of this analysis is that "On Demand" computing is only economical for very CPU-intensive (100,000 instructions per byte or a CPU-day per gigabyte of network traffic) applications.
This must be considered an attack on IBM's fairly visible On-Demand Computing campaign.
Beowulf clusters have completely different networking economics. [...] That is why rendering farms and BLAST search engines are routinely built using Beowulf clusters.
This reminds me of those Microsoft-funded TCO reports. They concede that Linux has cost advantages in a very specific field (webhosting; Beowulf clusters), because anyone intuitively knows it's true. For all the rest: use Microsoft stuff. That's what they are saying.
-------
Warning: Slashdot may contain traces of nuts.
The whole goal of distributed computing is to externalize costs. When someone else bears your cost...then, yeah, it's free. This is the idea behind P2P. P2P is significantly less efficient than specialized servers, but externalizes costs. In some cases, SETI type arrangements use real idle equipment. In other cases, it pushes real costs onto other unwitting groups.
Your employer has to pay for the electricity if you leave a bunch of computers on at night to help calculate protein folds. It is not necessarily a bad thing...just that an unwitting party is bearing a cost. In many cases the cost is of little consequence...in some cases it is.
You mean the one that uses the spare processing power of the XBox live users to number crunch for Microsoft?
And what OS where they using?
FRA: STFU GTFO
What if it became a self-employment option?
In other words, my home office, instead of being a revenue-sucking hole filled with computers was instead a source of distributed computing power I could sell on an ad-hoc basis? I eat the tab for the upkeep and get paid cash per work unit I'm able to get done.
This is a good analysis of the hardware side of the cost/benefit analysis of distributed computing, but that's nowhere near the full story.
For example, per the thesis of this article (that network communication is the largest expense of distributed computing), the Salesforce.com model isn't valid. Yet they're a great success story of computational outsourcing. Huh?
The key, I think, is that outsourcing eliminates distractions, and gets your employees back to working at your company's core competency. You're good at selling widgets, so you let Salesforce.com be your HR department.
This sort of analysis is useful for hardware, and could also be applied to meatware resources. It's the same essence of the TCO argument from Microsoft which, I think, does have validity.
When you bill your time at $50 an hour, you'll gladly pay $300 for an operating system rather than use a free one that takes 4 hours to install, 10 hours to learn command line computerese you never wanted to know anyway and many hours dealing with conversion and compatibility hassles.