Slashdot Mirror
Distributed Computing Economics
machaut writes "In a ClusterComputing.org article, Jim Gray, director of Microsoft's Bay Area Research Lab, provides an interesting economic analysis for building distributed systems. When do you choose a grid over a cluster or a supercomputer?
When does it pay off to move a task to the data vs moving the data to the task? He takes current hardware and networking costs into account to answer those questions."
Ungodly numbers of "Beo-Wolf" cluster jokes arriving now!
The preceding post was not a Slashvertisement.
How much does it cost to keep hundreds of regular computers (with all their extra peripherals) crunching away vs. a specially designed computer/set of computers.
When do you choose a grid over a cluster or a supercomputer?
When you have a really high-paying job where you are paid to make such decisions.
.. have already figured it out - let other willing users pay the power bill, bandwidth cost, etc. and crunch the data in their spare time. Seems to be working well for seti@home, etc.
Of course, if you are working with sensitive data (military stuff, major trade secrets, etc.) your security/privacy needs will outweigh the costs involved with doing it all in house.
Don't blame me, I voted for Kodos
interesting thought, but what is the difference between this and the age old concept of the cost/benefit relationship...? im not trolling, it seems that it is jsut that concept with a tech twist..
xao
http://TheHillforum.hopto.org
I was happy that Gray covered SETI@Home as I think the nature of SETI is akin to where certain aspects of distributed computing may go in the future. However, I argue that he left some some key parts of SETI economics at the door; most notably, data integrity and security. As I understand it, *over half* of SETI's processing power, bandwidth, and so forth is used to verify data integrity as it's using untrusted hosts to do it's calculations.
This doesn't make SETI a poor supercomputer, but it does change the economics of it. An economic model of computing resources which accounts for work done by untrusted hosts as involving different overhead as that done by trusted hosts would be a much more useful metric to think in terms of.
All the work must(should) be double checked to make sure everything is correct
Cats: All your base are belong to us.
Captain: Take off every sig !!
Conclusions
Put the computation near the data.
My own general take on all this is the Moore's Law for CPU/data costs vs time will beat the decrease in network latency costs vs time and we'll generally expect to see communications protocols become more "intelligent" to compensate up for the this barrier that cannot be overcome. BW will be relatively cheap, but the cost of building up and tearing down a connection will remain high enough to discourage multi-exchange handshaking (ie., UDP model vs TCP model).
"Provided by the management for your protection."
Just last year we were discussing data transfer over the time it would take to overnight some data in a package, worked out that it was faster and wouldn't clog up our line to burn the DVDs and send them through an international package service vs send it over the T1s. I think with all but the largest businesses this is probably still true for larger (Gigabytes) amounts of data. Network costs are too high to be putting data far from where it is to be used. Whether CEOs realize it or not, this has a great effect on the ways businesses with multiple locations structure their company and work together.
Wow, what a world. $1 will now buy:
1 GB sent over the WAN
10 Tops tera-CPU instructions
8 hours of cpu time
1 GB disk space
10 M database accesses
10 TB of disk bandwidth
1 large beverage
1 of everything in the $1 store
1 unlimited phonecall from some 10-10-### phone company.
5 packets of cool aid
10 packets of generic cool aid
2 cans of coke
When I was a child, data was expensive, and food was cheap...
This is an old maxim of design of any multi-tiered system. The reason is this: computation is largely about selecting and filtering data, before sending the results on to further tiers. This selection and filtering process requires many times more bandwidth towards the data source than it does towards the client layers.
This only stops being true when there is no significant data, i.e. when the computation creates the data, as in the author's examples of render farms.
Ceci n'est pas une signature
The author points out: "The ideal mobile task is stateless (needs no database or database access), has a tiny network input and output, and has huge computational demand."
"And of course, SETI@Home is a good example: it computes for 12 hours on half a megabyte of input."
So, for projects that fit this model, then they should save money over supercomputers. But few projects fit this model, with the author mentioning web and data processing, data loading, CFD, ie anything that "generates a continuous and voluminous output stream" as economically unfeasible. So, car companies really do need those supercomputers to virtually crash their cars. =)
It's a nice piece of analysis. Someone could have done it 8 years ago when Java came out; the facts are not significantly different (The values are different of course but the ratios involved are pretty similar. I did some thinking along these lines back then, and then in 2000 when considering working for a "hot P2P company" that an old acquaintance of mine was running.)
My thinking went something like this: There are only a few, "niche" applications which need more compute power and which people pay for (distributed rendering, CFD, FEA, maybe a couple others). Maybe you could build that into a 10-30 million dollar business if you overcame a zillion obstacles but it didn't look like a billion or multi-billion dollar business. The applications for which people buy beefy servers, and which have a monetary payback, are mostly database applications. For those, you need to move the entire database near to the number-crunching PC, and that's not really feasible due to the cost of transporting Gigabytes of data or the unlikelihood that the PC's hard disk can store all the giga/terabytes of information potentially relevant for the computation. Not to mention the security problem.
And Jim Gray's analysis lays out in more precise economic terms why it doesn't make sense. I like how he even calculated the relative merits of a Beowulf-like cluster of PCs versus P2P which I never really analyzed (I lumped them together as basically similar.)
That said, has anybody even made a stab at designing or implementing a relational database with a P2P architecture? I know that there's Oracle Cluster Server, but I'm thinking of something more low-end and more distributed.
--LP
And how is this different from you or I act?
I don't know for you, but I make GPL software, I give it away for free and therefore I give time and money to the community, partly to pursue a certain idea of the computer industry I desire.
In a way, it's just like people who run the Seti@Home client : they don't do it just "to get a free screensaver" like that Microsoft guy narrowly thinks, they also do it because they want to feel part of a greater, more noble effort than just getting rich quick.
When was the last time Microsoft gave anything open-source or for free that didn't serve one of their short, medium or long term plans ? I mean, it's okay, they're there to make money and they admit it, there's nothing wrong with this goal as long as they try to achieve it morally and legally, but why should it be the same for everybody ?
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
I'm sorry, I don't follow your maths here...
There are 2678400 seconds in a month (assuming 31 days...), so that makes 2678400 Megabits transmittable in a month, or 334800 MegaBytes. Each of your $100 buys 3348 MB, which is 3.3 GB - same order of magnitude as the author suggests...
Perhaps you meant 2678400Mb per month.
A few years back when Grid computing was all the rage we sat down with some investment partners and worked out the figures. We came pretty much to the same conclusion. The "average" commercial supercomputing application (pharma, oil drilling, simulation) would not benefit from "free" cycles on the network.
Essentially, any commercial computation valuable enough to require that amount of effort can justify purchasing a hundred thousand node beowulf cluster and run locally. The reduction in network costs, the advantages of total control and tight security more than pay for the difference in computing cost.
Non-commercial computations such as SETI will benefit from grid computing, and we expect to see more efforts long those lines (RSA, Mersenne, Stanford DNA). But remember, we were thinking about starting a business, and none of those pay for the services, so we moved on.
This is already true. Most email traffic these day seems to be marketers talking to spam filters.
We only look at the cost of SETI from our perspective here on earth...but if you ever consider the enormous cost space aliens have to incur to make their secret communications appear as background noise, then I think more people would oppose the project.
I think it's probbably safer to say that seti@home has a huge surplus of computational power, and uses it to verify each result (though it's not strictly necessary to do so). With only one data source (Aerecibo) you can only produce data so quickly, and once you have enough computational power to do the analysis in real time any extra is just surplus that can be used to verify. They did, however later add some extra analysis to the data to take better advantage of the huge surplus of computing power they have.
The important point though, is that for seti@home each individual workunit, while important isn't critical to the whole project. If a small percentage of workunits aren't computed perfectly it's not catastrophic. In other words there's a certain amount of tolerance for innacuracy. For a project like the OGR (Optimum Golomb Ruler) by distributed.net each workunit must be calculated perfectly, as the goal is to prove which ruler is the optimum one. If workunit isn't verified you haven't really proven anything, since it's possible (and probbably likely) that hardware failure produced an innaccurate result somewhere in the millions of workunits calculated. (Or perhaps a modified client produced innacurate results). Other distributed computing tasks have different amounts of tolerance for innacurate results.
Your underlying point is a good one though. For some projects the need for integrity of the results is very high, so larger computing power may be necessary to verify each result.
AccountKiller
...but, there's other programs that people might find more socially useful/productive than SETI.
How 'bout...this from United Devices? They do a variety of biologically related projects, the most popular one, as far as I can tell, being cancer research...I've been running it for almost 2 years, and I have 100,000 points...how many points do you have?
And what OS where they using?
FRA: STFU GTFO
no doubt ... to date the Grid is very java centric. Now maybe .NET could deliver a speedup, but the nice thing about Java is (a) the latest 1.4.2 JREs use the PIII SSE and P4 SSE2 register sets for better float and double performance, and (b) you can put some serious unix servers in the grid for bonus speed.
One thing Jim ignored is cost of software. Because MS effectively charge per-CPU for their system, you cannot afford to build a beowulf cluster on windows, let alone a full grid. So if MS do want to play in grid space, they need a way to price their platform so it makes economic sense. Didnt see that in the paper.
(nb, MS do clustering already, it is just focused at DBs and big IIS installations, and it costs big numbers)