To Grid Or Not To Grid?

dbgimp writes "In my job at a (large) investment bank I am constantly being pushed to use grid technology. I have many problems with this (not least that our data center is at best 100 Mb/s and our software is actually more data than computation heavy). A typical batch job takes 10-30 minutes consisting of around 10,000 trades. I would far rather spend the time and money on multi-core machines and optimizing the software than on the latest fad technology. I am interested to hear from other people in a similar position and, in particular, why or why not they chose grid software over improving the existing code to leverage better processor technology, and which grid software they chose to use and why. Or, conversely, why they chose not to use grid software."

Comments

Well compare costs

[jellomizer]

I would say give them the full price to get it done right then have management decide. Be Sure to give them the full quote with the cost of making 10gbs Network or faster for these systems. Then you need to realize how much of the code can be parallelized, estimate you time it will take you to make the changes, and add in proper debugging time. Next find alternative solutions that will increase performance. For example except for grids you have clustering which works better for some applications which have fast calculations but a lot of users that just make it slow. Grid Computing works well when you have a large segments of code with minimum communication between each system. If you need a lot of CPU to CPU communication then you will need to get a real supercomputer where the processors communicate across the bus.

Or You just need to index your tables.

The rule of thumb is go the safe rout unless you are told by higher ups to do otherwise, have higher ups sign off on the more risky method (to save your butt) then get the method working focusing on getting the job done right and stop complaining how bad decision it was.

Re:Well compare costs

[DaveV1.0]

Well, that is half the work. For this instance, he should do a cost benefit analysis.

Just providing costs comparisons boils down to "Your way costs X, my way cost Y." But, that may not matter to someone who wants to be buzz-word compliant. When an executive gets it in his head that "this" is better than "that", the best way to handle it is to show that "this" will give a give a crappy ROI while "that" will give a great ROI.

Unfortunately, sometimes even that does not work and you end up doing it the boss' way, becaue he is the boss.

Different technology

[doctor_nation]

Maybe it's just me, but you sound like you're just resisting going to a different technology where you have to learn a new set of skills. Grid/multi-processor computing is definitely not simple, but depending on how many spare CPU cycles you have, you'll get a much faster and larger speedup in your runs than if you tinker with the code to make it run faster single-threaded. Also, won't you need to do some multi-threaded on a multi-core machine as well? (I'm not particularly familiar with multi-cores, so I could be wrong).

I recommend learning how to use grid computing and convert the program. Not only will your code run faster at the end of the day, you'll gain a valuable set of skills that will look great on a resume.

hmm.

[pizza_milkshake]

I would far rather spend the time and money on multi-core machines and optimizing the software than on the latest fad technology.

think about that for a second.

I forgot

[Colin Smith]

7. It's not a fad. The technology has been around since the 80s IIRC, possibly earlier. The word "grid" is a fad, but not the technology. They started as network or batch queueing systems. The word "grid" is like the word "middleware". It isn't well defined and means a bunch of different things to different people.
8. Off the top of my head, freebies include Torque, GridEngine, Condor.
9. Yes it would be a Beowulf of those. Mwhahaha!