How Far Can Large Commercial Applications Scale?

clusteroid81 asks: "I've been working with customers who run large commercial applications on big iron (16-32 symmetric multi-processor systems - 64GB or more memory ). There are always numerous other front-end servers involved, but the application on the back end server is often difficult to spread across multiple systems or clusters due to the application architecture. Scaling is done by increasing memory and processor counts. As things progress, the bottleneck is usually contention within the application or operating system. Are there folks here on Slashdot who work with large single system commercial applications? What kind of processor counts and memory do the applications have and how well do they scale?"

Comments

It all depends on the applications...

[georgewilliamherbert]

I've run oracle on 32 processor Sun E10Ks with reasonably linear speedup from few-processor performance, back in the Solaris 7 days.

I've run (now obsolete) ATG Dynamo on the same, with similar results.

I've run Apache (1.3.x) on the same, with similar results.

I've seen applications which stopped scaling well at much less than that.

"Large business applications" isn't specific enough.

Enterprise

[Procyon101]

It depends on how much enterprise you have in them. Enterprise is expensive, but when added liberally you can scale to huge amounts.

I like to add a couple hundred enterprise myself.

Re:Enterprise

[MBCook]

Luckily there are lots of examples of Enterprise quality out there. The Daily WTF has lots of great stuff. Here are two recent examples.

scale by hashing

[pikine]

I'm still studying computer science with little practical experience, but you can divide certain aspects of your application by hashing---you hash datasets or queries. This distributes the workload across a cluster of computers. However, implementing hashing requires you to make intrusive changes to your code, and maybe most companies aren't willing to do so. Hashing generally has to be implemented from the very beginning, which requires foresight. Google is the one company that does it well.

Re:scale by hashing

[dgatwood]

There are many ways to divide up a set of queries. It all depends on what the application is, how much data sharing is needed, etc.

One way divide the data is per-user or per-group. Divide data according to its owner so that each user account is hosted on a given machine and has first-class access to his/her own data and his/her group's data, but second-class (network-based) access to everyone else's data.

Another way, as you mention, is to do hashing based on some well-defined key, but for this to be useful requires that the front end be thoroughly abstracted from the back end so that multiple front ends share multiple back end stores. Otherwise, you are probably just moving the bottleneck around. It also requires that this key be known in advance, which means that it doesn't generally work well if, for example, you need to do a join on two tables and one of those tables is scattered across multiple machines. The only way that it would work for such use would be if either the key being used for the join is the hashed key or if each machine has a table index that spans multiple machines' content, at which point, you are going to have cache coherency problems.

Which brings us to a fairly nice compromise solution: a replicated database with each of the outer-ring database servers being read-only caches with some sort of built-in cache consistency protocol, and the central database accepting write queries from clients, but with all the read queries directed to the outer ring. Makes for seriously scalable database access.

This, of course, assumes that the app in question is a front-end for a database. If you're doing some other sort of application, then all bets are off. Give us more information.

It may seem offtopic....

[riprjak]

...but have you considered trying to contact the EVE-Online developers at CCP.

Their game is little more than a MASSIVE database application supporting tens of thousands of simultaneous users... They have lag issues but, on the whole, seem to be scaling bloody well.

Re:It may seem offtopic....

[Anonymous Coward]

Or better yet, call Blizzard and ask for tips about scalability and reliability. Then do the opposite.

yes

[larry bagina]

I did some freelance work a few years back for a client. They were converting some custom inhouse applications from a 64 processor Cray Superserver 6400 to a cluster-based approach. I can't comment on what they were doing, but they needed all the ram and cycles they could get ahold of.

Anyhow, they started out on a 4-way machine and had scaled up to the 64-way without many code changes. If it had been cost effective, they would have kept on scaling upwards.

Vague question... Vague answers

[subreality]

Different problems in computer science scale differently. You haven't given us enough data to really know what problem you're solving, so you're really not going to get a reasonable answer.

I work for a company that has a large commercial application. We knew we needed to scale our data set and processing power to be huge, so we made sure from the start that the heavy lifting could be divided into little chunks, and thrown to the cluster. For our purposes, back end scalability is basically linear. When we need more, we just bring another rack of little 1U critters online. There are a few theoretical bottlenecks, but we'll never see them before we have our own nuclear power plant to run the data centers.

For other applications we use, there is *no* scalability. The algorithm has to be single threaded. It doesn't matter if I run it on a cluster, or a machine bristling with CPUs. So we basically buy the data center equivalent of a gaming PC: The fastest processor and memory that fits our budget.

So there are the ends of the spectrum. Your scalability will be somewhere between zero and infinity, depending on the problem at hand.

Re:Vague question... Vague answers

[Mr Z]

Some problems are like the "baby" problem. It takes nine months to make a baby, no matter how many couples are assigned to the problem. BUT, if the task is to make 1000 babies, you can still do that in 9 months—if you can find 1000 couples. But, if you only need one, you're stuck. It's a parallelism granularity problem.

Other times you get stymied by serial bottlenecks in an application. Sometimes you can gain fractional benefit from additional compute resource by allowing various CPUs in the cluster compute redundant results in lieu of waiting for intermediate results from prior computation. For example, one problem I was working on recently had this problem. It was an optimization problem that built up "new answers" from "previous answers" in an attempt to find the shortest sequence of operations to meet some criterion. The kernel operation iterated over pairs of previous results, combined them, and determined if the new result was unique. (There are more details. I'll keep this brief.)

At its heart, the algorithm was effectively a breadth-first-search shortest path algorithm, where the edges in the graph are described algorithmically, not discretely. The issue is, when doing such a search, where the exits from a particular state (node) aren't known explicitly apriori, you can't mark the discovered states as "visited" to cull the traversal through the state space without serializing everything. In fact, in this particular problem, I was converting the algorithmic description of the edge connections into an explicit description.

The trick to parallelizing here is to periodically divide your work queue of "nodes to visit" among your compute nodes, and then merge the results back, knowing that you will have many redundant "node visits." You can filter these out with some other structure. In this case, my total state bitmap was 512MB--easily held in one node--so the merge process looks like a "Hey, have I seen this? Nope? Pass it on." Even the merge can be performed hierarchically, so eliminate redundancies in stages. At each level of the hierarchy, you can subdivide the state space you're merging to gain parallelism that way.

So, sometimes there ARE ways to speed up serial computations, but at the expense of computing redundant intermediate results.

--Joe

Very little to go by ...

[kbahey]

Your description is very little to go about suggesting solutions ...

You have to tell us many many specific things before we can suggest specific solutions. All we know is that the application runs on a 32 cPU system, and has 64 GB. This is all about the hardware. The application is a "large commercial application", and there is "contention within the application or the operating system". We do not even know what the hardware is, nor what operating system it is.

Anyways, here are some generic suggestions form past experience, most of it on UNIX systems, many with Oracle, and most with commerical non-web systems.

- Is the application CPU bound, memory bound, or I/O bound? If you do not know then you have to find out first, then attack the area of

- Is the application transactional in nature or batch? Is it an operational system, or a decision support type of application?

- Does the application use a database (probably does)? Is the database on the same box that runs the application? If so moving the database to a separate box with a fast connection (FDDI or Gigabit Ethernet) may help things.

- Does the application uses queues or message passing? Do these queues fill up at certain peak hours causing slow downs?

- Can you benchmark/load test the application on a similar box? If you have transaction generation/injection tools, then you can simulate the real load and then run tools for profiling, performance and the like in real time (e.g. sar, vmstat, top, ....etc. if you are on a *NIX type of system).

Performance tuning is an iterative process that is more of an art than a science. Start with the 80/20 rule, and get the low hanging fruit (attack the easiest and most obvious area that would gain you some performance, then move to the next area, ...etc until you hit the diminishing returns areas).

My experience with Solaris/Oracle

[brokeninside]

One place I used to work had a system that scaled up to well over 20 Sun boxes each with 10 more CPUs. It all depends on having the design right. For example, if you have a batch job, you architect the job to follow a master/worker paradigm where a master process doles out chunks of works to worker processes that may or may not be running on the same machine (think SETI@Home). Not every job can be redesigned to to this, but it it's a fairly easy way to do a large number of different tasks. Further, there's no reason that this design couldn't be used by Linux/PostgreSQL or some other Free Software stack rather than Solaris/Oracle. There are also other paradigms. Perhaps you should do a search on scholarly comp sci papers instead of asking /.. The problem of scaling is not exactly new. Quite a few papers have been written on various way to solve the problem depending on what sort of computational tasks you have to accomplish.

Not far enough.

[Onan]

Do you mean to ask how far things can scale "vertically", by buying progressively bigger individual machines? That's an easy one: never far enough.

Even if you can magically get a single system that's big enough for your needs forever, you'll still pay orders of magnitude too much money for it, and get no added reliability through redundancy.

Any application that requires a solitary, unique, big server is just definitionally broken. It needs to be redesigned to allow it to be spread over an arbitrary number of small systems in geographically diverse locations. For reliability, your serving infrastructure needs to be at least n+1 at every layer to allow for planned maintenance, unexpected failures, and site-destroying disasters. And for scale, it needs to allow you to continue to plug in more batches of cheap little machines and get more throughput.

Re:It's the network!

[multimediavt]

I'm gonna go ahead and disagree with you there. The network alone is not to blame. Also, keep in mind that the latency differences between most 10GigE implementations and Myrinet are radically different especially once you get above the hardware and protocol levels. They are getting better, Force10's new 10GigE switches being good examples, but they're not that close when you put something like MPI and then a poorly implemented-algorithm wise-application on top of that. Another thing to keep in mind is that there are other interconnect technologies like Infiniband and Quadrics that may give you better performance.

The real scaling issues (in a lot of cases) are within the application itself. Some applications scale really well. I'll use scientific codes as examples. For instance, we've gotten LAMPSS (a molecular dynamics code) to scale very well across our 1024 node, 2048 processor cluster. It is capable of using the entire system to process jobs; all 2048 processors with an Infiniband interconnect and MVAPICH. However, applications like AMBER, another molecular dynamics code, don't scale at all well beyond 256 processors on our system. It's not a fault of the hardware, the network, or the message passing interface in a lot of cases. It's simply that the algorithm used in the code just doesn't scale well beyond a certain point. The code just isn't optimized well, or it just won't scale, period. There are other code bases that are being used by our researchers that do well in an SMP, shared-memory architecture, but simply won't run at all in a distributed memory, cluster architecture. Some because they require a large memory footprint, others simply because the problem the code needs to solve cannot be decomposed and spread across nodes in a cluster. As far as performance goes, we've actually seen some codes, like the quadrature code (APREC) run by David Bailey of LBL, actually achieve super-linear gains. He ran a series of jobs in his quest to do the largest one-dimensional quadrature calculation (which he achieved and published at SC04) starting with one processor and scaling to 512 nodes (1024 processors). At the 16, 64, and 256 processor range, his code actually got 17.66, 69.79, and 270.17 times speed up over a single processor, respectively. Now this is not typical behavior. Typically, you don't get this kind of speed up (usually you do see significantly lower efficiency; in the range of 15 to 20 percent in a lot of cases), and his code did fall off to 919.22 times speed up for 1024 processors. My point is, the application itself has as much impact on performance as the architecture it is being run on. And, don't forget compiler differences, but this could go on for days.

I would strongly urge the original poster to talk to the vendors that develop the software you use and simply ask them if the reason they don't make a cluster version of the software is due to economic reasons, or simply because the application just won't work in that architecture. Remember, computing is a right-tool-for-the-right-job arena. There's no single platform that will do everything for everybody.

Large Project on Server Cluster

[psalm33]

My company has developed a large software project on a server cluster for the backend. Our server-side architecture is (in theory) scalable as large as we want to go. We use BEA Tuxedo to assign different applications to different servers, and all the databases are available via a SAN. The Unix servers use are currently configured with 4 to 8 CPUs each, and 8 to 16 GB memory. The server cluster is currently configured between 2 and 10 servers for our current deployments, though we could scale larger simply by rearranging the tuxedo configuration files if we needed to.

Now, some server-side apps in our system are architected to scale very well, and some we have had to spend the last few months tweaking the code as we grow with our current customer's deployment. In general though, our system tends towards lots of specific apps running simultaneously to handle individual tasks, rather than a small number of large, monolithic apps. I think it is very much making sure you have large system scalability in mind from the beginning, and not starting small and then realizing "Oh no! We never realized we'd have to handle THIS much traffic!" Our project is a perfect example of learning that lesson over and over as we've had to tweak or rewrite pieces of it as we add more and more clients to our customers' deployment. It can be done, but depending on how you've written your apps, it may not be easy.

Short-sighted project management

[CarrotLord]

In my experience, the custom applications I deal with seem to be built with not just incorrect assumptions regarding load, but *no* assumptions regarding load. When I first fired up one particular application in a production environment, we were seeing 6000 incoming messages per second. I asked the lead developer what we should be expecting to see. He had no idea.

This is caused by short sighted project management, which translates into short sighted programming. The necessary questions about throughput aren't asked, because it all works fine on the developers' PC with a test load. In our case, we eventually got the application running OK, but changes that have been made since have not taken into account anything to do with I/O, so the fact that our CPU usage is not maxing out seems to indicate to the development team that we are not bound by the server performance, and hence have not reached any scalability thresholds.

Obviously this is madness. If one was to investigate the scalability of this application properly, one should be looking at where I/O happens, where interprocess communication happens, where object creation and destruction happens, and so on... There is no other way to scale an application -- you have to define what the "load" is, find what happens when you increase it, work out where any bottleneck is, and how parallelisable this bottleneck is. Anything less is no more than buzzwords.