Can .NET Really Scale?

swordfish asks: "Does anyone have first hand experience with scaling .NET to support 100+ concurrent requests on a decent 2-4 CPU box with web services? I'm not talking a cluster of 10 dual CPU systems, but a single system. the obvious answer is 'buy more systems', but what if your customer says I only have 20K budgeted for the year. No matter what Slashdot readers say about buying more boxes, try telling that to your client, who can't afford anything more. I'm sure some of you will think, 'what are you smoking?' But the reality of current economics means 50K on a server for small companies is a huge investment. One could argue 5 cheap systems for 3K each could support that kind of load, but I haven't seen it, so inquiring minds want to know!"

"Ok, I've heard from different people as to whether or not .NET scales well and I've been working with it for the last 7 months. So far from what I can tell it's very tough to scale for a couple of different reasons.

1. currently there isn't a mature messaging server and MSMQ is not appropriate for high load messaging platform.
2. SOAP is too damn heavy weight to scale well beyond 60 concurrent requests for a single CPU 3ghz system.
3. SQL Server doesn't support C# triggers or a way to embed C# applications within the database
4. The through put of SQL Server is still around 200 concurrent requests for a single or dual CPU box. I've read the posts about Transaction Processing Council, but get real, who can afford to spend 6 million on a 64 CPU box?
5. the clients we target are small-ish, so they can't spend more than 30-50K on a server. so where does that leave you in terms of scalability
6. I've been been running benchmarks with dynamic code that does quite a bit of reflection and the performance doesn't impress me.
7. I've also compared the performance of a static ASP/HTML page to webservice page and the throughput goes from 150-200 to about 10-20 on a 2.4-2.6Ghz system
8. to get good through put with SQL Server you have to use async calls, but what if you have to do sync calls? From what I've seen the performance isn't great (it's ok) and I don't like the idea of setting up partitions. Sure, you can put mirrored raid on all the DB servers, but that doesn't help me if a partition goes down and the data is no longer available.
9. I asked a MS SQL Server DBA about real-time replication across multiple servers and his remark was "it doesn't work, don't use it."

Comments

You've got bigger issues.

[Anonymous Coward]

First, you didn't really specify anything except in generalities, but there's a few things that pop out from my experiences:

1. Why are you wed to C#, especially in regards to triggers? How many tiers exist, and are you pumping a lot of data back-and-forth.

2. Your scaling numbers are low already, especially under ASP and static HTML.

3. You never really define concurrent requests. For some people, it means simultaneous requests, and for others, it means simultaneous transactions. But you really are looking at fairly low numbers there, in either case.

4. Scaling this should involve looking at where you choke. One common choke point that keeps killing people is in open database connections. Are you running a pool? How large? How many connections does a page take? The single most common problem I've seen in scaling is poorly implemented connection pooling, thereby causing a ton of stuff to wait. Check this, check, then check again.

5. Sync versus Async shouldn't really be coming into play yet on the db.

6. When designing for light-weight systems, you want to minimize the tiers, and minimize the data passed back and forth. Just by reading this, I'm worried that you created a very elegant, but impractical, system that isn't suited to the hardware limitations.

Re:Christ, those machine figures!

[gmack]

I hear you.. one of my previous employers had a php/apache system on a dual CPU pIII 500 system with 256 mb ram that easilly handled 500 customers at any given moment.

Not even a hiccup. then the bright guy tried to do that on windows 2000 for another customer.. choked at about 100.

I'll take good performance with low spec hardware over the ability to scale on 10+ CPU systems anyday.

.NET Benchmarks

[fine09]

Hello,
I have been Developing a .NET Portal Application for the past few months. I ran a quick Test on our application just to see how it would run.
Specs Are as Follows:

App Server:
Duron 800
512 MB RAM
40GB HD 7200RPM


DB Server:
Celeron 500
640 MB RAM
20GB HD 7200RPM


As you can see, these are not server class machines, but they seem to run the app alright. I ran a simulation of this application based on the IBS Portal www.asp.net running 150 Concurrent Requests Per Second:

The average Requests per second on this app were 98.51. So, IMHO on low quaility hardware, the .NET platform can handle about 100 Requests per second before it starts to get hot.

Re:Any more information?

[Synithium]

We have switched from Windows Svr 2k and ASP to Apache 2 and PHP 4 on the front end. On the back end we use java 1.4 and broke our application apart to run multiple master/slave processes in a tree system (Process A, Master I, Machines a-d. Process B, Master II, Machines e-h...) to do data analysis for the requests. (This is a data mining sort of thing with analysis and a search). The DB starting becoming a bottleneck after we got up to 200 concurrent processes, which we fixed by breaking apart the DB and placing half on another server and running 2 simultaneous DB connects per slave process and this could continue for some time i'm thinking.

If that gets too goofy we may end up partitioning the requests in the beginning and mirroring two seperated complete systems, but we're not really envisioning it ever getting that big.

Of course, most of the problem is simply the back end keeping up because the front ends don't do much at all except call the java app and return the data it gets...

Ackward Worries, Threading and Responsiveness

[metacosm]

Well, first of all -- you really didn't give us enough information to give a good answer, but -- it is a slashdot question, and getting all the info would ruin half the fun!

Anyway. If you can't support 100 requests a second on 50k of modern hardware, you have huge design issues and other problems. Just from your short description of the project, I fear you have crawled into over-engineered land because alot of the technologies are much more useful on seperate boxes/distributed enviroments.

• #1. MSMQ is mature. I have seen no evidence that it won't scale. How many messages are you planning on piping through it per request? Why do you need MSMQ for an application running on a single box?
• #2. SOAP is "heavy-weight" -- I guess I would have to ask what "light-weight" item are you comparing it too? The number you dropped is utter bullshit, but their is going to be an upper limit to it.
• #3. Yes, SQL Server doesn't support C# triggers or embedding C# apps -- WHY THE HELL WOULD YOU WANT THAT -- seperate your database from your application. Lots of the technology you mentioned in your question are for seperating layers, and then you wanna do something SILLY like cram C# into the database.
• #4. All requests are not even close to equal, it totally depends on the quality of your requests/stored procedures, how normalized or denormalized your data is -- and how intelligent you DBA is.
• #5. Very fuzzy question, but on 50k of hardware for a custom develop internal app -- you should be able to scale. Again, how stuff scales depends on how it is written.
• #6. Uhh, ok, not really a question. Maybe avoid reflection. :)
• #7. Ummm, just ran a local test on an MS box (2k3) and my numbers didn't drop nearly as much -- check you settings. I went from around 290 rps to 180 rps.
• #8. Good DBA, optimize stored procs, get the data cached, use sync calls.
• #9. It used to not work, I have recently got it working, but I still don't trust it, and most DBA's I know don't trust it either. But, test it out, in most of the cases were this is needed/used -- work arounds a very human-work intensive and often work like crap.

Good Luck. Remember that C# Web apps can be multi-threaded, and remember to optimize the parts of your application that MATTER. A wise man once said "Premature optimization is the root of all evil". Find the slow parts, fix them, get the most bang for buck. Also, remember to keep those pieces loosely-bound to each other, no C# code in the DB!

--MetaCosm
P.S. I hope you haven't over-engineered this tool as badly as it sounds like you have :)

Re:Ackward Worries, Threading and Responsiveness

[Dasein]

I have to pick a nit on someone and you're in the wrong place at the wrong time.

A lot of folks are lambasting this guy because he wants to do C# inside SQL Server. Most are saying, like you, that he just doesn't get it because you should separate the database from the application. That's true but it doesn't invalidate the need to have stored procedures (in any language you want be it PL/SQL or C#).

The idea behind a stored procedure is that your application may actually scale better by putting some of the logic "close to the data" because there is less contention for machine resources other than CPU.

For scalability, it's *generally* true that you want no processing to happen on the database because database servers are generally more expensive to scale. However, moving selected bits of logic to the database tier can result in huge scalability improvements.

It's not one-size-fits-all and unless you have a good working understanding of the problem, which is impossible with the data given, it's probably not a good idea to yell "WHY THE HELL WOULD YOU WANT TO DO THAT" at someone. Give the guy the benefit of the doubt.

If you think that stored procedure in C# (or any .NET language) isn't going to happen, you haven't been paying attention.

Yes, .Net can scale--IF...

[John Murdoch]

Hi!

Executive summary:
Yes.

Boring details:
I'm goofing off, perusing SlashDot at the end of a dinner break. We're shipping a big project to a customer on Monday--the project is written in .Net (mostly C#, some components in VB), including Windows forms and ASP.Net web pages. (Why both? The project incorporates multiple applications for different kinds of users.) As part of pre-shipment testing we're in the midst of extensive testing, including load testing.

The Windows applications communicate with the data tier using SOAP/XML, using synchronous messaging. Practically every message involves a database transaction with SQL Server 2000. Across a range of loads we are seeing round-trip message responses (from receipt of the inbound XML message to return from the web service) averaging less than 90 ms per message. That 90 ms average can be misleading--some of our messages involve extensive processing and/or lots of data. Some of the transaction work we're doing with SVG images involve SOAP messages with payloads greater than 1 MB, so the average gets dragged out.

Based on our testing, we anticipate supporting hundreds of simultaneous users--in a near-real-time environment--from a single web service. As we scale out on larger projects we may need to scale the number of web servers (although IIS on Windows 2003 is supposed to be substantially faster--YMMV), but we won't need to scale the database. Using a similar messaging architecture for a different client I have a project supporting 400+ users on a single SQL Server.

This is SlashDot, after all...
Obviously you're going to get a lot of "why not use...?" posts, and I'm sure I'll get flamed for having the temerity to admit to using .Net. And recommending it. But you asked, so I'll answer: .Net is scaleable in terms of the final application, and .Net is scaleable in terms of the size of the development team that is involved. This project involves 19 developers (a total of 60+ individual projects in the nightly build) and we're able to manage the entire thing remarkably well. Developing web service applications with .Net is remarkably easy to do; developing sockets apps is unbelievably simpler than using WinInet.dll. And the web developers are extremely happy working in ASP.Net--I don't know where you heard that ASP.Net is slower than ASP, but that's simply not true. ASP.Net is significantly faster.

With regard to other comments
I'm the data/messaging architect on the project: I can speak to the comments about messaging, reflection, and SQL Server. As with any Microsoft-based development project, you have to think carefully, and think critically, about how to design your application. Microsoft will always give you a quick! easy! fun! way to rapidly produce a prototype. You have to dig deeper, and think harder, to produce a scaleable application. The quick! easy! fun! technology du jour is .Net Remoting. Quick to prototype, barks in production. Like OLE, it's a great way to make a Pentium 4 box emulate an original 8086 IBM PC. (Far smarter to manage communication with XML-based messaging. It just takes more coding.)

That SQL Server doesn't permit triggers to be written in C#--so? Transact-SQL is suitable for database development. We could ask for more (such as integrating stored procedures and other database code into Visual SourceSafe). There is talk that the next version of SQL Server will permit coding in .Net languages--that'd be cool, but I'll wait and see.

The single most compelling argument for .Net
Mono--an Open Source implementation of the .Net Framework. You might look into this particularly for clients that are choking on server pricing--but you might also pay careful attention, because a robust Mono project will encourage/force Microsoft to compete on features and functionality, instead of a take-what-we-give-you mentality. That's a Very Good Thing.

Yes

[bmajik]

A guy down the hall from me was in charge of taking customer web apps written in V6 technologies (vb, asp, etc) and porting them in several ways to .net. They did extensive scalability testing on these apps. They measured requests/sec vs # of cpus, etc etc, to see how asp.net utilized multiproc machines.

What im saying here, is that you are not the first person to ever consider how .net might run for database driven web apps, of an arbitrary size. Infact, much of it is designed for _exactly_ that.

Re: SQL server 2000
SQL server 2000 has more performance then you know what to do with, even on non-ridiculous hardware. Give it processors with lots of L2 cache (xeons) and lots of ram, and read all the docs about keeping MDF and LDF files on separate volumes (as well as tempdb) and you'll find that life is thrilling.

Data point: On a quad HT P4 Xeon with 8GB of ram and 12 spindles (a significantly less than $50k box) we support 1800 simultaneous connections, doing OLTP work against a ~15GB database. The most commonly hit table in the system has about 10 million rows that get added and deleted in batches of between 20 and 10,000, and updated singly or in bulk. Other apps select from this table on a polling basis (i.e. decision monitors). We could make our db and app design much "better" w.r.t performance, but we don't need to - the money we save not having to do genius level feats of programming, app rewrites, and perf tuning more than pays for the occasional new hardware or upgrade.

Continuing, Run perf monitor on your SQL server machine. Look at the physical spindle(s) that hold your MDF. If you're reading from them, buy more ram until you're not :) Look at the I/O per sec rate to your tempdb disks and primary LDF disk(s). It is seriously to your advantage to go with an individual spindle for each role, because IO rate is what is so critically important to SQL server. Also, avoid RAID5 like the plague, as it decimates IO Rate.

You can tune SQL server without application changes until you're blue in the face, honestly. Use profiler to see what kind of queries you're doing. Put those queries in Query Analyzer and show the execution plan. QA breaks it down for you and shows execution time percentages of each sub-tree of the execution plan. If you've got something eating 80% of your time and its doing a table scan, do whatever you can to put some selectivity in that query (i.e. an index, or maybe a query change).

If you want to save yourself some headaches, setup management tasks to recalc indexes over the weekend (or nightly, if you see that much index fragmentation after a day).

Re:Why are they running Windows then?

[IntlHarvester]

Every program a client has that uses any version of SQL Server needs constant fixing, and is incredibly slow for 7 users. I could give you a list of programs, but it's just about every program on the market. One of them has an option to use paradox database files,

I think I know what you are talking about.

I'm guessing this software is mainly decade old desktop packages that were originally designed to run on Paradox/dBase/FoxPro and ported to SQL Server or Oracle because that's the trendy thing to do. (If you see "BDE", the Borland Data Engine, it's a good sign that this is what you've got.) The thing is, the apps aren't really ported to use a RDBMS design. They still use "Flat Files" and have their own key/indexing system and old style coding.

The one I'm familar with is the very popular "Goldmine" sales package (had to get data from it's schema for an app I built). Doesn't even use Primary Keys, much less non-clustered indexes. Instead it's got these dbase-style bogo keys which look like "AAAA", "AAAa", "AAaa" and so on. But SQL Server is running in case insensitive mode, so all of the key comparison is done on the client! It also appears to do record locking on the client-side. No wonder an almost trivial application only supports 20-some users on a P4 server. :P

Hopefully as someone targetting DB2, you aren't making the same kinds of error, because you'll see the same issues no matter the RDBMS.

There are intentional problems between NT4 and Win2K+ using NT4 as a file server with SMB

I'm guessing this is the "rogue master browser" problem. Sucks, but an unofficially well known issue.