Slashdot Mirror
Ask Slashdot: Capacity Planning and Performance Management?
An anonymous reader writes: When shops mostly ran on mainframes, it was relatively easy to do capacity planning because systems and programs were mostly monolithic. But today is very different; we use a plethora of technologies and systems are more distributed. Many applications are decentralized, running on multiple servers either for redundancy or because of multi-tiering architecture. Some companies run legacy systems alongside bleeding-edge technologies. We're also seeing many innovations in storage, like compression, deduplication, clones, snapshots, etc.
Today, with many projects, the complexity make it pretty difficult to foresee resource usage. This makes it hard to budget for hardware that can fulfill capacity and performance requirements in the long term. It's even tougher when the project is still in the planning stages. My question: how do you do capacity planning and performance management for such decentralized systems with diverse technologies? Who is responsible for capacity planning in your company? Are you mostly reactive in adding resources (CPU, memory, IO, storage, etc) or are you able to plan it out well beforehand?
Today, with many projects, the complexity make it pretty difficult to foresee resource usage. This makes it hard to budget for hardware that can fulfill capacity and performance requirements in the long term. It's even tougher when the project is still in the planning stages. My question: how do you do capacity planning and performance management for such decentralized systems with diverse technologies? Who is responsible for capacity planning in your company? Are you mostly reactive in adding resources (CPU, memory, IO, storage, etc) or are you able to plan it out well beforehand?
Pay attention. This is VERY complicated....
We ask our users what their plans are.
You cannot save yourself from understanding the architecture. Then you need data. Then you combine the two.
is this a trick question?
My powers of prediction are inadequate; can anyone sell me a good, working crystal ball? Or, better, maybe /. can just give me one...
Maybe if you're Netflix or Facebook you worry about this sort of thing. But most enterprises I work with this is becoming less of an issue. Most CPUs sit idle , memory is plentiful and the only item that is always running out is disk capacity, but even that is so cheap to replenish that it costs more to chase someone to cleanup then to simply keep adding disk.
I know this doesn't answer your question, but how much capacity are we talking about here? For reference, I'm talking at least some fortune 500 shops I've dealt with who aren't terribly compute intense. Less than 10PB disk, ~1500 physical hosts, ~10,000 VMs.
This question is why scale out architecture has become so popular. It is virtually impossible to plan for 2-3 years down the road in most environments. Since you cannot plan, it is paramount that you can add compute/storage/hyperconverged nodes as needed.
It is simple, I use a mainframe and buy capacity when needed.
Capacity planning is a rather anachronistic idea from the days when hardware was expensive and procurement cycles were extraordinarily long. The better approach these days is to craft application level service level agreements and let that drive capacity needs. Virtualization, cloud resources and on demand computing have in my opinion rendered most of the art/guesswork of capacity planning irrelevant.
in Soviet Amerika, Capacity Plans YOU!
Massively overestimate HW because the beancounters will cut it back even if only to prove they can.
Explain the following, repeatedly, until it penetrates the thick skull of whoever fails to understand it: Hardware is CHEAP, data is EXPENSIVE. It is cheaper to buy all the HW now than to try and process or recreate all the unprocessed or lost data the lack of HW caused.
Get somebody to sign off on the final configuration. It will almost certainly be suboptimal for the task, and that signature will give you the protection you need when scapegoat(s) are required.
Speed of implementation in various organizations (or even departments, divisions, etc) runs a spectrum of "do stuff on more or less a whim" to "go through eight years of planning meetings to discuss the possibility of actually doing something." On the former end of the spectrum you buy extra capacity. At the latter end of the spectrum it doesn't matter, because you won't get the budget to buy extra capacity.
Help save the critically endangered Blue Iguana
Planning? That's what VMs are for. Just throw more hardware at the resource pool.
There are a lot of tools you can use to help with capacity, be it VM farms, SANs/NASes, cloud providers, chassis/blades. Only a few points of advice:
1: Everyone will sell their product as the silver hammer, where each target is a nail. The VNX guys will sell their SAN as a be all and end all, even if you just use CIFS/SMB. The security vendor will be selling you exotic appliances for encryption for your tape silos. The PC guy will be selling you tons of 1U racks and try to convince you that the onboard drive array is better than a SAN if they don't have a SAN product, otherwise, how slick their HBAs work when used with their SAN.
2: Don't forget security. It may be cheaper to have one VM cluster for everything, but it be wiser to keep one client's hyper-sensitive stuff on one VMWare datacenter [1], while the other client who is running some backend stuff for an app would be in a different container.
3: Before committing to purchase something, grab manuals and documentation, and read on the device. You might find it doesn't do what you want. Don't forget to take into account type of I/O and other items. I have had to deal with a terabyte/hour of random writes, and the only solution for that was going with either a caching HBA that had that much SSD so it would turn the random writes into an easy to digest sequential stream for the SAN, or go pure SSD. Sequential I/O is a lot easier and a lot cheaper to deal with than random I/O. Similar with I/O that is often cached versus I/O that never is reused.
[1]: A datacenter is a VMWare object type. Can't vMotion across it, and is intended to provide distinct separation between items.
I used to work for the (late, lamented) Sun Microsystems, and when we needed to give a credible answer to a price-sensitive customer, we used Teamquest Model. It pulls time-based info out of production-systems stats, so it doesn't add to the load, and then off-line does a classic queuing-system model of the system, working all in time units. That then allows the customer (really meaning me!) to ask what to expect from some specific configuration, and compare different systems for their price-performance tradeoffs.
For common setups, we have spreadsheets based on what Model said, so the salespeople typically don't know there's a cool mathematical model behind the scenes (;-)) That's probably true of other vendors who use TQ models: it runs on anything modern, so lots of vendors use it.
I have nothing to do with the company: they just allowed me to save $1.2 million once for a new datacenter, so I'm really really impressed by them.
--dave
davecb@spamcop.net
Ask your vendors for their capacity modeling and planning tools. Hopefully they can provide something as easy as a spreadsheet.
Although I'm sure the answer would always be "you need to buy more of our stuff", at least that will help you ask the right questions and get semi-reasonable answers. GIGO.
You don't even have to ask your users. Just use NoSQL, for everything. Use it for storing the data, use it for the back end business logic, use it for the middleware, and use it for the front end. Thanks to the CAP Theorem and JavaScript (all NoSQL uses JavaScript), you don't need to worry about scaling at all. That's the beauty of NoSQL: effortless and infinite scalability.
According to ancient lore, a pair of low-level HP engineers put in a request for a Saturn V rocket and a vice president killed their request. HP would have been a different company if it had to listened to their users.
It's mostly a crapshoot, and often the PHBs don't take your advice anyway.
and let them figure out how to make it work. why bother?
That depends are you getting the information you need?
Are your business analysts/architects even able to answer questions such as, how many net new users, concurrent users, able to summarize the typical workload? Back in the day and i'm only in my early 40's, this stuff used to be well defined. We used to have large documents which go down to the level of expected network load. So its either as you said, its too difficult given the diversity of the systems or they just don't know how to do their jobs anymore. I honestly think its about a 20/80 split. Yes the environments are more difficult to manage, but BA's/architects haven't adapted or frankly just don't care.
BA's can't give me any information which would help me forsee or estimate how much load a project/change is going to have on the environment. So when i'm asked if we need new hardware, I just usually tell them to make sure they plan a proper load test and be prepared to spend money.
In my company, its my job to make sure lights on runs well and highlight any issues related to capacity. For new projects, then its part of the project team which I may or may not be a part of.
Storage, for us, seems to be the largest constraint, with memory and cpu coming in behind. Since we can't get much information, we just make sure we have all our servers hooked up to a large san so we can quickly provision more space.
"Thanks to the remote control I have the attention span of a gerbil."
Translation:
"Dear Product^WReaders, we at Dice want to know if 'Capacity Planning' is the new buzzword. Please provide anecdotal evidence to the approximate value of resume candidates with these keywords."
You should be using your monitoring system to gather performance data, and then analyzing that data.
I am partial to check_mk right now, but I've done this kind of thing on nagios with pnp4nagios. When you have your monitoring system gathering network interface data, disk usage, cpu utilization, etc, and storing it in some kind of database like rrd, influxdb, or graphite, it isn't that much of a stretch to examine that data as an aggregate and graph trends. It really is amazing all the stuff you can figure out with this technique.
Too much network bandwidth? Too much storage capacity? Too much CPU?
Usually the drill seems to involve a lot of begging and pleading for money from management. The intermediate levels get dinged if they have to go back to the well, but they don't seem to get dinged if budgeted money ends up buying unused capacity.
I don't doubt there are places which do heavy audits and ask hard questions about why you have a SAN with a bunch of free space or why your 10 gig NICs are running at sub-gigabit utilization and cause all manner of pain and suffering for excess capacity already budgeted and bought.
But usually it doesn't seem to happen that way. Management barely supplies enough resources to meet their running demands and line management buys as much excess capacity as they can beg, borrow or steal because they know they will be punished for buying too little.
These days capacity planning comes down to have the right tool set for the job. I like VMturbo. There are a few others out there that will get the job done. VMturbo is nice because it is platform agnostic and can help you decide where to place workloads not only based on pure performance numbers, but also on resource cost. (For example, HyperV is likely less expensive than VMware in most situations).
It is also worth considering an Application Performance Monitoring (APM) tool. Being able to identify exactly where the application is slow, and whether or not is an issue with the code or the underlying OS / infrastructure will save a lot of time during troubleshooting, and also help identify rooms to proactively allocate resources to head of potential bottlenecks.
On a similar subject, a tool that provides deep visibility into the database layer helps a lot for the same reasons. A lot of junior admins make the mistake of assuming that high database server utilization is indicative of under provisioned hardware. In reality, poorly written queries will bring down even the beefiest of database servers. While you get information with the built in management tools, a dedicated monitoring platform (like Spotlight from Dell for example) will help you develop historical trends, while at the same time providing real time troubleshooting capabilities.
Most of the time the network is the last bottleneck. In Cisco shops you can utilize NetFlow to figure out where the problems are. Or if the company you are working for has money to burn, the UCS infrastructure stack is very robust and comes with a whole slew of management and monitoring tools that can be leverage to discover latencies before they impact production environments too severely.
Just use AWS and scale out as needed. Your capacity planning then becomes more of a question which reserved instances to buy. AWS is not suitable for 100% of applications, but if you ask how to do capacity planning then your use-case is unlikely to be that 1% that doesn't fit the AWS model.
Not sure how it is done elsewhere, but I think we tend to balance load tenuously. Adding more applications and users slowly. When things start slowing down, users will start to complain, bring on a few more servers, repeat.
However, one thing I will say, there is a danger in overcapacity. Managing multiple servers can be difficult (apparently), and it can cause some pretty hard to nail down issues in applications, particularly in legacy applications.
I've had a couple of instances where a couple (not all) severs were configured differently, and applications would perform differently, depending on what server you were currently randomly connected to. Made user complaints really had to decipher as sometimes it would work fine, and other time not. Like having the default system date set to MM/DD/YYYY VS DD/MM/YYYY with legacy systems that just accept whatever... I've also had a few instances where certain services were turned off some but not all servers... again with similar results in that performance really depended on what servers you are being assigned to, and what the specific application is doing...
It was a customer's center so I'm being vague, but it was more than 10%
davecb@spamcop.net
Today, with many projects, the complexity make it pretty difficult to foresee resource usage. This makes it hard to budget for hardware that can fulfill capacity and performance requirements in the long term.
How is it any harder than ten years ago, twenty years ago, thirty years ago? Either you know what you have and how it is performing today or you don't. Either you know what the user demand(s) is/are or you don't. Either you know what options you have for hardware, software and services needed or you don't. Heaven forbid you do your job and find out BEFORE you start a new project plan.
I know why this person posted this Ask /. anonymously. Either they're completely incompetent or their organization is when it comes to IT project planning and basic knowledge of their existing systems and users. OR, this is just another stupid Dice plant to get data from us for their recruiting metrics. Just seems like somebody is either out of their depths or is pulling a fast one.
I just imitate holmes on homes and overbuild everything. eventually you outgrow everything, but it extends useful lifetime by quite a lot.
You might try your question on the Capacity Planning and Performance Evaluation group on LinkedIn instead. There are lots of lurkers there who may be of assistance.
tl:dr; version of our lessons and suggestions
At the end, you'll have something like this for each component of the system - e.g. "if I'm CPU bound on component X, and CPU of X linearly goes up with API_calls/s, and I'm currently at 5000 API/sec at 50% CPU, then I have total capacity for 9000 API/sec (with a 10% buffer) and free capacity for another 4000 API/sec.
Now divide and conquer - let each component owner the responsibility to manage capacity of their system based on business needs provided to them.
One I've seen is Fujitsu's QoS when I worked there. Monitors windows, linux, solaris (apparently hpux and aix) and has nice capacity reporting and alerting. Not sure if they have a public demo. I've only seen it with MS they've done and as per usual most of their internal products aren't marketed well. Have a look.
Take a look around you organization...do you have anyone who has a firm understanding of the breadth of the technologies used in your systems? Does that same person have experience with performance testing and capacity analysis? Since your asking Slashdot the answer is very likely NO so go find a consultant who has that knowledge and hire him/her ASAP. Capacity analysis of distributed systems is not something to learn on the fly. Hire a pro.
The key is good automated reporting. We have reporting and alerting when we are getting close to capacity. We keep these usage reports which means every year we can forecast capacity for the year based on the previous year. At the start of the year we forecast and make purchase recommendations. Extra capacity is purchased and installed. Simples.
Migrate to the cloud, either public or private.
Convert your metrics into time units so you can say something like "I need 6 CPU/S per 100 users at nor more that 80% utilization" . The math is less weird than trying to work in percentages of something you're going to replace with a CPU that's 12% faster (;-))
davecb@spamcop.net
At Nokia Mobile Phones I was the principal performance engineer in the PCA group for the proxy browser department - PCA meaning Performance, Capacity, and Analytics. Before Microsoft took over the division we were most of the way to move our systems from our own data centers to the Amazon EC2 cloud. Why? Because adding capacity to existing data centers with our own hardware had a number of major costs - hardware, networking, power, cooling, and size. Adding a rack to an existing data center was a serious capital cost and took time, for approval, specifications, purchase, installation, etc. Then we needed staff to see to the maintenance of the gear. Moving this to AWS meant that we could easily scale up or down our capacity as needs changed, even hour by hour, and only pay for the capacity that we actually needed. The savings were HUGE! With our own gear, we had to plan for worst case (highest) usage, and even when it wasn't being used, the cost was still there. With AWS we could compute what our needs were going to be in few hours, and spin up or down servers as appropriate. If we weren't using them, we weren't paying for them.
As for performance, I designed and built tools in C++ that ran on each server (both ours and Amazon's) and pushed 10 billion data points per day to both a Hadoop cluster in AWS for real-time performance/reliability analytics, and an AWS Redshift database for click-stream analytics.
then cloud computing is pretty much aimed at you. (At least if your need is likely to be variable.)
The whole point of the cloud is that you only pay for what you use. If your needs are wildly variable from one month (or day) to the next, it might make sense to rent time/storage/throughput via the cloud.
If your needs generally only increase, and increase at a predictable pace, then it probably makes more sense to buy your own hardware.
Exactly. That is one of the things we consider in the blog.
I ran the Capacity Management practice for a leading provider of financial data servers for 10 years. We had a dedicated team doing the capacity planning for the company. However, a key part of my practice was making the performance data available to the application analysts as well as the centralized capacity planning team. By fostering this partnership with the application teams, we were able to understand exactly what was going on with each system and develop capacity plans togethger. We settled on the TeamQuest set of products (for full disclosure, I am now working for TeamQuest as a Product Manager). There are many tools out there that can you help with with some of the initial tasks like gathering performance metrics in order to move from Chaotic to Reactive. However, as you move up the maturity curve to proactive, service and value, the available technology really starts to thin out. Some of the key elements you want to pay attention to are: - Data Granularity. Some tools only go to the 15 minute level. TeamQuest can collect data down to 1 sec intervals which some of our customers find to be essential. For many customers 1 minute or 5 minute granularity is sufficient. You will also want to view performance data at the process level to find the culprit(s) for high CPU utilization (for example). - Problem Resolution. You will want an easy to use flexible interface to view the performance data in fine granularity to assist in problem determination. Automated correlation analysis is a huge plus in this area as the problem you are looking at may actually be a victim of something else going on in your infrastructure - Prediction. Performance of computing systems, unfortunately, is not linear. So simple linear trending will only take you so far. You will need more sophisticated modeling technology to really understand when response time and throughput is going to suffer. - Guidance, You need a set of analytic tools which tell you exactly what will cause performance issues and when & understand what it will take to prevent the problems from occurring in the first place.