Ask Slashdot: Capacity Planning and Performance Management?

An anonymous reader writes: When shops mostly ran on mainframes, it was relatively easy to do capacity planning because systems and programs were mostly monolithic. But today is very different; we use a plethora of technologies and systems are more distributed. Many applications are decentralized, running on multiple servers either for redundancy or because of multi-tiering architecture. Some companies run legacy systems alongside bleeding-edge technologies. We're also seeing many innovations in storage, like compression, deduplication, clones, snapshots, etc.

Today, with many projects, the complexity make it pretty difficult to foresee resource usage. This makes it hard to budget for hardware that can fulfill capacity and performance requirements in the long term. It's even tougher when the project is still in the planning stages. My question: how do you do capacity planning and performance management for such decentralized systems with diverse technologies? Who is responsible for capacity planning in your company? Are you mostly reactive in adding resources (CPU, memory, IO, storage, etc) or are you able to plan it out well beforehand?

Comments

Only you can decide that.

[mlts]

There are a lot of tools you can use to help with capacity, be it VM farms, SANs/NASes, cloud providers, chassis/blades. Only a few points of advice:

1: Everyone will sell their product as the silver hammer, where each target is a nail. The VNX guys will sell their SAN as a be all and end all, even if you just use CIFS/SMB. The security vendor will be selling you exotic appliances for encryption for your tape silos. The PC guy will be selling you tons of 1U racks and try to convince you that the onboard drive array is better than a SAN if they don't have a SAN product, otherwise, how slick their HBAs work when used with their SAN.

2: Don't forget security. It may be cheaper to have one VM cluster for everything, but it be wiser to keep one client's hyper-sensitive stuff on one VMWare datacenter [1], while the other client who is running some backend stuff for an app would be in a different container.

3: Before committing to purchase something, grab manuals and documentation, and read on the device. You might find it doesn't do what you want. Don't forget to take into account type of I/O and other items. I have had to deal with a terabyte/hour of random writes, and the only solution for that was going with either a caching HBA that had that much SSD so it would turn the random writes into an easy to digest sequential stream for the SAN, or go pure SSD. Sequential I/O is a lot easier and a lot cheaper to deal with than random I/O. Similar with I/O that is often cached versus I/O that never is reused.

[1]: A datacenter is a VMWare object type. Can't vMotion across it, and is intended to provide distinct separation between items.

Anybody get fired for buying too much?

[swb]

Too much network bandwidth? Too much storage capacity? Too much CPU?

Usually the drill seems to involve a lot of begging and pleading for money from management. The intermediate levels get dinged if they have to go back to the well, but they don't seem to get dinged if budgeted money ends up buying unused capacity.

I don't doubt there are places which do heavy audits and ask hard questions about why you have a SAN with a bunch of free space or why your 10 gig NICs are running at sub-gigabit utilization and cause all manner of pain and suffering for excess capacity already budgeted and bought.

But usually it doesn't seem to happen that way. Management barely supplies enough resources to meet their running demands and line management buys as much excess capacity as they can beg, borrow or steal because they know they will be punished for buying too little.