I strongly agree: performance
is a functional requirement,
so test it from the time your app
first replies to a test. If
your target is 1/10 second and you're
only taking 1/100th, stop! it's
fast enough.
Once it's built, re-measure and
see what resources it needs to meet your
speed target per 1000 users, and
stick that in a spreadsheet. use
that to do what-ifs for the hardware
you start with and plan to grow to, so
the manager can know how much
it costs, and price it so you can afford
resources when the business grows.
Yes, one absolutely should start small
and build only as you get the business and a profit margin to pay for the growth!
If he's talking about his own site, and
not a more general problem, then he seems to have staff he can put on tracking the
actual growth and correlate it with
resource usage: that gives him an amount
of resources per N users, and he can
start the plan there. Along with
his current (wildly approximate) growth rate.
The other thing he needs is to know
when he 'hits the wall". Assume one page
view takes 1/10 second at his server,
with a single CPU (queue) active. At a load of 10
page requests, he's at 100% utilization
(of the app/queue, not the cpu) and he's
hit the inflection point. After that,
performance will fall off.
Now he can look at his current growth and say "I'll hit the knee on my 12-processor machine next week: I won't
buy more processors, I'll tell apache to reject anything over 120 simultaneous
page requests, and then have a script send me email".
If he get the email tomorrow, growth is super high. If not until next month, low.
And finally he looks at his budget and decides when to buy more resources,
balancing cost against whatever he makes off the site at his current load.
If he plans a big marketing push and
expects 50% more, then he gets to decide if he'll but resources now or try the marketing effort first and buy only if it works.
--dave (brutally oversimplified, you understand) c-b
Throwing hardware at a problem means
the writer failed to use his sysadmin
staff to do basic capacity planning while
there wasn't a problem.
And as johnlcallaway, said, the
problem isn't usually CPU: most bottlenecks are either disk I/O or code-path length.
I'm a professional capacity planner,
and it seems only the smartest 1% of companies
ever think to bring me in to prevent
problems. A slightly larger percentage
do simple resource planning using
the staff they already have. A
good example of the latter is Flickr,
described by John Allspaw in
The Art of Capacity Planning, where
he found I/O was his problem and
I/O wait time was his critical measurement.
Failing to plan means you'll
hit the knee in the response-time curve, and instead of of a few fractions of a second, response time will increase (degrade) so fast that some of your customers will think you've crashed entirely.
And that in turn becomes the self-fulfilling prophecy that
you've gone out of business (;-()
Alas, the people who fail
to plan seem to be the great majority,
and suffer cruely from their failure.
The last few percent are those unfortunates whose
professional staff planned, warned, and were ignored. Their managers pop up, buy some
CPUs or memory to solve their I/O problem, scream at
their vendor for not solving the problem and then suddenly go quiet.
The hardware usually show up on eBay, so I think you can guess what happened.
Er, it's another Unix and it's been specifically optimized (de-pessimized???)
for x86 desktop use in recent
years. Unless Linux and BSD also don't
qualify, it should.
--dave (who uses both RedHat and Solaris on his laptops) c-b
In that era, the bandwidth to memory was pitiful, too, so anything that made the
instructions smaller was good. We've passed through the inverse case around the time of the first RISC chips, and are now back in that timing regime.
This reminds me of a large company
which outsourced enthusiastically,
until at one point they discovered they'd outsourced decisions about maintenance...
causing the outsourcer to have control over the maintenance budget.
As you might expect, after it ballooned, they started in-sourcing!
Giving others control over financial decisions is almost always unwise,
even if doing so is the newest,
coolest idea of the week.
So I buy a dual or quad-socket box, rather
than a single one, for the class of (database)
app that doesn't scale well horizontally.
If it does scale horizontally, I have the
choice of x86-64 or RISC, whichever gives me the
best bang for the buck, always assuming
datacenter-quality hardware. For good x86s,
the cost are substantially higher than
white-box equipment, so my customers often find
that IBM, Sun or HP machines yields them a
cost saving.
In the specific case of the Sun Niagara,
I see a particular database transaction set take 1.8 seconds on average, versus 1.3 with a faster non-threaded multi-core processor, and the drop-off in
performance under load (the infamous "hockey-stick" curve)
is substantially less with the threaded machine.
And it's about 20% cheaper to buy
and in TCO.
As always, different cases yield different answers. For something parallellizable and
for which individual-node failure and recovery is fine, one
should use a Google-style solution, with
lots of x86s and a retire-on-failure policy.
It's taken a surprisingly long time
to actually catch someone using a camcorder
to record a movie...
since most are stolen from the projection booth.
--dave
Sure, until you get something reasonably large.
I'm a capacity planner, and small customers
for me have 28-36 CPUs, so they're running
IBM, Sun or H-P gear.
Also, large numbers of dispatchable threads
are really good when you're under what one
of my colleagues calls "a normal overload".
The latency growth of a T5240 is much more
gentle than a much faster X64 machine, so
it handles really evil loads well.
To be fair, a lot of x64 threads would be
the best possible case, but I don't think
even my richer customers would put in a beowulf cluster for an app server (;-))
I quite disagree: open source needs something to
run on, and the price-performance of the
Niagara is impressive. In the machine room,
I'd rather have Sun, IBM or H-P gear than
anything built on mas-market PC parts: I hate
fixing critical components (;-))
Open source software, on the other hand,
is improved by being in a mass market: the
price is already as low as you can get, so
the effort goes into improving the quality.
It's very welcome in my machine room.
Sun mostly sells hardware, and adds anything
whatsoever the customer needs to build a
solution. This includes software, obviously,
but also cisco gear, racks, UPSs and the like.
Many moons ago, I wrote a disassembler
that emitted mock "c" syntax, which I could
turn into real C with a few global substitutes.
You can do the same with normal assembler
syntax and an ad-hoc parser, and therefor you
can turn the "accountant's assembler" from
add foo to bar giving zot
to
zot = foo + bar;
This is just a special case of the semantics-preserving transformation problem,
which my old employer used to do on a daily
basis (I worked in a porting shop).
The O'Reilly book, "Using Samba" was published
using a free license, prohibiting only commercial
large-scale printing for profit.
The book was then shipped with Samba, as the
Samba Team's official reference, and people started
reading it online, printing off small chunks
and using it.
When they wanted a complete copy to mark up or to read in the bathtub, they went to O'Reilly and
bought the nice printed copy on thin paper that
you could actually carry in one hand (;-))
Net result: it jumped to the top of Samba
book sales, and was very profitable for O'Reilly.
And all because my editor (Andy Oram) was
smart enough to
realize that he could try an experiment in new media with a little help from the Samba Team
Actually it was a different situation in
the U.S., with the key protecting other
confidential data that the warrant didn't
name.
The police could apply to the court for
a warrant for all the data, or certify
to the court that they would only seize
(copy) the named data. In an extreme
case, they could be required to use
a third party nominated by both prosecution
and the defense to separate them.
In The U.K., the defense argued a
self-incrimination, something that didn't
work for property.
That's a non-emergency one, you understand.
My aunt (now 101) broke a hip in her late
90s. It was replaced the day after, and she
was up walking in a week.
I had chest pains and in an hour I was hooked
to what I swear looked like one of the diagnostic beds
from Star Trek. It was a chest-muscle spasm,
fortunately, and I was released after
overnight observation.
Mind you, to get a patch of skin damage
removed, I had to wait over a month!
They're also blocking access to large numbers
of.org and.ca sites, including the.ca
site of my local member of parliament.
What was particularly annoying is that they set
the tab title to the title from her home page
("Martha: It's Time") and then reported that
the page was inaccessible.
That was the tipoff: the sites can be looked
up in their own DNS, so one can do that and
and connect to then by IP address
That in turn makes it look like they're doing it based
on a buggy rule system.
Slashdot Mirror
Just out of idle curiosity, what do you do?
I strongly agree: performance is a functional requirement, so test it from the time your app first replies to a test. If your target is 1/10 second and you're only taking 1/100th, stop! it's fast enough.
Once it's built, re-measure and see what resources it needs to meet your speed target per 1000 users, and stick that in a spreadsheet. use that to do what-ifs for the hardware you start with and plan to grow to, so the manager can know how much it costs, and price it so you can afford resources when the business grows.
--dave
Yes, one absolutely should start small and build only as you get the business and a profit margin to pay for the growth!
If he's talking about his own site, and not a more general problem, then he seems to have staff he can put on tracking the actual growth and correlate it with resource usage: that gives him an amount of resources per N users, and he can start the plan there. Along with his current (wildly approximate) growth rate.
The other thing he needs is to know when he 'hits the wall". Assume one page view takes 1/10 second at his server, with a single CPU (queue) active. At a load of 10 page requests, he's at 100% utilization (of the app/queue, not the cpu) and he's hit the inflection point. After that, performance will fall off.
Now he can look at his current growth and say "I'll hit the knee on my 12-processor machine next week: I won't buy more processors, I'll tell apache to reject anything over 120 simultaneous page requests, and then have a script send me email".
If he get the email tomorrow, growth is super high. If not until next month, low.
And finally he looks at his budget and decides when to buy more resources, balancing cost against whatever he makes off the site at his current load.
If he plans a big marketing push and expects 50% more, then he gets to decide if he'll but resources now or try the marketing effort first and buy only if it works.
--dave (brutally oversimplified, you understand) c-b
Throwing hardware at a problem means the writer failed to use his sysadmin staff to do basic capacity planning while there wasn't a problem.
And as johnlcallaway, said, the problem isn't usually CPU: most bottlenecks are either disk I/O or code-path length.
I'm a professional capacity planner, and it seems only the smartest 1% of companies ever think to bring me in to prevent problems. A slightly larger percentage do simple resource planning using the staff they already have. A good example of the latter is Flickr, described by John Allspaw in The Art of Capacity Planning, where he found I/O was his problem and I/O wait time was his critical measurement.
Failing to plan means you'll hit the knee in the response-time curve, and instead of of a few fractions of a second, response time will increase (degrade) so fast that some of your customers will think you've crashed entirely.
And that in turn becomes the self-fulfilling prophecy that you've gone out of business (;-()
Alas, the people who fail to plan seem to be the great majority, and suffer cruely from their failure. The last few percent are those unfortunates whose professional staff planned, warned, and were ignored. Their managers pop up, buy some CPUs or memory to solve their I/O problem, scream at their vendor for not solving the problem and then suddenly go quiet. The hardware usually show up on eBay, so I think you can guess what happened.
--dave
Er, it's another Unix and it's been specifically optimized (de-pessimized???) for x86 desktop use in recent years. Unless Linux and BSD also don't qualify, it should.
--dave (who uses both RedHat and Solaris on his laptops) c-b
And BSD, which currently supports ZFS as well.
--dave
In that era, the bandwidth to memory was pitiful, too, so anything that made the instructions smaller was good. We've passed through the inverse case around the time of the first RISC chips, and are now back in that timing regime.
--dave
One might say the same thing about p-code (;-))
--dave
It isn't every day that the glossies "get it", or even pay attention.
Next: the Wall Street Journal (;-))
--dave
This reminds me of a large company which outsourced enthusiastically, until at one point they discovered they'd outsourced decisions about maintenance... causing the outsourcer to have control over the maintenance budget.
As you might expect, after it ballooned, they started in-sourcing!
Giving others control over financial decisions is almost always unwise, even if doing so is the newest, coolest idea of the week.
--dave
So I buy a dual or quad-socket box, rather than a single one, for the class of (database) app that doesn't scale well horizontally.
If it does scale horizontally, I have the choice of x86-64 or RISC, whichever gives me the best bang for the buck, always assuming datacenter-quality hardware. For good x86s, the cost are substantially higher than white-box equipment, so my customers often find that IBM, Sun or HP machines yields them a cost saving.
In the specific case of the Sun Niagara, I see a particular database transaction set take 1.8 seconds on average, versus 1.3 with a faster non-threaded multi-core processor, and the drop-off in performance under load (the infamous "hockey-stick" curve) is substantially less with the threaded machine. And it's about 20% cheaper to buy and in TCO.
As always, different cases yield different answers. For something parallellizable and for which individual-node failure and recovery is fine, one should use a Google-style solution, with lots of x86s and a retire-on-failure policy.
--dave
It's taken a surprisingly long time to actually catch someone using a camcorder to record a movie... since most are stolen from the projection booth. --dave
Sure, until you get something reasonably large. I'm a capacity planner, and small customers for me have 28-36 CPUs, so they're running IBM, Sun or H-P gear.
Also, large numbers of dispatchable threads are really good when you're under what one of my colleagues calls "a normal overload". The latency growth of a T5240 is much more gentle than a much faster X64 machine, so it handles really evil loads well.
To be fair, a lot of x64 threads would be the best possible case, but I don't think even my richer customers would put in a beowulf cluster for an app server (;-))
--dave
App server as you might guess, and databases with SAN storage.
I quite disagree: open source needs something to run on, and the price-performance of the Niagara is impressive. In the machine room, I'd rather have Sun, IBM or H-P gear than anything built on mas-market PC parts: I hate fixing critical components (;-))
Open source software, on the other hand, is improved by being in a mass market: the price is already as low as you can get, so the effort goes into improving the quality. It's very welcome in my machine room.
--dave
Sun mostly sells hardware, and adds anything whatsoever the customer needs to build a solution. This includes software, obviously, but also cisco gear, racks, UPSs and the like.
--dave (biased, you understand) c-b
ZFS does what I consider raid 10, 5 and 6.
--dave
In Canada it would be theft of computer and telecommunications service, for the purpose of delivering an unasked-for ad.
--dave
Many moons ago, I wrote a disassembler that emitted mock "c" syntax, which I could turn into real C with a few global substitutes.
You can do the same with normal assembler syntax and an ad-hoc parser, and therefor you can turn the "accountant's assembler" from add foo to bar giving zot to zot = foo + bar;
This is just a special case of the semantics-preserving transformation problem, which my old employer used to do on a daily basis (I worked in a porting shop).
And yes, I'll do this for money if asked (;-))
--dave
The O'Reilly book, "Using Samba" was published using a free license, prohibiting only commercial large-scale printing for profit.
The book was then shipped with Samba, as the Samba Team's official reference, and people started reading it online, printing off small chunks and using it.
When they wanted a complete copy to mark up or to read in the bathtub, they went to O'Reilly and bought the nice printed copy on thin paper that you could actually carry in one hand (;-))
Net result: it jumped to the top of Samba book sales, and was very profitable for O'Reilly.
And all because my editor (Andy Oram) was smart enough to realize that he could try an experiment in new media with a little help from the Samba Team
--dave
Actually it was a different situation in the U.S., with the key protecting other confidential data that the warrant didn't name.
The police could apply to the court for a warrant for all the data, or certify to the court that they would only seize (copy) the named data. In an extreme case, they could be required to use a third party nominated by both prosecution and the defense to separate them.
In The U.K., the defense argued a self-incrimination, something that didn't work for property.
--dave
That's a non-emergency one, you understand. My aunt (now 101) broke a hip in her late 90s. It was replaced the day after, and she was up walking in a week.
I had chest pains and in an hour I was hooked to what I swear looked like one of the diagnostic beds from Star Trek. It was a chest-muscle spasm, fortunately, and I was released after overnight observation.
Mind you, to get a patch of skin damage removed, I had to wait over a month!
--dave
Rogers in Canada is one who does that, then forges a search page for your convenience (;-))
Worse, they do the same for many valid .ca and .org sites.
--dave
Some of the small resellers buy raw bandwidth, so you can avoid the brain-damage.
--dave
They're also blocking access to large numbers of .org and .ca sites, including the .ca
site of my local member of parliament.
What was particularly annoying is that they set the tab title to the title from her home page ("Martha: It's Time") and then reported that the page was inaccessible.
That was the tipoff: the sites can be looked up in their own DNS, so one can do that and and connect to then by IP address
That in turn makes it look like they're doing it based on a buggy rule system.
--dave