So, if there is a better way, I am dying to hear it, unfortunately solaris/freebsd is not an option...
Why not? Given the amount of pain you've endured under Linux, the costs of a transition to a new platform are small. Your time isn't free. Use the right tool for the job, man.
Caching is only worthwhile if the data can benefit from higher bandwidth.
No, caching is only worthwhile if the application/ssytem can benefit from lower latency. Increased bandwidth is generally something of a side-effect of that, but any sustained high-bandwidth operation will quickly overrun or saturate your cache.
Since moving to an Intel X-25M in my laptop, I must say I can never go back to a spinning disk for my worksation. It's now so much faster than my work desktop (which has a 2x7200K RAID-0 setup and more RAM and better CPU) that I will probably dump my desktop at the office.
But anyway, the real big-win for SSDs is in the server space. We can't get enough IOPS for some of our databases, which already have 64 GB of RAM as cache. Even some of our web servers that have massive content libraries are disk-bound. Our SAN vendor (LeftHand now HP) has really dropped the ball on this one. We're shopping only because they don't offer SSD cluster nodes yet. They probably don't want to cannibalize sales on their higer-end monolithic SANs I suppose./p
Snapshots are not a substitute for off-site backups. Think fire, theft, flood, filesystem or disk controller corruption. Do you rsync those snapshots to an off-site system?
We actually had a lot of software problems as well. State management for NAT was a mess for a while, and the "reboot" solution was often the only thing that helped. We also had our site-to-site VPNs drop randomly every 4-6 hours, even though other devices (Sonciwalls and Ciscos) worked fine in parallel over the exact same infrastructure. Neither issue was ever resolved to our satisfaction.
I'm talking about the CheckPoint/Nokia IP350, which wasn't even introduced until late 2002. Not fair comparing old and new, I suppose, but we had lots of trouble with those little bastards over the years.
We use Sonicwall's enterprise gear (a pair of 4500s at each site) and they're quite stable, and have an assload of features (VPN, IPS, QoS, HA, web filtering, you name it). Way better than the Checkpoints they replaced. The opaque config file is the only major drawback I've run into (all we can do is version it and roll back, no merging, search-and-replace, etc.)
Except the whole decompilation step violates the license agreement, and then the vendor can sue you personally, sue your employer, or just terminate your license for cause. Sure, you might make arguments about interoperability, research, and other fair-uses for the decompilation, but I don't want to be on the wrong end of that legal bill.
Don't give cause people with more money and lawyers than you unless you're prepared for a long, expensive fight based on your principles. If you can get the EFF or some other angel to agree to back you beforehand, go ahead, but I wouldn't risk losing my job or my home over something like this. There's no moral imperative to be a martyr when dealing with a software company.
Microsoft dropped the "database on raw partition" in 2005 as I recall, because the performance advantage was just a few percent even if your workload was totally disk-bound. Also, not having an actual file to move around, resize, etc was so inflexible that it just wasn't worth it. Finally, I'm sure a lot of idiot admins set this up, then had some other admin accidentally reformat the partition containing valuable data later on.
Most common server-based applications are embarrassingly parallel, so long as user interaction is involved. Multiple users = multiple threads. If your server application isn't parallel, you're doing something really wrong in your code, or doing something pretty uncommon.
But there is no solution to the exact same issue on Linux, is there? It's not a preventable issue with traditional file-system semantics, which is why you need something like a document management system to address it.
I'd guess more like 20, which means you have a factor of 10 difference between peak and average rather than my "conservative" 2.
Reasonably close. Aritmetic mean is 31 req/s. My point was that planning for 60 req/s would leave us short of capacity forty hours of every week! The overwhelming majority of sites have regional audiences, and most other sites I've been involved with have similar patterns. Truly global sites are quite rare.
You're probably almost entirely US-based?)
Yep... a little Hawaii, Guam, Europe and Middle East in there too.
I accept that there are exceptions, but the idea that most people need a complex setup to scale to 10M requests per day doesn't seem right, and your post doesn't change my mind about that.
No, I agree with you entirely. Our application actually uses a single (replicated for HA and reporting) commercial SQL database instance. It isn't the bottleneck, even at full load. We have, like you, many application servers, and only a few static content servers.
The NoSQL crowd is locking their data away, hiding it from standard analysis and reporting tools. Not to mention leaving relational integrity up to the application layer (which typically doesn't have transactional semantics.) Both are dumb moves unless you have absolutely no choice (like Google or Amazon).
The only thing I disagreed with was the "two times average load" example you used.
Let's be conservative and say that you need to handle peak load of twice average
That's not conservative at all.
Most business-related sites are significantly more "spikey" than what you illustrate. Our 50th percentile is about 4 dynamic page requests per second. 95th percentile is about 125 req/sec. 99.9% is 202 req/sec and peak is 400 dynamic page req/sec (times about six if you include static items).
So we target the 99.9% for capacity planning, but that still leaves ten minutes per week where the application is quite slow. (It happens around 13;00 on Wednesdays US Central US time.)
Capacity planning is hard to do right, and is never as simple as "two times average load"
Well, it is an edge case to be sure. Files with permissions different than their parent directory, but also include permissions which the file editor cannot sensibly set themselves, are definitely a rarity. We have about 1M office documents on Windows servers here, and have never encountered the isse in over 12 years.
I'm fairly certain that SQL Server inherited its TIMESTAMP keyword from Sybase, and that usage of TIMESTAMP pre-date SQL-89 and SQL-92 usages of that keyword.
In short, they can't fix it properly, because it would break a ton of existing (very critical) applications that use the existing Sybase and MSSQL semantics of TIMESTAMP. Microsoft deprecated its usage of TIMESTAMP long ago, but they can't just change it without pissing off a lot of people. Oracle is in the same boat with many of its features that "violate" the ANSI standards.
It's sort of like bitching about IE6 not supporting CSS2 features. IE6 predated the CSS2 standards ratification. It's actually the fault of those writing the standards: they ignored widely-used software and practices. In this case, they chose to use the TIMESTAMP keyword when something like DATEWITHTIME would have been clearer and would not have collided with anybody.
In my experience, MSSQL is actually the most ANSI-compliant of the major commercial DBs.
You clearly don't know what you're talking about. MSSQL has never lost a byte of data on me since 1996, and one of our SQL server apps has 700,000 users doing 1200 requests per second. You've never actually used it in production, have you? It is by far Microsoft's best product. It is also the most ANSI-compliant of the majort databases (save Postgresql of course).
Software patents aren't evil, they're simply counter-productive and inefficient as the result of a huge government bureaucracy fucking things up. There are certain bits of software that are so innovative they should be patentable. Usually these are very complex or original algorithms found in academic circles or in complicated applications like signal/image processing. For example, I think quicksort is patent-worthy, as well as DCT-based lossy audio compression. But the USPTO fucked it up and granted patents to everybody for everything, just so some bureaucrat could get justify getting his budget increased year after year.
To start a flame-war, I wonder why so many bemoan the US government fucking up copyright, patents, and broadband policy yet want the same government to run our healthcare system.
I actually only know of one major "enterprise" vendor that requires active maintenance for patches (security or otherwise): Cisco. Lots of minor "enterprise-ish" software vendors do the same. They won't even speak with you without a maintenance agreement, and you also can't download anything without one.
Regarding your signature, can't any system, including *NIX systems which use ACLs can also suffer the exact same issue? Having permission to write a file does not imply permission to change its ACLs. So the whole "write temp file and atomically rename" paradigm causes trouble.
My reading of the article makes me think I may have been dealing this problem while editing the configuration files on a Linux-based storage appliance.
Iraq had the world's fourth largest standing Army in 1990 at the start of Gulf War I. Modern coalition forces measuring perhaps 1/3 their destroyed the Iraqis in 100 hours of ground war, at a cost of some 150 coalition casualties. And on a per-unit basis, the Iraqis of 1990 were likely better and more modernly equipped than the average PLA unit of today.
As for the Russians in Stalingrad, winter weather and logistical ineptitute is what really stopped zee Germans. The Wheremacht was able to penetrate Russia all the way to Stalingrad before being cut off and overwhelmed, despite numerical inferiority in men (but not materiel). Too far, too fast. And at least according to Wikipedia, the number of troops at the time of the Russian counter-offensive was roughly equal (~1M apiece). So it wasn't quantity that made the Russians win, it was stubborn bravery, plenitful ammunition, and the ability to fight in the Russian winter.
Number of troops stopped being an effective metric of military power about 100 years ago. In infantry combat, training and leadership counts for much more (see the Israelis and US ground forces in every conflict since WWII). The Russians used to say "quantity has a quality all its own", but they were kidding themselves.
If you then consider the enormous advantages in armor, naval, and air power that a country like the USA, Russia, UK, etc. would have over China in a conflict, and China is very much a paper tiger.
Even the Chinese know this. The massive size of the Chinese military serves political purposes. First, it creates a large number of people loyal to the government because that's where their bread is buttered. And it also provides them with the means to police/coerce/control 1.X billion people.
If the server admins at your co-location provider really are making minimum wage, you need to find another provider, pronto. When selecting vendors, due diligence is not optional. A simple facilities tour where you get to bounce questions off the admins would reveal that they are totally unqualified (which is the only way they would take such a job for minimum wage.)
You did tour your co-location provider's facilities before signing up, didn't you? And investigated their financials (under NDA if necessary)? If something about their salary costs look wonky, and cannot be explained, and their pricing is too good to be true, you stay away.
That was tried before, many years ago, but abandoned as far as I can tell. I imagine they had a good reason. Probably the fact that two head motor assemblies means they would have to make the platter smaller to fit in a standard form factor case.
Slashdot Mirror
So, if there is a better way, I am dying to hear it, unfortunately solaris/freebsd is not an option...
Why not? Given the amount of pain you've endured under Linux, the costs of a transition to a new platform are small. Your time isn't free. Use the right tool for the job, man.
Caching is only worthwhile if the data can benefit from higher bandwidth.
No, caching is only worthwhile if the application/ssytem can benefit from lower latency. Increased bandwidth is generally something of a side-effect of that, but any sustained high-bandwidth operation will quickly overrun or saturate your cache.
Since moving to an Intel X-25M in my laptop, I must say I can never go back to a spinning disk for my worksation. It's now so much faster than my work desktop (which has a 2x7200K RAID-0 setup and more RAM and better CPU) that I will probably dump my desktop at the office.
But anyway, the real big-win for SSDs is in the server space. We can't get enough IOPS for some of our databases, which already have 64 GB of RAM as cache. Even some of our web servers that have massive content libraries are disk-bound. Our SAN vendor (LeftHand now HP) has really dropped the ball on this one. We're shopping only because they don't offer SSD cluster nodes yet. They probably don't want to cannibalize sales on their higer-end monolithic SANs I suppose./p
Snapshots are not a substitute for off-site backups. Think fire, theft, flood, filesystem or disk controller corruption. Do you rsync those snapshots to an off-site system?
We actually had a lot of software problems as well. State management for NAT was a mess for a while, and the "reboot" solution was often the only thing that helped. We also had our site-to-site VPNs drop randomly every 4-6 hours, even though other devices (Sonciwalls and Ciscos) worked fine in parallel over the exact same infrastructure. Neither issue was ever resolved to our satisfaction.
I'm talking about the CheckPoint/Nokia IP350, which wasn't even introduced until late 2002. Not fair comparing old and new, I suppose, but we had lots of trouble with those little bastards over the years.
We use Sonicwall's enterprise gear (a pair of 4500s at each site) and they're quite stable, and have an assload of features (VPN, IPS, QoS, HA, web filtering, you name it). Way better than the Checkpoints they replaced. The opaque config file is the only major drawback I've run into (all we can do is version it and roll back, no merging, search-and-replace, etc.)
Except the whole decompilation step violates the license agreement, and then the vendor can sue you personally, sue your employer, or just terminate your license for cause. Sure, you might make arguments about interoperability, research, and other fair-uses for the decompilation, but I don't want to be on the wrong end of that legal bill.
Don't give cause people with more money and lawyers than you unless you're prepared for a long, expensive fight based on your principles. If you can get the EFF or some other angel to agree to back you beforehand, go ahead, but I wouldn't risk losing my job or my home over something like this. There's no moral imperative to be a martyr when dealing with a software company.
Microsoft dropped the "database on raw partition" in 2005 as I recall, because the performance advantage was just a few percent even if your workload was totally disk-bound. Also, not having an actual file to move around, resize, etc was so inflexible that it just wasn't worth it. Finally, I'm sure a lot of idiot admins set this up, then had some other admin accidentally reformat the partition containing valuable data later on.
Why is noone asking to examine the C programming language, to adapt it to modern programming processes and methods?
They have, and it's called the D Programming Language. And it sucks.
Most common server-based applications are embarrassingly parallel, so long as user interaction is involved. Multiple users = multiple threads. If your server application isn't parallel, you're doing something really wrong in your code, or doing something pretty uncommon.
But there is no solution to the exact same issue on Linux, is there? It's not a preventable issue with traditional file-system semantics, which is why you need something like a document management system to address it.
I'd guess more like 20, which means you have a factor of 10 difference between peak and average rather than my "conservative" 2.
Reasonably close. Aritmetic mean is 31 req/s. My point was that planning for 60 req/s would leave us short of capacity forty hours of every week! The overwhelming majority of sites have regional audiences, and most other sites I've been involved with have similar patterns. Truly global sites are quite rare.
You're probably almost entirely US-based?)
Yep... a little Hawaii, Guam, Europe and Middle East in there too.
I accept that there are exceptions, but the idea that most people need a complex setup to scale to 10M requests per day doesn't seem right, and your post doesn't change my mind about that.
No, I agree with you entirely. Our application actually uses a single (replicated for HA and reporting) commercial SQL database instance. It isn't the bottleneck, even at full load. We have, like you, many application servers, and only a few static content servers.
The NoSQL crowd is locking their data away, hiding it from standard analysis and reporting tools. Not to mention leaving relational integrity up to the application layer (which typically doesn't have transactional semantics.) Both are dumb moves unless you have absolutely no choice (like Google or Amazon).
The only thing I disagreed with was the "two times average load" example you used.
Let's be conservative and say that you need to handle peak load of twice average
That's not conservative at all.
Most business-related sites are significantly more "spikey" than what you illustrate. Our 50th percentile is about 4 dynamic page requests per second. 95th percentile is about 125 req/sec. 99.9% is 202 req/sec and peak is 400 dynamic page req/sec (times about six if you include static items).
So we target the 99.9% for capacity planning, but that still leaves ten minutes per week where the application is quite slow. (It happens around 13;00 on Wednesdays US Central US time.)
Capacity planning is hard to do right, and is never as simple as "two times average load"
Well, it is an edge case to be sure. Files with permissions different than their parent directory, but also include permissions which the file editor cannot sensibly set themselves, are definitely a rarity. We have about 1M office documents on Windows servers here, and have never encountered the isse in over 12 years.
I'm fairly certain that SQL Server inherited its TIMESTAMP keyword from Sybase, and that usage of TIMESTAMP pre-date SQL-89 and SQL-92 usages of that keyword.
In short, they can't fix it properly, because it would break a ton of existing (very critical) applications that use the existing Sybase and MSSQL semantics of TIMESTAMP. Microsoft deprecated its usage of TIMESTAMP long ago, but they can't just change it without pissing off a lot of people. Oracle is in the same boat with many of its features that "violate" the ANSI standards.
It's sort of like bitching about IE6 not supporting CSS2 features. IE6 predated the CSS2 standards ratification. It's actually the fault of those writing the standards: they ignored widely-used software and practices. In this case, they chose to use the TIMESTAMP keyword when something like DATEWITHTIME would have been clearer and would not have collided with anybody.
In my experience, MSSQL is actually the most ANSI-compliant of the major commercial DBs.
You clearly don't know what you're talking about. MSSQL has never lost a byte of data on me since 1996, and one of our SQL server apps has 700,000 users doing 1200 requests per second. You've never actually used it in production, have you? It is by far Microsoft's best product. It is also the most ANSI-compliant of the majort databases (save Postgresql of course).
Software patents aren't evil, they're simply counter-productive and inefficient as the result of a huge government bureaucracy fucking things up. There are certain bits of software that are so innovative they should be patentable. Usually these are very complex or original algorithms found in academic circles or in complicated applications like signal/image processing. For example, I think quicksort is patent-worthy, as well as DCT-based lossy audio compression. But the USPTO fucked it up and granted patents to everybody for everything, just so some bureaucrat could get justify getting his budget increased year after year.
To start a flame-war, I wonder why so many bemoan the US government fucking up copyright, patents, and broadband policy yet want the same government to run our healthcare system.
I actually only know of one major "enterprise" vendor that requires active maintenance for patches (security or otherwise): Cisco. Lots of minor "enterprise-ish" software vendors do the same. They won't even speak with you without a maintenance agreement, and you also can't download anything without one.
Regarding your signature, can't any system, including *NIX systems which use ACLs can also suffer the exact same issue? Having permission to write a file does not imply permission to change its ACLs. So the whole "write temp file and atomically rename" paradigm causes trouble. My reading of the article makes me think I may have been dealing this problem while editing the configuration files on a Linux-based storage appliance.
Iraq had the world's fourth largest standing Army in 1990 at the start of Gulf War I. Modern coalition forces measuring perhaps 1/3 their destroyed the Iraqis in 100 hours of ground war, at a cost of some 150 coalition casualties. And on a per-unit basis, the Iraqis of 1990 were likely better and more modernly equipped than the average PLA unit of today.
As for the Russians in Stalingrad, winter weather and logistical ineptitute is what really stopped zee Germans. The Wheremacht was able to penetrate Russia all the way to Stalingrad before being cut off and overwhelmed, despite numerical inferiority in men (but not materiel). Too far, too fast. And at least according to Wikipedia, the number of troops at the time of the Russian counter-offensive was roughly equal (~1M apiece). So it wasn't quantity that made the Russians win, it was stubborn bravery, plenitful ammunition, and the ability to fight in the Russian winter.
Number of troops stopped being an effective metric of military power about 100 years ago. In infantry combat, training and leadership counts for much more (see the Israelis and US ground forces in every conflict since WWII). The Russians used to say "quantity has a quality all its own", but they were kidding themselves.
If you then consider the enormous advantages in armor, naval, and air power that a country like the USA, Russia, UK, etc. would have over China in a conflict, and China is very much a paper tiger.
Even the Chinese know this. The massive size of the Chinese military serves political purposes. First, it creates a large number of people loyal to the government because that's where their bread is buttered. And it also provides them with the means to police/coerce/control 1.X billion people.
(you'll get more by putting the money in the bank)
Umm.... not really. Most consumer savings acount now return less than 1%, and the very best are only about 1.5% APR.
Banks ain't what they used to be.
If the server admins at your co-location provider really are making minimum wage, you need to find another provider, pronto. When selecting vendors, due diligence is not optional. A simple facilities tour where you get to bounce questions off the admins would reveal that they are totally unqualified (which is the only way they would take such a job for minimum wage.)
You did tour your co-location provider's facilities before signing up, didn't you? And investigated their financials (under NDA if necessary)? If something about their salary costs look wonky, and cannot be explained, and their pricing is too good to be true, you stay away.
That was tried before, many years ago, but abandoned as far as I can tell. I imagine they had a good reason. Probably the fact that two head motor assemblies means they would have to make the platter smaller to fit in a standard form factor case.