...I usually start with documenting the configuration on a Wiki (or a file in a Subversion repository somewhere) and then shift things over to Puppet when I get the chance. The nice thing about the latter is that you know all the setup specifications are correct since that's what's actually being applied to the servers.
Documentation of system provisioning is just one small part of the question you're asking, but hopefully that helps.
> all those little hacks you avoid during > development could suddenly mean thousands of dollars saved
Yeah, that's the stuff I'm thinking of. "We won't generate this on the fly, we'll put it in a queue and process the queue every 5 seconds." "We won't immediately write this to all the servers, instead we'll rsync it up in a batch job." I wouldn't even call those "little hacks"... I'd say that those are architecture decisions that you make once you realize you need them. And, as you say, until you need that sort of thing it's a waste of time to code it up... wait until you're sure you have customers, then do all that.
Well said sir. Scaling is not about language opcode execution speed. Apps don't scale by eliminating a few JMPs and BEQs here and there. Instead, they scale by coming up with good _architectures_. Caching, partitioning, sharding, queuing, backgrounding... all that stuff. Given a good enough architect you could probably write EBay in Bash.
...I've been using Sphinx a lot recently and have been really pleased with it. The indexer is fast, there's good Ruby on Rails integration, and I don't worry about scalability since if it's good enough for craigslist it's good enough for me. Definitely worth a look for your next project that needs to do full text search.
> The Campaign for Nuclear Disarmament described the > collision as "a nuclear nightmare of the highest order".
The problem with this sort of overblown rhetoric is that it uses up the effect of these words. Had both submarines detonated all their warheads, that would have been "a nuclear nightmare of the highest order". This incident is just part of having a Navy. As Grace Hopper said, "A ship in port is safe, but that's not what ships are built for".
He may be lurking hereabouts, but if not, here's his bio. I've been doing open source for a fair while - 10 years or so - but he's been talking to companies and coming up with good answers to various arguments against open source for much longer.
One thing to watch for with mod_security is that it will block certain requests even when it's in 'logging only' mode. Still a useful tool, but keep a close eye on it.
That said, I kind of feel like MySQL still has an edge on PostgreSQL in terms of replication - I know Slony is out there, but my perception of it is that it doesn't handle schema changes easily, and MySQL replication via binary log shipping is pretty straightforward. I hear the PostgreSQL guys are working on something for out of the box replication though, so we'll see....
> I recently switched to sphinx (http://www.sphinxsearch.com/) its written in C
Minor nit - it's in C++. But yeah, it's totally awesome - fast when indexing, easy to scale horizontally, powerful query language, custom stop word lists, etc, etc. The APIs (I use the Ruby one, Riddle) make it easy to do nifty excerpt formatting (for example, note the highlighting around the word 'battle'), and there are a couple of different ways to integrate it into a Ruby on Rails app.
Speaking of Sphinx and Rails, here's a code snippet for escaping extended mode Sphinx queries. This will probably make its way into Riddle at some point, but, until then, there it is.
> What is the weird fascination with "eastern" stuff among upper middle management?
That extends to military leadership too; "The Art of War" is on both the Army and the Marine Corps reading lists. It's a little more appropriate in those cases, though...
...Air Force reading list. Several books about counterinsurgency, only one about flying airplanes and that's a historical piece (Fast Tanks and Heavy Bombers: Innovation in the U.S. Army, 1917-1945). That's the CSAF's reading list, and he would know where they're going...
...I'm not sure how accurate that is. In my experience S3 and EC2 enable small companies to do things they might not otherwise hassle with.
The article also says "The glory days of the UNIX system administrator and the Java programmer are dead and buried". Really? From what I've seen, good Unix sysadmins are in high demand - whether the servers are in your colo rack or in a RackSpace facility, you still need someone to mind the farm and twiddle the Puppet manifests. Not sure about Java programmers, but demand for Ruby (especially Rails) programmers is quite high.
> One of the biggest aces in the hole was PostgreSQL. > The cost for us to come in, set up and install everything > was cheaper than some other well known DB vendor's cost of database software alone.
Right on. Same goes for Sphinx vs any proprietary text indexing setup I've used. And it's fast, and there's good Rails/Ruby support via UltraSphinx and Riddle.
A good way to overcome this is to say "sometimes some junk gets in the plugs... unplug your adapter and then plug it in again." That way if the adapter is indeed unplugged, the person doesn't have to admit it - they can just plug it in and save face by saying "oh yes, it must have gotten loose or dirty or something". Seems like a good strategy.
And speaking of Rails, check out this job posting. Requirements are TS/SCI, Lotus Notes, and... Ruby on Rails experience. Seems like some high security clearance government folks may be converting some things from Notes to Rails.
Slashdot Mirror
...announces the Hope'N'Change Operating System. "Only 30% chance of crashing!"
...Exploring the Internet: A Technical Travelogue. He relates a couple of funny episodes dealing with various ISO bureaucrats... good stuff.
...I usually start with documenting the configuration on a Wiki (or a file in a Subversion repository somewhere) and then shift things over to Puppet when I get the chance. The nice thing about the latter is that you know all the setup specifications are correct since that's what's actually being applied to the servers.
Documentation of system provisioning is just one small part of the question you're asking, but hopefully that helps.
> all those little hacks you avoid during
> development could suddenly mean thousands of dollars saved
Yeah, that's the stuff I'm thinking of. "We won't generate this on the fly, we'll put it in a queue and process the queue every 5 seconds." "We won't immediately write this to all the servers, instead we'll rsync it up in a batch job." I wouldn't even call those "little hacks"... I'd say that those are architecture decisions that you make once you realize you need them. And, as you say, until you need that sort of thing it's a waste of time to code it up... wait until you're sure you have customers, then do all that.
> better to deal with them sooner than later.
Well said sir. Scaling is not about language opcode execution speed. Apps don't scale by eliminating a few JMPs and BEQs here and there. Instead, they scale by coming up with good _architectures_. Caching, partitioning, sharding, queuing, backgrounding... all that stuff. Given a good enough architect you could probably write EBay in Bash.
...I've been using Sphinx a lot recently and have been really pleased with it. The indexer is fast, there's good Ruby on Rails integration, and I don't worry about scalability since if it's good enough for craigslist it's good enough for me. Definitely worth a look for your next project that needs to do full text search.
For a quick demo of it, do some searches here.
> That quote is more likely to be
Fair enough, thanks for the correction!
> The Campaign for Nuclear Disarmament described the
> collision as "a nuclear nightmare of the highest order".
The problem with this sort of overblown rhetoric is that it uses up the effect of these words. Had both submarines detonated all their warheads, that would have been "a nuclear nightmare of the highest order". This incident is just part of having a Navy. As Grace Hopper said, "A ship in port is safe, but that's not what ships are built for".
> 1 Availability of Service Use Multiple Cloud Providers; Use Elasticity to Prevent DDOS
They should add another one:
11 Use Hard Limits to Prevent Bank Account from Being Drained During Prolonged Slashdotting
He may be lurking hereabouts, but if not, here's his bio. I've been doing open source for a fair while - 10 years or so - but he's been talking to companies and coming up with good answers to various arguments against open source for much longer.
Er, Austin... we won!
> I recommend mod_security
One thing to watch for with mod_security is that it will block certain requests even when it's in 'logging only' mode. Still a useful tool, but keep a close eye on it.
> Yeah, I use Postgresql a lot these days, but I also still use MySQL.
I'm a big fan of PostgreSQL; I recently upgraded RubyForge to PostgreSQL 8.3 and have been quite happy with the performance.
That said, I kind of feel like MySQL still has an edge on PostgreSQL in terms of replication - I know Slony is out there, but my perception of it is that it doesn't handle schema changes easily, and MySQL replication via binary log shipping is pretty straightforward. I hear the PostgreSQL guys are working on something for out of the box replication though, so we'll see....
> Kinda makes you wonder if government intervention is really necessary.
I think the government already has, and with predictable effects; for example, see this post about 'negative pricing' in the Texas wind farms.
> I recently switched to sphinx (http://www.sphinxsearch.com/) its written in C
Minor nit - it's in C++. But yeah, it's totally awesome - fast when indexing, easy to scale horizontally, powerful query language, custom stop word lists, etc, etc. The APIs (I use the Ruby one, Riddle) make it easy to do nifty excerpt formatting (for example, note the highlighting around the word 'battle'), and there are a couple of different ways to integrate it into a Ruby on Rails app.
Speaking of Sphinx and Rails, here's a code snippet for escaping extended mode Sphinx queries. This will probably make its way into Riddle at some point, but, until then, there it is.
...and, back online; here's the Ruby/ASP project which also hosts ruby-graphviz.
...which I just took offline for a quick database upgrade. Er, sorry, will be back online soon!
> What is the weird fascination with "eastern" stuff among upper middle management?
That extends to military leadership too; "The Art of War" is on both the Army and the Marine Corps reading lists. It's a little more appropriate in those cases, though...
...Air Force reading list. Several books about counterinsurgency, only one about flying airplanes and that's a historical piece (Fast Tanks and Heavy Bombers: Innovation in the U.S. Army, 1917-1945). That's the CSAF's reading list, and he would know where they're going...
How you get a job like 'Director of Green'?
...I'm not sure how accurate that is. In my experience S3 and EC2 enable small companies to do things they might not otherwise hassle with.
The article also says "The glory days of the UNIX system administrator and the Java programmer are dead and buried". Really? From what I've seen, good Unix sysadmins are in high demand - whether the servers are in your colo rack or in a RackSpace facility, you still need someone to mind the farm and twiddle the Puppet manifests. Not sure about Java programmers, but demand for Ruby (especially Rails) programmers is quite high.
> One of the biggest aces in the hole was PostgreSQL.
> The cost for us to come in, set up and install everything
> was cheaper than some other well known DB vendor's cost of database software alone.
Right on. Same goes for Sphinx vs any proprietary text indexing setup I've used. And it's fast, and there's good Rails/Ruby support via UltraSphinx and Riddle.
> the power cable had fallen out of the adapter.
A good way to overcome this is to say "sometimes some junk gets in the plugs... unplug your adapter and then plug it in again." That way if the adapter is indeed unplugged, the person doesn't have to admit it - they can just plug it in and save face by saying "oh yes, it must have gotten loose or dirty or something". Seems like a good strategy.
And speaking of Rails, check out this job posting. Requirements are TS/SCI, Lotus Notes, and... Ruby on Rails experience. Seems like some high security clearance government folks may be converting some things from Notes to Rails.
> I much prefer a pure-Ruby solution
> which I can throw behind any old load-balancer
Right on, yup, that sounds better to me too. But, for now, Passenger is a vast improvement over mongrel_cluster.yml and all that.