Dev / Testing / Staging-Acceptance-PreProduction / Production
Each developer should get their own virtualhost on the dev server that can mimic production (apache virtualhosts if possible, or use vmware), that they can upload and self-test to. Use wildcard dns for hostnames like username.dev.company.com.
Depending on how many paralell things are tested, you might want more testing systems. Testing boxes should be mirrors of production software-wise (or at least as close as you feel comfortable).
Staging/Preproduction might be optional if you're a small shop or don't need acceptance testing - this should be a more stable place to do system tests outside of the possibly buggy testing environ. again, should mimic production, possibly closer.
production - obvious.
If you can do it, do deployments from SVN via labels or tags (not sure which it uses) like RELEASE_2006-12-27_NOTE (and something like RELEASE_LIVE which moves), so you can track revisions which are on production. Instead of copying code up, you use svn to sync code down, probably through some kickoff script. this occurs to each environment. Take everyone's access away from production, and probably staging - testing you might give select people access. Only the people in charge of the machines themselves, and whoever's in charge of releasing code should have access. Use automation with secure keys and stuff to do deployments.
automated tests happen in testing on a machine that can automatically sync itself to the latest checked in code.
Potentially, all of the above (cept for probably production) could easily be virtual machines.
"(we have about 60000 requests per day currently most of them being cross table queries but finally they should be seven or eight times this number)."
MySQL can very happily handle >800 queries per second on a single machine with good indexes. if you've got complicated data in big datasets, go with cluster or reader/writer replication where you can, and have a read-replica for reporting.
Like others have said - any respectable db should be able to do this without problem.
Re:see also the canonical text on the subject
on
Pro MySQL
·
· Score: 1
I have to throw in my agreement - the High Performance MySQL book is a thing of greatness, which seems to cover all the bases for a professional using MySQL.
One of the better things about the book is it does explain how the underlying engines store and retrieve data, which gives good insight into performance problems and how to tune databases and requests for proper and efficient execution.
(short) $result = $db->query('select * from table where foo=? and bar=?',$data);
(prepare) $stmt = $db->prepare('select * from table where foo=? and bar=?'); $result = $db->execute($stmt,$data);
Works with mysql, pgsql, mssql... etc etc. MDB2 is the new version of this library which uses much the same syntax. Uses database-specific escaping/quoting automatically.
For a growing set of sites with growing needs, I had outgrown shared hosting 'reseller' accounts and got a pretty decent deal on a dedicated server ($100/mo rent-to-own type situation).
First, I ordered a second drive for the machine, which never got installed.
Second, I got double charged for the CPanel installation/subscription.
Third, Oktagone was globally DDoS'd numerous times and it seems that nothing was done about it.
That was while everything was still sort of OK, We were getting a decent deal and generally happy with the service. We were housed in a datacenter in NYC with wich Oktagone had leased space.
Aparently, Oktagone stopped paying their bills to the datacenter. They turned off the routers and switches, and held the machines for ransom. Calling the datacenter directly, we as the actual customers had no say in what was happening with the hardware we actually owned (ownership was actually in dispute, it seemed that Oktagone screwed us on that part too). Oktagone refused to pay the bills, and abandoned the datacenter. No server which we had paid dutifuly for, and for a while, no way to get the data off. Luckily the datacenter did eventually turn on ssh so we could copy the data off and onto new servers.
As of now, it looks as if Oktagone has been eaten by their creditors (as is fitting), and I'm happily using a dedicated server provided by ThePlanet/ServerMatrix.
More Pros: -Just needs ftp or ssh or whatever you can connect with (ftp by default)
More Cons: -uses DD for disk dump - this means if you have lots of empty space on a big disk it'll come over too, and likely, it'll be junk (making it hard to compress an image). the G4U site has some ideas on how to get around that.
Recently I'm using FreeBSD, but my disks are slightly different between a few of the machines, so I'm using dump/restore and the livecd portion of disk1 of freebsd 5.4 (and now 6) to create the partition, ifconfig, then ssh/cat > restore on the new drive (then rename, rebuild host keys, etc). works like a charm.
Am I correct in assuming these lines will replace the v20z and v40z servers, which have very similar specs?
The v[24]0z servers were not manufactured by Sun themselves, and they've mentioned that they're working on their own version.
I love my v20z servers - they are a great alternative to the crap Dell calls 1U servers. I hope these are as good, and maybe a little better supported.
The wikipedia entry also talks about the Open Source potential for OS/2 - Almost exactly what I was about to post - Microsoft likely still owns a good chunk of the code. There are pieces that IBM may be able to make available, but the core of the OS is already owned.
I was hit with the security exploit when the vulnerability was first announced in mid November (The Hilight bug at least). Since then I've upgraded php and phpbb on all my hosted sites (it ended up being resold sites that got me), and done some other things reguarding file rights and access.
The main thing though that I've done that I hope to help me stay a little in front of these types of exploits is implement mod_security and add some rules which block the more 'common' exploits and sql injections.
Does this seem like a reasonable thing to keep doing? I hate to prohibit hosted sites from having any prebuilt scripts like phpbb or phpdig or anything else, but I don't want to be a big target for exploits either.
My Article on EQII.com goes over a few points I would like to discuss further about the MyGuyGames article - I know I may be a little biased towards EQII, but I think I give a fair bit of merit to WoW, which is a great game in its own right.
I'd love to see the same type of article from a WoW enthusiast if there are issues I misstated or the article misstated. I think these types of articles are pretty interesting, even if they're ultimately just excersizes in preaching to the choir... Its easy to get locked into one of these games and think everything else is crap, and I think its only fair to give all the games their due attention.
I just installed the Branch nightly and I must say, LiveLinks rocks. It needs a little more functionality (ability to create a new LiveLink and paste a feed URL would be great) but otherwise is very nice.
For FireFox users, if you're using the Default theme, and see a lightning bolt, thats the livelinks.
Way to follow p-a and not read your own conclusions. sheesh.
watch some of the videos. look at some of the other screenshots released recently on www.eqii.com - yeah, they've had some bad ss picks, but in person, the game looks great.
I'm working on a project for my Senior Project that could take the Bayes method to identify webpages that are 'good' or 'bad' for a proxy or bridge based connection filtering or bandwidth limiting application.
Now, obviously for webpages its a bit easier to say 'good' 'bad', but this app (www.bandwidtharbitrator.com) already has some regular expressions for apps like Kazaa, Bittorrent, in the hopes of limiting the bandwidth. I wonder if a Bayesian system could be adapted to this domain? I considered it, but the person in charge of that part of the project is using a diff-like method (which I find silly).
Are there easy-to-plug-into APIs and libraries like that we could use to do all the 'hard work'? Is SpamBayes up to the task?
At the end of your free 30 days, you will recieve a bill from S-E prorated from the time of the end of your free period to the next billing cycle. Then its business as usual for that.
I payed something like $3.21 for my first bill on FFXI.
yeah... I think they do know the meaning of "alpha" - the state before feature-completeness. Beta is feature complete, but bug-hunting.
Or at least it should be.
Games that have gone into release without testing systems in 'Beta' (*cough*horizons*cough*ac2*cough*) do not know the meaning of the different phases.
A friend of mine interned at the Seagate R&D plant in Longmont, Colorado last year, doing testing runs for harddrive series, all IDE. They had BIG refrigerator looking things that did automatic testing (or actually any other low-level function) based on commands from a terminal.
If I'm not mistaken, they just upgraded their cabinets, so it is likely that either there are surplus cabinets around from the various manufacturers, or theres somewhere making em. They might be a bit expensive, but if you're looking at throughput this might be the key.
Give one of em a ring, find out what they can recommend. I'm sure you can find someone if you just look.
Hey, it sounds like they've at least kept up with Media Center Edition though. That is a piece of crap for a program. The interface is slow, the mp3 player is aparently not compatible with WMP9 (which conveniently comes in a patch of course) so new media cant be loaded into the MCE library, and the remote handler doesnt detect anything but the very commercial remotes (I've got an Ira and it has no idea what to do with it). Bleh. I went back to just programming functions into Girder and using winamp.
Ballistics and Bandits are games that take a unique spin to current games. Ballistics could be called the first supersonic racing game, where your insane speed turns a game of reaction into more of a game of pattern. The developers at E3 2001 were able to complete most of the courses at full supersonic speed, where we could barely make it at subsonic speed.
The game is one of the first to introduce pixel shading and other features of the GeForce 3.
Bandits I know less about, sadly. I've been sitting around waiting for word of Grin's ever-in-development game, Vultures. The guys at Grin were nice enough at E3 2001 to give me their concept-art posters they had posted in their booth (Kentia Hall). From what I remember the game has gone under a couple design revisions, but the detail they paid in the rendered weapons and the concept art I have is amazing. It should be a game I wont be able to miss. That is, if it ever does get released.
So what if these arent Popular games? They use cutting edge technology and were probably easy to get a hold of (Grin is full of nice guys), so maybe the lessons learned from porting these games will enable LGP to publish those games you are clamoring for?
Again I'm taken back to my argument that people shouldnt purely deride games. If they are supported through either fans or at least constructive criticism, the whole industry benifits.
I'd say if the content and workflow management works well, and has lots of publishing-type functionality, there is a big market of small companies or sites looking for easy to use, cheap, extensible solutions. I know I am. We've written a system of our own, but with us not being experienced with the ins and outs of content management, it feels kludged together.
Any online/print magazine or newsletter just starting up would benifit greatly from a slick solution, especially if it was open source.
I have known of many of these problems from my friends experiences playing the game. They convinced me to play for a whole month, and I said forget it guys, this game sucks. Ultima Online was WAY better, and guess what, in a couple months we were all back playing that.
Anyway, is anyone at all convinced that any of this will change with the new games being developed? SWG is close to being released, and I already see some problems...
My big concern is Everquest 2. Some of these issues are big and annoying enough to piss off everyone but the people who are playing EQ right now, and why would they want to lose their Uber characters?
We've had a lot of discussion of this over at eqii.com, and I dont think we've come to a decision whatsoever. It seems to me that for some reason people look past these huge problems to play the game, and I just dont understand it.
My company leases everything through Wells Fargo in Texas. Everything. Dell equipment, Canon Copiers, HP printers, no matter where they come from. We put in a purchase order, they send out the money.
This is what banks are good at:-), and most of the venders like it too.
Aanyway, why not do what a few other sites do... in Apache just reject anything with a referer from slashdot.org domain. redirect it to something like a tripod page that says "your link has been rejected - linked from slashdot" or something.
or heck, just drop the request. Make them mirror it.
This datasheet tells you all about the drive that is in about half of the xbox units out there.
More or less its a great little drive. One platter (they only use one side of the 2gb platter) means less operating parts and longer life, lower power consumption, and lower heat emission. Oh and they are also amazingly quiet too.
now the dvd drive, I bet the same thing happens with dvd players as well, i've seen it starting to happen on one of the players I've had frequent use of (old roommates) - but a cleaning with just the typical cd lens cleaner did the trick, at least for now. eventually a real cleaning may be in order. just like my cd player in my car.
good ole slashdot. 'of course its an xbox problem'.
(just in case that tag doesnt go through, Copy/Paste: http://www.seagate.com/cda/products/discsales/mark eting/detail/0,1081,277,00.html )
Slashdot Mirror
Dev / Testing / Staging-Acceptance-PreProduction / Production
Each developer should get their own virtualhost on the dev server that can mimic production (apache virtualhosts if possible, or use vmware), that they can upload and self-test to. Use wildcard dns for hostnames like username.dev.company.com.
Depending on how many paralell things are tested, you might want more testing systems. Testing boxes should be mirrors of production software-wise (or at least as close as you feel comfortable).
Staging/Preproduction might be optional if you're a small shop or don't need acceptance testing - this should be a more stable place to do system tests outside of the possibly buggy testing environ. again, should mimic production, possibly closer.
production - obvious.
If you can do it, do deployments from SVN via labels or tags (not sure which it uses) like RELEASE_2006-12-27_NOTE (and something like RELEASE_LIVE which moves), so you can track revisions which are on production. Instead of copying code up, you use svn to sync code down, probably through some kickoff script. this occurs to each environment. Take everyone's access away from production, and probably staging - testing you might give select people access. Only the people in charge of the machines themselves, and whoever's in charge of releasing code should have access. Use automation with secure keys and stuff to do deployments.
automated tests happen in testing on a machine that can automatically sync itself to the latest checked in code.
Potentially, all of the above (cept for probably production) could easily be virtual machines.
"(we have about 60000 requests per day currently most of them being cross table queries but finally they should be seven or eight times this number)."
MySQL can very happily handle >800 queries per second on a single machine with good indexes. if you've got complicated data in big datasets, go with cluster or reader/writer replication where you can, and have a read-replica for reporting.
Like others have said - any respectable db should be able to do this without problem.
I have to throw in my agreement - the High Performance MySQL book is a thing of greatness, which seems to cover all the bases for a professional using MySQL.
One of the better things about the book is it does explain how the underlying engines store and retrieve data, which gives good insight into performance problems and how to tune databases and requests for proper and efficient execution.
PEAR::DB supports almost the exact same method.
p
$data = array('one',2);
(short)
$result = $db->query('select * from table where foo=? and bar=?',$data);
(prepare)
$stmt = $db->prepare('select * from table where foo=? and bar=?');
$result = $db->execute($stmt,$data);
Works with mysql, pgsql, mssql... etc etc. MDB2 is the new version of this library which uses much the same syntax. Uses database-specific escaping/quoting automatically.
http://pear.php.net/manual/en/package.database.ph
For a growing set of sites with growing needs, I had outgrown shared hosting 'reseller' accounts and got a pretty decent deal on a dedicated server ($100/mo rent-to-own type situation).
First, I ordered a second drive for the machine, which never got installed.
Second, I got double charged for the CPanel installation/subscription.
Third, Oktagone was globally DDoS'd numerous times and it seems that nothing was done about it.
That was while everything was still sort of OK, We were getting a decent deal and generally happy with the service. We were housed in a datacenter in NYC with wich Oktagone had leased space.
Aparently, Oktagone stopped paying their bills to the datacenter. They turned off the routers and switches, and held the machines for ransom. Calling the datacenter directly, we as the actual customers had no say in what was happening with the hardware we actually owned (ownership was actually in dispute, it seemed that Oktagone screwed us on that part too). Oktagone refused to pay the bills, and abandoned the datacenter. No server which we had paid dutifuly for, and for a while, no way to get the data off. Luckily the datacenter did eventually turn on ssh so we could copy the data off and onto new servers.
As of now, it looks as if Oktagone has been eaten by their creditors (as is fitting), and I'm happily using a dedicated server provided by ThePlanet/ServerMatrix.
More Pros:
-Just needs ftp or ssh or whatever you can connect with (ftp by default)
More Cons:
-uses DD for disk dump - this means if you have lots of empty space on a big disk it'll come over too, and likely, it'll be junk (making it hard to compress an image). the G4U site has some ideas on how to get around that.
Recently I'm using FreeBSD, but my disks are slightly different between a few of the machines, so I'm using dump/restore and the livecd portion of disk1 of freebsd 5.4 (and now 6) to create the partition, ifconfig, then ssh/cat > restore on the new drive (then rename, rebuild host keys, etc). works like a charm.
Am I correct in assuming these lines will replace the v20z and v40z servers, which have very similar specs?
The v[24]0z servers were not manufactured by Sun themselves, and they've mentioned that they're working on their own version.
I love my v20z servers - they are a great alternative to the crap Dell calls 1U servers. I hope these are as good, and maybe a little better supported.
The wikipedia entry also talks about the Open Source potential for OS/2 - Almost exactly what I was about to post - Microsoft likely still owns a good chunk of the code. There are pieces that IBM may be able to make available, but the core of the OS is already owned.
I was hit with the security exploit when the vulnerability was first announced in mid November (The Hilight bug at least). Since then I've upgraded php and phpbb on all my hosted sites (it ended up being resold sites that got me), and done some other things reguarding file rights and access.
The main thing though that I've done that I hope to help me stay a little in front of these types of exploits is implement mod_security and add some rules which block the more 'common' exploits and sql injections.
Does this seem like a reasonable thing to keep doing? I hate to prohibit hosted sites from having any prebuilt scripts like phpbb or phpdig or anything else, but I don't want to be a big target for exploits either.
Is mod_security the 'easy' answer?
I'd love to see the same type of article from a WoW enthusiast if there are issues I misstated or the article misstated. I think these types of articles are pretty interesting, even if they're ultimately just excersizes in preaching to the choir... Its easy to get locked into one of these games and think everything else is crap, and I think its only fair to give all the games their due attention.
I just installed the Branch nightly and I must say, LiveLinks rocks. It needs a little more functionality (ability to create a new LiveLink and paste a feed URL would be great) but otherwise is very nice.
For FireFox users, if you're using the Default theme, and see a lightning bolt, thats the livelinks.
Way to follow p-a and not read your own conclusions. sheesh.
watch some of the videos. look at some of the other screenshots released recently on www.eqii.com - yeah, they've had some bad ss picks, but in person, the game looks great.
They merely toned down a couple of the agressive graphics settings and the game ran smoothly.
The min specs are really nothing too bad... maybe except for the ram requirement. Everything else is pretty much 'standard' by now.
Just dont assume you can buy an MX440 and run the game.
I'm working on a project for my Senior Project that could take the Bayes method to identify webpages that are 'good' or 'bad' for a proxy or bridge based connection filtering or bandwidth limiting application.
Now, obviously for webpages its a bit easier to say 'good' 'bad', but this app (www.bandwidtharbitrator.com) already has some regular expressions for apps like Kazaa, Bittorrent, in the hopes of limiting the bandwidth. I wonder if a Bayesian system could be adapted to this domain? I considered it, but the person in charge of that part of the project is using a diff-like method (which I find silly).
Are there easy-to-plug-into APIs and libraries like that we could use to do all the 'hard work'? Is SpamBayes up to the task?
At the end of your free 30 days, you will recieve a bill from S-E prorated from the time of the end of your free period to the next billing cycle. Then its business as usual for that.
I payed something like $3.21 for my first bill on FFXI.
Freelancer is a great game, you should see if it runs under WineX or something.
yeah... I think they do know the meaning of "alpha" - the state before feature-completeness. Beta is feature complete, but bug-hunting.
Or at least it should be.
Games that have gone into release without testing systems in 'Beta' (*cough*horizons*cough*ac2*cough*) do not know the meaning of the different phases.
A friend of mine interned at the Seagate R&D plant in Longmont, Colorado last year, doing testing runs for harddrive series, all IDE. They had BIG refrigerator looking things that did automatic testing (or actually any other low-level function) based on commands from a terminal.
If I'm not mistaken, they just upgraded their cabinets, so it is likely that either there are surplus cabinets around from the various manufacturers, or theres somewhere making em. They might be a bit expensive, but if you're looking at throughput this might be the key.
Give one of em a ring, find out what they can recommend. I'm sure you can find someone if you just look.
Hey, it sounds like they've at least kept up with Media Center Edition though. That is a piece of crap for a program. The interface is slow, the mp3 player is aparently not compatible with WMP9 (which conveniently comes in a patch of course) so new media cant be loaded into the MCE library, and the remote handler doesnt detect anything but the very commercial remotes (I've got an Ira and it has no idea what to do with it). Bleh. I went back to just programming functions into Girder and using winamp.
Ballistics and Bandits are games that take a unique spin to current games. Ballistics could be called the first supersonic racing game, where your insane speed turns a game of reaction into more of a game of pattern. The developers at E3 2001 were able to complete most of the courses at full supersonic speed, where we could barely make it at subsonic speed.
The game is one of the first to introduce pixel shading and other features of the GeForce 3.
Bandits I know less about, sadly. I've been sitting around waiting for word of Grin's ever-in-development game, Vultures. The guys at Grin were nice enough at E3 2001 to give me their concept-art posters they had posted in their booth (Kentia Hall). From what I remember the game has gone under a couple design revisions, but the detail they paid in the rendered weapons and the concept art I have is amazing. It should be a game I wont be able to miss. That is, if it ever does get released.
So what if these arent Popular games? They use cutting edge technology and were probably easy to get a hold of (Grin is full of nice guys), so maybe the lessons learned from porting these games will enable LGP to publish those games you are clamoring for?
Again I'm taken back to my argument that people shouldnt purely deride games. If they are supported through either fans or at least constructive criticism, the whole industry benifits.
I'd say if the content and workflow management works well, and has lots of publishing-type functionality, there is a big market of small companies or sites looking for easy to use, cheap, extensible solutions. I know I am. We've written a system of our own, but with us not being experienced with the ins and outs of content management, it feels kludged together.
Any online/print magazine or newsletter just starting up would benifit greatly from a slick solution, especially if it was open source.
Anyway, is anyone at all convinced that any of this will change with the new games being developed? SWG is close to being released, and I already see some problems...
My big concern is Everquest 2. Some of these issues are big and annoying enough to piss off everyone but the people who are playing EQ right now, and why would they want to lose their Uber characters?
We've had a lot of discussion of this over at eqii.com, and I dont think we've come to a decision whatsoever. It seems to me that for some reason people look past these huge problems to play the game, and I just dont understand it.
My company leases everything through Wells Fargo in Texas. Everything. Dell equipment, Canon Copiers, HP printers, no matter where they come from. We put in a purchase order, they send out the money.
:-), and most of the venders like it too.
This is what banks are good at
...moderated funny... gah.
Aanyway, why not do what a few other sites do... in Apache just reject anything with a referer from slashdot.org domain. redirect it to something like a tripod page that says "your link has been rejected - linked from slashdot" or something.
or heck, just drop the request. Make them mirror it.
More or less its a great little drive. One platter (they only use one side of the 2gb platter) means less operating parts and longer life, lower power consumption, and lower heat emission. Oh and they are also amazingly quiet too.
now the dvd drive, I bet the same thing happens with dvd players as well, i've seen it starting to happen on one of the players I've had frequent use of (old roommates) - but a cleaning with just the typical cd lens cleaner did the trick, at least for now. eventually a real cleaning may be in order. just like my cd player in my car.
good ole slashdot. 'of course its an xbox problem'. (just in case that tag doesnt go through, Copy/Paste: http://www.seagate.com/cda/products/discsales/mark eting/detail/0,1081,277,00.html )