Good recommendation on the online UPS side - nice, clean power and no switching inconsistency. Might go cheaper on the slave database servers but the master should be very well protected.
With a UPS, we will have things configured to be very conservative, shutting down the servers after no more than a minute or so on UPS power. Much nicer to be cleanly shut down than not.
We are looking to have 1-4 remote sites this year, with database slaves we can switch to being masters in an emergency, so single location failures shouldn't be able to be so disruptive. But not there yet. This year has been "only" going from 3-4 servers to 50, all with donations buying the equipment and everyone involved donating their time. We're in pretty good shape overall (and wonderful shape compared to this time last year!) but not yet where we need to be given the expectations of continuous reliability for the busiest internet sites. Gradually improving that.
Suspect right now is the same thing LiveJournal found: disk systems lying about having committed writes. LJ has developed a testing tool for that, which has the writing system tell a networked remote system what the disk system says it's written, then compares that to the state after the power off and on to find out if it lied. The system which did a full recovery doesn't have a fancy caching disk controller, just Linux RAID 0. Which worked nicely.
Easily. See what those saying that MySQL can't do what MySQL does are promoting.:)
LiveJournal found that it had some disk systems which lied about having committed writes. The have a preliminary tool which copies what it's writing to disk to a networked system and then compares the after power off and recovery state to what the disk system said it could do. Are going to make it available to the community as time allows.
I expect we're going to find the same at Wikipedia. Here's a pretty typical error log, this one from the server which was master database server:
050222 5:11:12 InnoDB: Database was not shut down normally. InnoDB: Starting recovery from log files... InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 303 1283776146 InnoDB: Doing recovery: scanned up to log sequence number 303 1289018880 InnoDB: Doing recovery: scanned up to log sequence number 303 1294261760 InnoDB: Doing recovery: scanned up to log sequence number 303 1299504640 InnoDB: Doing recovery: scanned up to log sequence number 303 1304747520 InnoDB: Doing recovery: scanned up to log sequence number 303 1309990400 InnoDB: Doing recovery: scanned up to log sequence number 303 1315233280 InnoDB: Doing recovery: scanned up to log sequence number 303 1320476160 InnoDB: Doing recovery: scanned up to log sequence number 303 1325719040 InnoDB: Doing recovery: scanned up to log sequence number 303 1330961920 InnoDB: Doing recovery: scanned up to log sequence number 303 1336204800 InnoDB: Doing recovery: scanned up to log sequence number 303 1341447680 InnoDB: Doing recovery: scanned up to log sequence number 303 1346690560 InnoDB: Doing recovery: scanned up to log sequence number 303 1347688389 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 14 row operations to undo InnoDB: Trx id counter is 1 935480064 050222 5:11:13 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 8617985. InnoDB: You may have to recover from a backup. 050222 5:12:20 InnoDB: Page dump in ascii and hex (16384 bytes):
Observe that the database engine went back to its last checkpoint, noticed the partial transaction and undid it and was rolling ahead in the write-ahead log when it encountered a database page which failed its checksum test. That failed checksum test is why I think it's a problem with the disk system lying about what was written. You can get that when a database page spans two drives in a stripe set and one has committed the update while the other hasn't.
In more typical situations MySQL simply applies the updates and all is well. I've had a server set up to exceed RAM with swap turned off and get killed every ten minutes for hours and recover every time.
Just to be complete:
The database servers have dual redundant power supplies. TWO breakers at the colo tripped, taking out both.
The systems are a mix of SCSI and SATA, so no point in arguing about one being lousy. SATA and Linux win if you want a winner: it was a SATA box using Linux RAID 0 whoch completed full recovery. It wasn't one of the normal servers - it was used for backup and offline report generation.
Two different disk controller makers, one each for SCSI and SATA.
Battery backed up write cache on most of the main server disk controllers but the one without the battery backup for the write cache had the same problem (which shouldn't surprise anyone - that one should be expected not to recover well).
After LJ's experience we were after UPS systems in the racks but hadn't yet checked whether the local fire code allows them. Some don't, for el
Yes, so does the InnoDB engine in MySQL. Doesn't help so much when the drive system has lied about which page writes have been committed to disk and you have a RAID system where a page is spanning a couple of drives and one wrote its update to the page while the other didn't, which is what I speculate happened. That leaves you with a database page which fails its checksumming. Can recover from individual bad pages but not worth doing it when there's a complete copy available. Since we don't need to recover them, we aren't. Copying from one with no apparent damage instead.
Did, of course, make a copy of the databases before trying to restart some ofthem, so we'd have that recovery option if it turns out that we need it.
The best copy we have is on a lowly pair of 250GB SATA drives using Linux RAID 0 and since thats the best it's the one we used.
Every main database server had corrupt database pages. That is, 3 systems with battery backed up write caching controlles and SCSI drives and 2 SATA systems with write caching SATA controllers, one battery backed up the other not, two different SATA disk drive makers.
Involved:
Two completely different caching controller brands
Two different SATA drive makers
Seagate only on the SCSI drive maker side
Obvious speculation involves the controllers not telling the drives not to write buffer or the drives not listening. No point in getting into SCSI or SATA or this disc controller or that controller fights when there's this much variation involved.
Yes. I wrote that cached page and it's now a bit out of date. IF, and it's not certain, local fire regulations permit the use of UPS systems in the racks we're going to be installing them. Decided on that after LiveJournal's unfortunate experience. But don't yet have them.
The database servers have dual redundant supplies and the colo tells us that TWO circuit breakers tripped. Fun. Not. Do try to avoid having the same happen to you - losing both circuits isn't fun.
Since at least one of our MySQL database servers has so far restarted successfully with all InnoDB data intact, perhaps you'd care to reconsider your assessment that MySQL is incapable of doing what it just did?
For the rest, we'll see as we get to them and, for any that fail, then look to see whether it was the disk controller or the disk drive lying about having the data written to battery backed up RAM or the disk surface.
Wikipedia hasn't suffered a day of downtime yet for this reason and looks to be down for no more than a few hours this time. A previous incident lasting more than a day was a human or three screwing up and having two copies of the server software writing to the same database files without any locking to prevent conflicting updates. The result of that shouldn't surprise anyone.
So far one of our database servers has completed a successful recovery (we're working through them all). On a gigabit link it takes something between 90 minutes and 4 hours to rsync from one to another. As soon as we have two database servers working, we'll be restoring service in read only mode. Likely to be that 90 minutes to 4 hours from now as worst case.
I'll post followups to this post later, as we're closer to being fully recovered.
Consider the Linux kernel. That uses headers with substantially the same functionality as those of Unix. It's conceivable that you can build under Linux a program using those headers which will function correctly on a Unix system. Does that make it a derived work of Unix?
The question here is the difference between linking and compatibility. Linux can use those Unix values without infringing the copyright of Unix. Otherwise Linux is greatly threatened by the SCO action. Similarly, it's conceivable to produce a Linux-compatible program which isn't a derivative work of Linux.
Whether you're doing so depends on the details of what is happening. If it's just the facts of an interface, it's going to be tough to argue that it's linked when that's contrary to the specific laws regarding reverse engineering for compatibility being legal. Someone may try anyway, of course.
Very great care is needed here, lest you end up using arguments which in effect make every Windows program and the WINE project derivative works of Windows because they are compatible with it or programs which work with it.
If there's nothing copyrightable the combined work is still (for copyright purposes) the same work as it was before, so there's no basis for an infringment complaint. A typo correction in a large body of text is one example of such a change which is ineligible for copyright coverage. Some bug fixes may also qualify, depending on significance to the work as a whole.
A public domain (rather than ineligible for copyright) change would be more interesting but that's completely free, even more so than the license-restricted GPL, so it's not very useful to take any legal action. What copyright law basis would you use for arguing that a linked component dedicated to the public domain was a license infringment? Take care that you don't end up accidentally arguing that you can't use public domain pieces in GPL programs.
The lawyers would also need to consider whether the change was either de minimis or fair use.
I know I wouldn't want to argue that a bug fix or security flaw closing modification wasn't fair use.:)
We aren't planning to shut down the Florida center and that's where the master copy of the data is kept and where we have some 50 servers already. You're going to be hearing about a fund-raising drive soon as well. Yes, the Google deal is very nice but mixed revenuse and hosting is the key to independence and doing our job properly. And that's exactly what we're doing. We're looking for a mix of donations from the general public, donations from larger charities (like the $40,000 grant we recently received from the Lounsbery Foundation - thanks!) and donated equipment and resources.
I addressed some of the other issues in an earlier post. Recommended reading if you want to know how the guy doing much of the capacity planning is thinking.
Google is doing a good thing. Still, being concerned is natural. We're well aware of the need to be independent and are looking out for ourselves and those who rely on us. The best way to help? Easy enough: if you've skills or equipment we need we need, offer those. If not, listen for us requesting money and respond when we ask, to whatever extent makes sense for you.
You're both a bit right. Here's a highlight view of some of the things happening during that week:
New squid cache servers in Paris. After network bandwidth issues there were resolved they speeded up access in bits of Europe. But they also slowed down all page saves because saves also tell the Squids to remove (flush) related pages from their cache and those more distant servers took longer to flush. Can be tens of thousands of flushes to do. Squid flushing/purging is now much faster and longer term it's being modified to be taken completely out of the save loop. This one really hurt save speed for a while, as the developers sorted out what was happening and improved the way purging was done. Net result: all purging of squids is much more efficient and saves remain faster than they used to be. The way pages are delivered to those who aren't logged in was also improved, so style sheets are served from the Squids for them now - that makes page views for those not logged in less sensitive to Apache web server load.
Load balancing pain. Load balancing is what chooses which Apache web server gets the next request. The previous system wasn't very even so many requests were getting sent to the most heavily loaded Apaches when they should have been sent to a less loaded one instead. It's been a long-standing problem for us. During the week or so you're talking about we were testing several different replacement load balancing systems to find those which would give a good result:
Pen seemed better than running modified Squids on the Apaches but wasn't good enough.
Perlbal from the Livejournal people worked very well and gave a nicely even load balance. Brad and Mark from LJ were very helpful in getting it described and set up. We took two Apache web servers to use for this in case they used too much CPU. As it turned out they used only about 10% on each machine. But that left us two Apaches not building pages...
Our Squid expert wrote a replacement ICP client to run on the Apaches. That also produced an even load balance so around the end of the week we switched to using it. Freed up those two Apaches to go back to page building duty. So, until we need its other features, Perlbal isn't in use - not quite the best solution for us today (but may be in the future).
So, we left that week with much improved load balancing for the Apaches. Much more consistent page load times now.
Bugs in MediaWiki 1.4 beta left the Apaches filling their available child slots. That combined with some specific web crawlers could sometimes let the crawlers take the site down by leaving no or very few free children to handle requests. Also increased Apache load a bit. The most important ones have been fixed but there's still an occasional stuck child. To deal with that we have a script restarting one Apache web server every 5 or so minutes, to ensure that it can't rise to a troublesome level.
The crawler/Apache child problem and a too high setting for maximum children would let the Apache server on some Memcached machines take so much RAM that the system swapped. Very bad news because that caused Memcached to respond very slowly - far too slowly, so all Apaches filled all child slots and the site appeared dead. That's been dealt with now. Not a Memcached issue as such - just the usual don't let the box swap to death situation. While tracking this down assorted other Memcached-related things were improved.
As a temporary workaround, the two Memcached machines we were using at that time had Apache stopped on them. That left us four Apaches short for a few days. That took us beyond the critical Apache CPU shortage point and Apache load and wait times rose significantly. Better than a dead site but not at all good. Response time drop with loss of machines isn't linear beyond a certain point and four more Apaches out of service took us beyond that point.
Also during that week the handling of database updates was substantially improved, so lock waits are mostly gone, wh
It's a vital open source/ free culture copyright issue as well. Consider the Wikipedia encyclopedia. It's largely under the GFDL. Anyone can redistribute it. Anyone should also presumably be able to say that it is what they are distributing, since redistribution is a fundamental purpose of the GFDL.
Yet there's also an interest in people not being able to claim that they are wikipedia.org, the site where the work is being built.
A trademark on "wikipedia" might seem like one way to proceed, in part because that covers all of the other domains in various countries, until you notice that trademark law may then restrict of others from distributing the title page part of the work, "Wikipedia, the free encyclopedia".
Now, it seems fairly clear that the Wikimedia Foundation would inevitably have to be granting a license to use the trademark name because that's required for others to distribute the work, something the Foundation is required to do by the GFDL license the Foundation is granted by the authors of the encyclopedia.
The United States Supreme Court has touched this general area of the interaction between copyright and trademark law in its Dastar v. Twentieth Century Fox Film Corp decision. That held that a trademark couldn't be used to prevent the publc domain right to use a work. That's correctly taken as a major victory for the public domain but very similar logic also seems to apply when it comes to trademarks for open source works, particularly when, as in the Wikipedia case, the prospective trademark owner isn't the author of the work but simply a (albeit very important!) licensee.
Still, it's uncomfortably messy and it would be nice if a future GFDL version clarified that trademark law may not be used to prohibit distribution of the correctly named work either, as it does now for technnical means.
This is an area which the Creative Commons ShareAlike License seems to better resolve in the wording of its license grant, though it's still not as explicit as it might usefully be.
The views in this post are mine alone. No part of this post should be taken as reflecting the views of the Wikimedia Foundation or its Board - indeed, it's possible that the Foundation or one or more of its Board members might disagree, since it's not something I've discussed with the whole board.
MySQL ships with the InnoDB database engine as a standard part. InnoDB will write to its log with every completed transaction, flushing that to the hardware. If you lose power, it'll reapply the transactions from that log automatically, without hassle, when the database server is restarted. The log writes are sequential so you can get both excellent write caching (into dirty database pages in RAM) and transaction durability (via those immediate flushes of the log to disk). InnoDB is also the database engine which offers transaction support, so if you want the normal transaction guarantees like durability, you're going to be using it anyway. This isn't a theoretical capability - I've had production Wikipedia servers go down with tens of thousaands of dirty database pages in RAM and come back up with all data, while I was asleep.
Alternatively, though it's still in alpha, there's the choice to use the Cluster engine to spread the load over many redundant servers via mirroring in the database storage engine. Very interesting technology for me, though still too early for Wikipedia to use.
For those with high read load or worries about single machine failure, replication is also standard and it's easy to set up a web server or two to do replication in the background to get the last bits of data saved before a disaster hit the master. On the Wikipedia technical to do list is a replicating slave in Paris, so the building falling down on the main site will still leave even very recent updates available.
The combination of InnoDB and replication makes it easy for me to sleep well. The one thing it doesn't yet do as a standard part of the package to complete the picture is automatic failover to the best slave if the master dies and stays dead. I think the failover bit is on the to do list for the 5.n series. Cluster does, though - one of the reasons why I'm watching it with interest.
None of this means PostreSQL is bad. Just different.
In this case, it's more a question of maximising use of limited resources. Timeouts and saving problems at present are generally because of overloaded Apache web servers. A batch of 10 more (taking us over 50 machines total) was delivered a few days ago and will be entering service soon. On the database side I'm putting together specifications for our next batch of machines.
With enterprise considerations, putting a database-driven top 100 site together with equipment costing in the $100,000 range is probably contraindicated.:) While the equipment, tools and people are capable, that does produce some load stresses. Fortunately, we can deal with those, though exponential growth with doubling times in the 8-16 week range when not performance-limited are always a challenge. If money was being lost due to response time or problems, the economics would shift dramatically, in favor of larger budgets. In our case, we might even get increased revenue from modest problem levels, though not from significant ones, since that cuts growth.
For web applications the enterprise trend today seems to be to move away from big iron when it comes to delivering services directly to end users, even if the big iron remains present in the background. Very much a case of picking the best tool for each job, in my opinion. That could easily be a mixed environment.
I do wonder what Microsoft, Oracle or other major corporate players would charge a corporation for a license for a 6 dual CPU server license with Wikimedia query and traffic levels. Anyone know?
Wikipedia uses MySQL's InnoDB storage engine, with row level locking and lots of concurrent inserts and selects. It uses the standard MySQL replication and full text search as well. One database server took the site to the top 1500 or so worldwide. Now, with more than half of the load for search, 5 handle the online load for around 200 million queries and about 1.2 million inserts or updates per day for what Alexa currently ranks as 81st most popular English language site. Before some recent code improvements increased slave server use we had one server doing sustained rates in excess of 3,000 queries per second.
We'll soon have 40 web servers hitting those database boxes, so we'll take all the speed we can get.:)
Simply, MySQL gets the job done and has proved sufficient scalability and reliability in practice. PostgreSQL is also clearly an excellent database server. Each has pros and cons. Pick whichever fits the job at hand.
Memcached, Squid and PHP are also key parts of the scalability picture for Wikipedia.
No plans for Google ads. We do send search to either Google or Yahoo now when the database servers are under an uncomfortable amount of load. I'm usually the person making that decision and I decide solely based on balancing server load and reduced service.
Instead of static caches we're using Squid caches which get updated automatically when the content changes. We're looking to place more of these in other places which use significant amounts of bandwidth or are far from Florida in response time terms.
We're aware of the risk of excessive dependence on one donor and are looking to avoid it. We're entirely happy to talk with other companies who want to share in being seen to be helping something obviously good, limited only by the suitability of the offers for our needs. I don't know what the Wikimedia Foundation board would say but personally I'm entirely happy to accept hosting from Yahoo, Microsoft, AOL or any other significant player. In my personal opinion, neutral means just that - including neutral toward all companies in the business.
If any competitor is concerned about Google getting undue benefit or prominence, the solution is simple enough: talk with us as well.
We have offers of various sorts in Europe (though more, particularly from major carriers with excellent peering arrangements, would be very welcome) and the US. We don't yet have a substantial offer in Asia and that's a major hole I'd personally love to see filled. We're very popular in Japan and a location with good peering in Japan would be good for service there.
By this time next year I'd like to see 2-6 major remote sites with database slaves and apache web servers, capable of taking over as master if there's a failure in Florida, plus 10-20+ remote Squid caching sites. A massive amount of work (and donations) required to get that done.
We've already been blocked from China ourselves on several occasions. I've little doubt that it'll happen again and in other places as well from time to time.
It's absolutely certain that we have some unknown, uncorrected copyright infringement, offensive content, politically incorrect items for various parts of the world and assorted other things some or many people find undesirable. If the chairman of the board or president of the country is making headlines worldwide for some indiscretion, expect it to be in the article. Nobody who is unduly concerned about such things should consider offering hosting - we can't guarantee the absence of such content, just that we will try to be neutral.
We're not only interested in hosting and bandwidth. I'm particularly interested in high performance disk drives or systems, high capacity RAM modules (database servers like RAM but 32GB of ECC costs $11,000...) or whole high power database servers. To give some idea, I'm thinking in terms of three quad Opterons with 32GB of RAM and 12-16 15,000 RPM SCSI drives to keep up with demand for just the English language encyclopedia project over the next 6-9 months.
No part of this post should be taken as representing the official views of the Wikimedia Foundation or any members of its board. It is, of course, blatant soliciting for donations, as you'd expect from the guy who does much of the capacity planning...:)
Yes, there are software patents which are truly novel and represent a truly significant invention.
Then there are things like this one, which are so obvious that only an incompetent or inexperienced professional in the field wouldn't consider them the first time they were asked to make a shorter URL containing latitude and longitude without using server-side information.
Now, I do have some background in this area, for I've personal experience with latitude and longitude converted to integers for smaller storage (in a leading distance calculation tool for ship owners). And I've personal experience with changing the base to make smaller representations (in the parts of file names after the period, so more can fit in three characters, as part of a payroll application for ship owners). The former is routine computer science. The latter used standard C language libary function, which represents an integer in any specified base. Neither of those represented any brilliant insight. They were simple professional competence, not requiring any great degree of skill.
Regrettably, patenting the obvious and basic tools is what is happening today and in my opinion the harm from patenting the obvious approaches and tools is far greater than the possible benefits from the truly innovative things. Better not to have the software patents and to use trade secrets, copyright and simple business efficiency benefits to reward the innovation.
With apologies for the effect on your income and a note that I've rarely encountered a lawyer I didn't consider friendly and helpful: progress in the field would best be advanced by abolishing your specialty.
It should work in exactly the manner you've said is wrong. If Ford pretends to be selling Toyota cars, that would be a passing off problem but that's not what is happening here. The ads are correctly describing what is being sold and the trademark is used solely to ensure that the ads appear in their correct position, adjacent to the intended search results. Not replacing them, but adjacent to them.
Google is acting in the manner most consistent with the interests of the consumer.
I suggest that Louis Vuitton deal with the fact that yes, they do have competitors who do buy ad space adjacent to their products. Trying to insist that competitors products not be placed adjacent to theirs is not in the interests of consumers. It's Louis Vuitton which is acting improperly and should be fined in this case.
Meanwhile, I'll remember that Louis Vuitton apparently has so little confidence in their products that they need to try approaches I consider to be unethical to try to stop consumers from finding out about the alternatives in the marketplace.
Please note that Slashdot displays the destination of links adjacent to them, so no reader of this post can possibly be confused about where any of the links I have used goes. And yes, this post is intended to be amusing as well as make a serious point about competition, lack of consumer confusion and ethics.
Some here may not appreciate MySQL. On the other hand, for one of the top 100 English language sites and one of the top 200 in all languages, according to Alaexa.com, I see around 200 million queries per day on MySQL,combined with some 250,000 edits/posts (many updates per edit) per week. For this application. MySQL definitely gets the job done. Each has strengths and weaknesses, though - and each is also stronger and weaker than other database products as well.
What's original? People have been sending email to mobiles via relays (commonly called gateways) since 1987 or earlier. Want to send an email to a ship? On the CompuServe system (in this 1992 newsgroup post you'd send it to TLX:(telex number) and the relay would send the email to the telex machine on a ship (pretty mobile!). The mobile could also originate an emailt o go back to you. I used to tell people my "telex" address which would really email me through that system. That was predated by the Fidonet system, as described in this 1997 newsgroup post.
Slashdot Mirror
Good recommendation on the online UPS side - nice, clean power and no switching inconsistency. Might go cheaper on the slave database servers but the master should be very well protected.
With a UPS, we will have things configured to be very conservative, shutting down the servers after no more than a minute or so on UPS power. Much nicer to be cleanly shut down than not.
We are looking to have 1-4 remote sites this year, with database slaves we can switch to being masters in an emergency, so single location failures shouldn't be able to be so disruptive. But not there yet. This year has been "only" going from 3-4 servers to 50, all with donations buying the equipment and everyone involved donating their time. We're in pretty good shape overall (and wonderful shape compared to this time last year!) but not yet where we need to be given the expectations of continuous reliability for the busiest internet sites. Gradually improving that.
Gory details will be provided.
Suspect right now is the same thing LiveJournal found: disk systems lying about having committed writes. LJ has developed a testing tool for that, which has the writing system tell a networked remote system what the disk system says it's written, then compares that to the state after the power off and on to find out if it lied. The system which did a full recovery doesn't have a fancy caching disk controller, just Linux RAID 0. Which worked nicely.
Easily. See what those saying that MySQL can't do what MySQL does are promoting.:)
LiveJournal found that it had some disk systems which lied about having committed writes. The have a preliminary tool which copies what it's writing to disk to a networked system and then compares the after power off and recovery state to what the disk system said it could do. Are going to make it available to the community as time allows.
I expect we're going to find the same at Wikipedia. Here's a pretty typical error log, this one from the server which was master database server:
050222 5:11:12 InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 303 1283776146
InnoDB: Doing recovery: scanned up to log sequence number 303 1289018880
InnoDB: Doing recovery: scanned up to log sequence number 303 1294261760
InnoDB: Doing recovery: scanned up to log sequence number 303 1299504640
InnoDB: Doing recovery: scanned up to log sequence number 303 1304747520
InnoDB: Doing recovery: scanned up to log sequence number 303 1309990400
InnoDB: Doing recovery: scanned up to log sequence number 303 1315233280
InnoDB: Doing recovery: scanned up to log sequence number 303 1320476160
InnoDB: Doing recovery: scanned up to log sequence number 303 1325719040
InnoDB: Doing recovery: scanned up to log sequence number 303 1330961920
InnoDB: Doing recovery: scanned up to log sequence number 303 1336204800
InnoDB: Doing recovery: scanned up to log sequence number 303 1341447680
InnoDB: Doing recovery: scanned up to log sequence number 303 1346690560
InnoDB: Doing recovery: scanned up to log sequence number 303 1347688389
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 14 row operations to undo
InnoDB: Trx id counter is 1 935480064
050222 5:11:13 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 8617985.
InnoDB: You may have to recover from a backup.
050222 5:12:20 InnoDB: Page dump in ascii and hex (16384 bytes):
Observe that the database engine went back to its last checkpoint, noticed the partial transaction and undid it and was rolling ahead in the write-ahead log when it encountered a database page which failed its checksum test. That failed checksum test is why I think it's a problem with the disk system lying about what was written. You can get that when a database page spans two drives in a stripe set and one has committed the update while the other hasn't.
In more typical situations MySQL simply applies the updates and all is well. I've had a server set up to exceed RAM with swap turned off and get killed every ten minutes for hours and recover every time.
Just to be complete:
Yes, so does the InnoDB engine in MySQL. Doesn't help so much when the drive system has lied about which page writes have been committed to disk and you have a RAID system where a page is spanning a couple of drives and one wrote its update to the page while the other didn't, which is what I speculate happened. That leaves you with a database page which fails its checksumming. Can recover from individual bad pages but not worth doing it when there's a complete copy available. Since we don't need to recover them, we aren't. Copying from one with no apparent damage instead.
Did, of course, make a copy of the databases before trying to restart some ofthem, so we'd have that recovery option if it turns out that we need it.
Every main database server had corrupt database pages. That is, 3 systems with battery backed up write caching controlles and SCSI drives and 2 SATA systems with write caching SATA controllers, one battery backed up the other not, two different SATA disk drive makers.
Involved:
Obvious speculation involves the controllers not telling the drives not to write buffer or the drives not listening. No point in getting into SCSI or SATA or this disc controller or that controller fights when there's this much variation involved.
May be longer so I withdraw that time estimate.
Yes. I wrote that cached page and it's now a bit out of date. IF, and it's not certain, local fire regulations permit the use of UPS systems in the racks we're going to be installing them. Decided on that after LiveJournal's unfortunate experience. But don't yet have them.
Yes. It's in our plans regardless of what happens with Google.
The database servers have dual redundant supplies and the colo tells us that TWO circuit breakers tripped. Fun. Not. Do try to avoid having the same happen to you - losing both circuits isn't fun.
Since at least one of our MySQL database servers has so far restarted successfully with all InnoDB data intact, perhaps you'd care to reconsider your assessment that MySQL is incapable of doing what it just did?
For the rest, we'll see as we get to them and, for any that fail, then look to see whether it was the disk controller or the disk drive lying about having the data written to battery backed up RAM or the disk surface.
Wikipedia hasn't suffered a day of downtime yet for this reason and looks to be down for no more than a few hours this time. A previous incident lasting more than a day was a human or three screwing up and having two copies of the server software writing to the same database files without any locking to prevent conflicting updates. The result of that shouldn't surprise anyone.
So far one of our database servers has completed a successful recovery (we're working through them all). On a gigabit link it takes something between 90 minutes and 4 hours to rsync from one to another. As soon as we have two database servers working, we'll be restoring service in read only mode. Likely to be that 90 minutes to 4 hours from now as worst case.
I'll post followups to this post later, as we're closer to being fully recovered.
Consider the Linux kernel. That uses headers with substantially the same functionality as those of Unix. It's conceivable that you can build under Linux a program using those headers which will function correctly on a Unix system. Does that make it a derived work of Unix?
The question here is the difference between linking and compatibility. Linux can use those Unix values without infringing the copyright of Unix. Otherwise Linux is greatly threatened by the SCO action. Similarly, it's conceivable to produce a Linux-compatible program which isn't a derivative work of Linux.
Whether you're doing so depends on the details of what is happening. If it's just the facts of an interface, it's going to be tough to argue that it's linked when that's contrary to the specific laws regarding reverse engineering for compatibility being legal. Someone may try anyway, of course.
Very great care is needed here, lest you end up using arguments which in effect make every Windows program and the WINE project derivative works of Windows because they are compatible with it or programs which work with it.
If there's nothing copyrightable the combined work is still (for copyright purposes) the same work as it was before, so there's no basis for an infringment complaint. A typo correction in a large body of text is one example of such a change which is ineligible for copyright coverage. Some bug fixes may also qualify, depending on significance to the work as a whole.
A public domain (rather than ineligible for copyright) change would be more interesting but that's completely free, even more so than the license-restricted GPL, so it's not very useful to take any legal action. What copyright law basis would you use for arguing that a linked component dedicated to the public domain was a license infringment? Take care that you don't end up accidentally arguing that you can't use public domain pieces in GPL programs.
The lawyers would also need to consider whether the change was either de minimis or fair use.
I know I wouldn't want to argue that a bug fix or security flaw closing modification wasn't fair use.:)
I addressed some of the other issues in an earlier post. Recommended reading if you want to know how the guy doing much of the capacity planning is thinking.
Google is doing a good thing. Still, being concerned is natural. We're well aware of the need to be independent and are looking out for ourselves and those who rely on us. The best way to help? Easy enough: if you've skills or equipment we need we need, offer those. If not, listen for us requesting money and respond when we ask, to whatever extent makes sense for you.
So, we left that week with much improved load balancing for the Apaches. Much more consistent page load times now.
Yet there's also an interest in people not being able to claim that they are wikipedia.org, the site where the work is being built.
A trademark on "wikipedia" might seem like one way to proceed, in part because that covers all of the other domains in various countries, until you notice that trademark law may then restrict of others from distributing the title page part of the work, "Wikipedia, the free encyclopedia".
Now, it seems fairly clear that the Wikimedia Foundation would inevitably have to be granting a license to use the trademark name because that's required for others to distribute the work, something the Foundation is required to do by the GFDL license the Foundation is granted by the authors of the encyclopedia.
The United States Supreme Court has touched this general area of the interaction between copyright and trademark law in its Dastar v. Twentieth Century Fox Film Corp decision. That held that a trademark couldn't be used to prevent the publc domain right to use a work. That's correctly taken as a major victory for the public domain but very similar logic also seems to apply when it comes to trademarks for open source works, particularly when, as in the Wikipedia case, the prospective trademark owner isn't the author of the work but simply a (albeit very important!) licensee.
Still, it's uncomfortably messy and it would be nice if a future GFDL version clarified that trademark law may not be used to prohibit distribution of the correctly named work either, as it does now for technnical means.
This is an area which the Creative Commons ShareAlike License seems to better resolve in the wording of its license grant, though it's still not as explicit as it might usefully be.
The views in this post are mine alone. No part of this post should be taken as reflecting the views of the Wikimedia Foundation or its Board - indeed, it's possible that the Foundation or one or more of its Board members might disagree, since it's not something I've discussed with the whole board.
MySQL ships with the InnoDB database engine as a standard part. InnoDB will write to its log with every completed transaction, flushing that to the hardware. If you lose power, it'll reapply the transactions from that log automatically, without hassle, when the database server is restarted. The log writes are sequential so you can get both excellent write caching (into dirty database pages in RAM) and transaction durability (via those immediate flushes of the log to disk). InnoDB is also the database engine which offers transaction support, so if you want the normal transaction guarantees like durability, you're going to be using it anyway. This isn't a theoretical capability - I've had production Wikipedia servers go down with tens of thousaands of dirty database pages in RAM and come back up with all data, while I was asleep.
Alternatively, though it's still in alpha, there's the choice to use the Cluster engine to spread the load over many redundant servers via mirroring in the database storage engine. Very interesting technology for me, though still too early for Wikipedia to use.
For those with high read load or worries about single machine failure, replication is also standard and it's easy to set up a web server or two to do replication in the background to get the last bits of data saved before a disaster hit the master. On the Wikipedia technical to do list is a replicating slave in Paris, so the building falling down on the main site will still leave even very recent updates available.
The combination of InnoDB and replication makes it easy for me to sleep well. The one thing it doesn't yet do as a standard part of the package to complete the picture is automatic failover to the best slave if the master dies and stays dead. I think the failover bit is on the to do list for the 5.n series. Cluster does, though - one of the reasons why I'm watching it with interest.
None of this means PostreSQL is bad. Just different.
In this case, it's more a question of maximising use of limited resources. Timeouts and saving problems at present are generally because of overloaded Apache web servers. A batch of 10 more (taking us over 50 machines total) was delivered a few days ago and will be entering service soon. On the database side I'm putting together specifications for our next batch of machines.
With enterprise considerations, putting a database-driven top 100 site together with equipment costing in the $100,000 range is probably contraindicated.:) While the equipment, tools and people are capable, that does produce some load stresses. Fortunately, we can deal with those, though exponential growth with doubling times in the 8-16 week range when not performance-limited are always a challenge. If money was being lost due to response time or problems, the economics would shift dramatically, in favor of larger budgets. In our case, we might even get increased revenue from modest problem levels, though not from significant ones, since that cuts growth.
For web applications the enterprise trend today seems to be to move away from big iron when it comes to delivering services directly to end users, even if the big iron remains present in the background. Very much a case of picking the best tool for each job, in my opinion. That could easily be a mixed environment.
I do wonder what Microsoft, Oracle or other major corporate players would charge a corporation for a license for a 6 dual CPU server license with Wikimedia query and traffic levels. Anyone know?
Wikipedia uses MySQL's InnoDB storage engine, with row level locking and lots of concurrent inserts and selects. It uses the standard MySQL replication and full text search as well. One database server took the site to the top 1500 or so worldwide. Now, with more than half of the load for search, 5 handle the online load for around 200 million queries and about 1.2 million inserts or updates per day for what Alexa currently ranks as 81st most popular English language site. Before some recent code improvements increased slave server use we had one server doing sustained rates in excess of 3,000 queries per second.
We'll soon have 40 web servers hitting those database boxes, so we'll take all the speed we can get.:)
Simply, MySQL gets the job done and has proved sufficient scalability and reliability in practice. PostgreSQL is also clearly an excellent database server. Each has pros and cons. Pick whichever fits the job at hand.
Memcached, Squid and PHP are also key parts of the scalability picture for Wikipedia.
No plans for Google ads. We do send search to either Google or Yahoo now when the database servers are under an uncomfortable amount of load. I'm usually the person making that decision and I decide solely based on balancing server load and reduced service.
Instead of static caches we're using Squid caches which get updated automatically when the content changes. We're looking to place more of these in other places which use significant amounts of bandwidth or are far from Florida in response time terms.
We're aware of the risk of excessive dependence on one donor and are looking to avoid it. We're entirely happy to talk with other companies who want to share in being seen to be helping something obviously good, limited only by the suitability of the offers for our needs. I don't know what the Wikimedia Foundation board would say but personally I'm entirely happy to accept hosting from Yahoo, Microsoft, AOL or any other significant player. In my personal opinion, neutral means just that - including neutral toward all companies in the business.
If any competitor is concerned about Google getting undue benefit or prominence, the solution is simple enough: talk with us as well.
We have offers of various sorts in Europe (though more, particularly from major carriers with excellent peering arrangements, would be very welcome) and the US. We don't yet have a substantial offer in Asia and that's a major hole I'd personally love to see filled. We're very popular in Japan and a location with good peering in Japan would be good for service there.
By this time next year I'd like to see 2-6 major remote sites with database slaves and apache web servers, capable of taking over as master if there's a failure in Florida, plus 10-20+ remote Squid caching sites. A massive amount of work (and donations) required to get that done.
We've already been blocked from China ourselves on several occasions. I've little doubt that it'll happen again and in other places as well from time to time.
It's absolutely certain that we have some unknown, uncorrected copyright infringement, offensive content, politically incorrect items for various parts of the world and assorted other things some or many people find undesirable. If the chairman of the board or president of the country is making headlines worldwide for some indiscretion, expect it to be in the article. Nobody who is unduly concerned about such things should consider offering hosting - we can't guarantee the absence of such content, just that we will try to be neutral.
We're not only interested in hosting and bandwidth. I'm particularly interested in high performance disk drives or systems, high capacity RAM modules (database servers like RAM but 32GB of ECC costs $11,000...) or whole high power database servers. To give some idea, I'm thinking in terms of three quad Opterons with 32GB of RAM and 12-16 15,000 RPM SCSI drives to keep up with demand for just the English language encyclopedia project over the next 6-9 months.
No part of this post should be taken as representing the official views of the Wikimedia Foundation or any members of its board. It is, of course, blatant soliciting for donations, as you'd expect from the guy who does much of the capacity planning...:)
Yes, there are software patents which are truly novel and represent a truly significant invention.
Then there are things like this one, which are so obvious that only an incompetent or inexperienced professional in the field wouldn't consider them the first time they were asked to make a shorter URL containing latitude and longitude without using server-side information.
Now, I do have some background in this area, for I've personal experience with latitude and longitude converted to integers for smaller storage (in a leading distance calculation tool for ship owners). And I've personal experience with changing the base to make smaller representations (in the parts of file names after the period, so more can fit in three characters, as part of a payroll application for ship owners). The former is routine computer science. The latter used standard C language libary function, which represents an integer in any specified base. Neither of those represented any brilliant insight. They were simple professional competence, not requiring any great degree of skill.
Regrettably, patenting the obvious and basic tools is what is happening today and in my opinion the harm from patenting the obvious approaches and tools is far greater than the possible benefits from the truly innovative things. Better not to have the software patents and to use trade secrets, copyright and simple business efficiency benefits to reward the innovation.
With apologies for the effect on your income and a note that I've rarely encountered a lawyer I didn't consider friendly and helpful: progress in the field would best be advanced by abolishing your specialty.
Google is acting in the manner most consistent with the interests of the consumer.
I suggest that Louis Vuitton deal with the fact that yes, they do have competitors who do buy ad space adjacent to their products. Trying to insist that competitors products not be placed adjacent to theirs is not in the interests of consumers. It's Louis Vuitton which is acting improperly and should be fined in this case.
Meanwhile, I'll remember that Louis Vuitton apparently has so little confidence in their products that they need to try approaches I consider to be unethical to try to stop consumers from finding out about the alternatives in the marketplace.
Please note that Slashdot displays the destination of links adjacent to them, so no reader of this post can possibly be confused about where any of the links I have used goes. And yes, this post is intended to be amusing as well as make a serious point about competition, lack of consumer confusion and ethics.
Some here may not appreciate MySQL. On the other hand, for one of the top 100 English language sites and one of the top 200 in all languages, according to Alaexa.com, I see around 200 million queries per day on MySQL,combined with some 250,000 edits/posts (many updates per edit) per week. For this application. MySQL definitely gets the job done. Each has strengths and weaknesses, though - and each is also stronger and weaker than other database products as well.
What's original? People have been sending email to mobiles via relays (commonly called gateways) since 1987 or earlier. Want to send an email to a ship? On the CompuServe system (in this 1992 newsgroup post you'd send it to TLX:(telex number) and the relay would send the email to the telex machine on a ship (pretty mobile!). The mobile could also originate an emailt o go back to you. I used to tell people my "telex" address which would really email me through that system. That was predated by the Fidonet system, as described in this 1997 newsgroup post.
NASA applies lossy compression at source. They still can't get back all of their imagery, or anything close to all of it.