Slashdot Mirror
Slashdot Posting Bug Infuriates Haggard Admins
Last night we crossed over 16,777,216 comments in the database. The wise amongst you might note that this number is 2^24, or in MySQLese an unsigned mediumint. Unfortunately, like 5 years ago we changed our primary keys in the comment table to unsigned int (32 bits, or 4.1 billion) but neglected to change the index that handles parents. We're awesome! Fixing is a simple ALTER TABLE statement... but on a table that is 16 million rows long, our system will take 3+ hours to do it, during which time there can be no posting. So today, we're disabling threading and will enable it again later tonight. Sorry for the inconvenience. We shall flog ourselves appropriately. Update: 11/10 12:52 GMT by J : It's fixed.
*Clap clap clap*
Curiosity was framed; ignorance killed the cat. -- Author unknown
... roll over to be a last post?
Anyone could have made the mistake.. good to keep us all in the loop though :)
And let this be a reminder to the kids - RTFM, twice!
welcome our 2 to the power of X overlords.
Last post!
Alright, who's the joker who posted the 16,777,216th comment?
:D
Thanks for breaking slashdot, jerk
LegendMUD
Its like y2k, only worse!
Does this mean that comment id#16777215 has the longest thread in history?
Can anyone actually find it to see - I tried but could only get to 16777217, its likely to be in a journal or just a reply to an older article.
liqbase
As if a thousand geeks all made the same damn "last post!" joke at once. . . . . .
I mean, look how quick we got to 16M comments. 4.1 Gigacomments will come in hardly any time at all. I predict we'll be doing all this again in merely a few weeks!
SIGSEGV caught, terminating
wait... not that kind of sig.
I wonder who posted comment #16777216. That person should win some sort of "I borked Slashdot!" award.
Slashdot Burying Stories About Slashdot Media Owned
...why wasn't this problem discovered on the dev system in advance?
"I use a Mac because I'm just better than you are."
...comment 16777215.
Mmmm... CT, are you sure the parent index was your only problem?
"2^24 comments ought to be enough for anyone" -- CmdrTaco
Some of you are asking which comment it was that got the cid 16,777,216. The answer is that none did. For redundancy, Slashdot is now running multiple-master replication which skips values for auto-increment. Our db-1 assigns odd-numbered primary key IDs, and db-2 assigns even-numbered. Right now writes are going to db-1 so newly created rows will have only odd IDs.
The comment that got 2**24-1 was this one, if anyone cares :)
Sorry about the inconvenience, everyone.
That's cool, I'll just pretend I'm on Digg, with its 1981 Commodore 64 BBS-style threading.
Wait..sorry Commodore fans. I know it had better threading than Digg.
Haggard admins? Does this mean that the Admins will go buy some meth and get a massage?
Uh, this is a reply to the 8th post down from the top (remember to use this like an array and zero reference). Yes, I'm talking to you, admdrew.
You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced, and it would actually be the one after that one that would break it. You laugh but these kinds of problems plague a lot of coders?
If you don't agree with me, please respond below and reference my comment ID.
My work here is dung.
... should have been enough for anyone.
We don't see the world as it is, we see it as we are.
-- Anais Nin
No threading? Welcome to Farkdot.
Lars T.
To the guy who modded me down from perfect to terrible Karma - Apple haters still suck
Dupe! I TOTALLY posted this story like, last WEEK man! (I laugh, but I betcha someone might post this in seriousness)
USE colorful confetti ON heavily-armed clown
Flogging and Haggard in the same sentence? If we can get "crystal meth" in, we'll hit the trifecta!
The truth about Scientology, Xenu, and you: Operation Clambake
I used to work at Comair. Remember, that airline that stranded about 10,000 people in the airport a couple of Christmases ago? Same deal. Program was capable of handling only a certain number of changes. Hopefully your president won't have to resign.
So is the bug still in the CVS revision of Slash, or was it fixed 5 years ago and Slashdot never applied the patch?
Give a 2^0-year Slashdot subscription to the guy who hit the limit and one to the the first non-administrator guy who successfully posted after the fix.
If you can find the first guy who COULDN'T reply due to the limit, give him one too. He deserves something for his trouble.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Actually, comment 16,777,217 couldn't break it, because that comment's parent cid could have only been 16,777,215. Up until then, there wouldn't have been an overflow value put into the db.
There's no telling which comment it is, because (16,777,217 + 2n) might not have been a reply, meaning it would come up correctly.
:(){
Does this mean that Slashdot is going to denegrate into Digg now?
My blog
Reply to comment number 16786251:
...but it's probably not zero-referenced. Typically, ids in SQL start at 1.
``You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced''
Please correct me if I got my facts wrong.
I certainly admit I wasn't thinking 0-based when I wrote that. The question is, though, should we blame the person who wrote the last valid comment (therefore ruining the fun for the rest of us), or whoever wrote the first broken comment?
Also, is everyone going to add the obligatory 'parent' link on their posts today?
[ Parent ] - [ Reply to this ]
LegendMUD
Any thoughts on making the DB publicly accessable other than through teh Dot? Not sure what I'd do with all that data, but I'm sure these's a grad student somewhere who'd love the opportunity...
If brevity is the soul of wit, then how does one explain Twitter?
Brillant!
I always wondered where Paula Bean ended up...
mod parent up
The best education consists in immunizing people against systematic attempts at education. - Paul Feyerabend
Uhhhh who's your daddy?
A fool throws a stone into a well and a thousand sages can not remove it.
Slashdot being a news (for nerds) site, I would expect that the usage patterns are such that a huge majority of the content accessed by users is very recent -- say, perhaps, 90% of the database hits are for stories and comments that were posted in the last week.
So why, pray, is this usage pattern not accounted for in the database design?
Mod parent ... wait..
http://slashdot.org/~themusicgod1/journal/137880 ...ok, so it was obvious...
Take all the time you need, I'm more than willing to refrain from posting durin.... Oh shit!
If Slashdot released the Slashcode more frequently, with more/better comments/docs, and encouraged some of the many of us who complain about bugs/features to help the project, then it's more likely that someone would have debugged this bug earlier.
Open source - it's not just a buzzword, it's a way of life.
--
make install -not war
And this is why you should not have arbitrary limits in your programs, ladies and gentlemen. Not even limits on the values your numbers can represent
Now this is a real Slashdotter! This guy knows how to build an infinite computer!
Terrorists can't threaten a country's freedom and democracy. Only lawmakers and voters can do that.
Why on earth does MySQL have a 24 bit integer datatype? On what platform does it even remotely make sense to use that in the first place? It's going to get cast to 32 bits for any arithmetic operations anyway, and on most platforms today alignment requirements are going to pad the extra byte in memory and disk, so you're not even saving any space. Why even give someone the option over choosing between 16 bit and 32 bit integers?
This
poot_rootbeer asks why all the comments are in one table, when the data access pattern is such that 90% of our hits are on only the most recent entries in that table.
The answer is that we used to do it this way but it's a huge pain. In 2000 we converted from having two tables for 'stories', recent and archived, and merged them together. The performance hit was not big, and it made the code so much simpler it was a no-brainer.
It's the database's job to cache properly whether we split the table or not, and the database does that just fine. The only performance problem could be when there is a rush of inserts, or updates to the same sets of rows, spanning both newer and older portions of the table, and that just doesn't happen.
If we did want to do this we wouldn't split the tables manually; the code complexity is too high a price to pay. In MySQL 5.0 we would use a MERGE engine, which has issues of its own but would involve smaller changes to our code. That's still not worth it for us. What we're probably going to do is wait for MySQL 5.1 to get out of beta and then do some performance testing on tables partitioned by date and see if that gains us anything. For example, a SELECT on our comments table could be limited with a WHERE clause to only retrieve rows with a date >= the discussion object's date, which for 90% of our queries MySQL 5.1 could optimize to only look at the most recent partition. If the gains turn out to be significant, then since partitioning involves very limited code changes, we'll probably do that. Generally speaking, though, database performance is not a problem for us. So far our main bottlenecks have been CPU and RAM on the webheads. As long as we don't do anything stupid our database performance has been fine, though, as today proves, we are quite capable of being stupid.
[ Parent ]
Would not have happened if Slashdot used PostgreSQL.
Let the flamewars begin...
Unthreaded flame wars are much less enjoyable.
The number 2^24 is of interest to digital computer artists, as that is the number of unique colors combined in the commonly implemented "True Color" RGB8 space. That color space is looking pretty limiting in some respects, but that is truly a lot of unique colors when you think about it. A 16 megapixel image does not need to repeat any color used.
If all slashdot posts from the history of Slashdot were sorted into color bins,Once that were done, people could simply post their replies as a reference to existing posts. "Hey, #938D3A to you, buddy!" "Know what I think of that? #F2C2A9!"
[
One day, Cmdr Taco is designing his database, and he sits down at a table with three integers on it. First, he tries the baby bear's integer, but exclaims "2 meager bytes is way too small for my appetite."
Next, he tries Papa bear's integer, but proclaims "4 bytes is way too big for my little site, I'd just end up wasting so much."
Finally, he tries Mama bear's integer, and extols "3 bytes is just right," not noticing it was really the same as Papa Bear's bowl in disquise.
This
Get your mod points ready, this is off topic, but considering the current state of discussion anyway, I don't feel so bad about it.
/. admins won't (and shouldn't) consider releasing a copy of the /. DB to the public, something occurred to me.
/. are owned by the poster, according to that one line that shows up on all the comment pages (specifically, "The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.") At the same time, though, /. doesn't provide a method for having comments you've made removed from the DB.
/. from displaying them in future? Or is there some kind of implicit license in posting on /.? Did I clicksign an agreement covering this when I joined (this was getting on towards a decade ago, so I really don't remember the joining process at all)?
Regardless, while writing this post regarding why the
Comments on
If I own the copyright on the comments I've made, shouldn't I be able to rescind publication rights on them, and prevent
Or are publication rights, once granted, irrevocable?
Of course, I suppose asking questions when there's no way for people to hit reply is a specific form of vague insanity...still, I'm curious.
Reality has a conservative bias: it conserves mass, energy, momentum...
I'd really like to see it. I bet it goes something like, "what's this stupid web thingy anyway? I bet it'll never make it to version 2.0..."
The Kai's Semi-Updated Website Thingy
Ten points for honesty!
At least they didn’t try to make bullshit excuses. I respect them for being up front about the real nature of the issue.
Quantum materiae materietur marmota monax si marmota monax materiam possit materiari?
As your punishment, you should write some kind of data-mining algorithm that starts from the point you disabled threading and try to construct intelligent threads based on the subject and the body of comments...
Twelve-and-three-quarter inches. Unyielding. This wand belonged to Bellatrix Lestrange.
I guess this is another thing to add to the MySQL gotchas page. Of course, in a decent database engine, like PostgreSQL, if you alter a column data type then the indexes are updated to reflect this.
With all teh funnae posts about it, let me be the first one to ask: why were you using 3-byte integers to begin with? Why would anybody anywhere ever use these for any reason at all? What advantage to these have? Why was this table laid out like this? This doesn't make sense to me at all. Were you really imagining that shaving a byte off each post was going to save you DB space? I can't quite believe that. But than what exactly would be the motivation for using such an odd integer size?
We're all born with nothing.
If you die in debt, you're ahead.