Slashdot Mirror
Corrupted Databases Are Fun
Notice how all these stories are disappearing? Is it because
I'm a censorship oriented Nazi? According to my inbox, thats
the only rational explanation. The truth is that we've been
having database problems for several days. Stories are
randomly disappearing. Its making me really cranky. I
haven't been eating or sleeping much. But I'm working on
it. I'm really sorry about all this crap. Believe me, its
pissing me off more than you.
I've often wondered why you use a database. I would have thought text files would have been more than sufficient.
Notice how things seemed to work better before slashdot became a full time job? KISS
The URL for ProFTPd and WuFTPd vulnerabilities discovered. is still accessible.
Rob, can you manually insert a URL into the top-level index for a previous day?
When windows9X startx fucking up? what do you do? REBOOT! Same with anything else, including linux (though more often than not you ought to reboot to see how much you may have fucked it up since the last time you turned on your machine :-) Save the database you have for all prosperity (before it all goes to hell) and start from scratch, whether or not you want to use the same database app or not is your choice.
Troll Line: Did MSQL ever do this shit?? Switch to it. BWAHAHAHAHAH!!!
wonder if any of the other comercial apps like oracle could "donate" their database package in exchange for some free advertising?
Man, do something. I know I could do without slashdot for a few days (damit I'm addicted!!!!)
You bastards!!!
Heh couldn't resist...
Did we actually geta taste of our own medicine for once or did something just all of a sudden "goof?"
Skippy
Actually, smart guy, flat files would be perfect. In the top level, you have directories serially numbered corresponding to the story ID number. Then inside that directory you have the story file, then a folder for each reply, inside which you store the reply text and another folder for each reply to that.
To keep an association between the author and the comment, you make a symlink from the author's directory to the comment. For ACs, you need not do this step.
Do you get it?
Pulling all of these comment threads out of a database is sick. Do you think USENET servers store all their articles in an RDBMS? Hell no.
OTOH, the RDBMS is perfect for managing many thousands of user accounts and preferences.
The Slash source is, naturally, open. Take a look at it ind you will see that it is quite inefficient.
Later,
Jeffrey Baker
jwb@cp.net
too lazy....
The source is GPL and on ftp.slashdot.org Read alittle before you post
Why don't you just use Sybase, for christ sakes?
It's free for Linux.
Sybase is much easier to administer than Oracle...
and supports features (like stored procedures) that you won't find with mySQL/mSQL.
Actually, it is *very* efficient when organized correctly. 'static-dynamic' is a class of design I've used for more than 4 years on the web when I wanted mega-scalability. A web server set up that way can take punishment like nobody's business and never even break a sweat.
Besides, it might be rather cool to see /. under the hood.
Is that why you guys haven't anounced that Sega :) But now I realize it is :) The only reason why they :)
FINALLY released a firm date for the U.S. and
European dreamcast (9/9/99.) At first I thought
were anti-sega nazis
becuase computers suck
are a hobby of mine is because I enjoy pain
I actually wonder why you use a relational
database. Threaded discussions are perfect
for hierarchical database. In fact, you could
just save all comments as files in nested
subdirectories and allow visitors access
to a specific portion of a dedicated HDD.
Or, if you feel there is security problem here,
you could use a hierarchical database package.
Rock on!
That's exactly what I did with my database!
I store everything in a PostgreSQL database and update only modified pages periodically. It really works quite well, once you have the bugs out of your code. I should probably set it up on a cron job or something.
http://www.edseek.com/
(shameless plug)
Geben Sie dem Weibchen ein anderes Bier!
d
We know slashdot sucks. It has been proven in your own mind. Congratulations.
Now your mission is to come up with a solid, superior plan.
Oh, and stick around day and night to monitor the site so your s00perjenius plan doesn't go awry.
Deal? If you can deal with that then flame away. Let's see your plan right here, right now.
(babylon@netroplex.com)
Boom.. slashdot not there... Boom.. try again.. again .. again.. again.. check linuxtoday.com.. check slashdot.. not there.. check freshmeat.net... check slashdot.. still not there.. check www.news.com.. boring.. check slashdot... it's there, but news from last june...check www.macosrumors.com.. MOSR sux now days.. check slashdot.. wow.. nothing ut icons..reload.. still icons.. Hmm.. check linuxtoday.com.. nothing new.. check 32bitsonline.. wait wait wait.. try www.zdnat.com.. hahahahahahaha... check slashdot.com hey, ney story.. Jerry who?
You have a problem with the time stamps! Hope this helps
mecca
The one second, one minute etc arguement doesn't really fly. You end up doing more work than you need to in order to get the work done. You should really analyze the needs and then makes the changes only once every x seconds, and then only if the data has changed.
The only y2k Problems are with MOFOS LIKE YOU who spread this hyperbole. Go F yourself
Calmly reinstall everything, make sure you have the latest Debian so everything is nice and secure, and keeping your current database file, revert to an old, working version of slashdot.
The new features are not really that useful anyways.
Microsoft SQL 7.0 is one hell of a product. Would have handled this load no worries.
You know I'm right.
You do have sanity checks for your database, don't you? I've found it is nice to have DB scripts which look for things which do not make sense.
In this case I see three obvious checks:
1. Search for all articles with dates before Slashdot began or after today's date.
2. Search for all articles which do not appear in one of the daily retrieves (use a query similar to the "two days ago" link maker, but for all days).
3. Search for all replies whose parents cannot be found.
Create a bunch more by looking at the relationships between your tables and things which could get out of synchronization (yup, I can spell sync).
Pizza and Jolt always help me in these situations.
:-)
:-) keep smiling.
I especially like how icons from two days ago
are showing up in the title bar now.
yep, that sounds like Rob. Notice that slashdot has *NO* Pro-Microsoft stories in archives. Although I distinctly remember seeing them posted and then disappearing. Damn this news site -- I want freedom of speech!!!
8)
(ah, sarcasm is so relaxing)
Considering your database corruption and the root compromise of the Slashdot server last year, isn't it a strange coincidence that the story about the vulnerabilities in the WuFTPd and ProFTPd software should be the one to disappear?
Rob et al, have you checked your servers in detail to see if they have been compromised again and re-installed from scratch?
The semi-live threaded replies make this very difficult to do without a database on the backend. It can be done with a script that actually modifies the HTML each time a comment is posted but it would have to watch very very carefully so nothing else runs at the same time and it would take a while to update all the pages. Another way would be to have a master-index type of program which would generate each and every page from a set of text files but that would be ungodly slow.
Just so we don't have another freshmeat-like incident because of dumb people bitching, sorry to hear about the problems, Rob...but thanks for working your ass off keeping Slashdot going :-)
MySQL is hardly Open Source(tm), I didn't know there was any pretense that is was...
Don't think of it as a flame---it's more like an argument that does 3d6 fire damage
I noticed a couple of comments disappear, too. And yea, they really vanished, not downgraded.
Weird things are happening lately indeed.
I wonder what tcx (the mysql guys) has to say about that, did you buy support?
- illuminaut, arbiter elegantiarum.
Rob the Party Pooper. Shooting down all the wonderful conspiracy theories. Well, this semi-downtime gives everybody a chance to rest up their flamethrowers. Right?
BURP!!
(with apologies to... M**PT!!)
--
--
=8^
Just wanted to say thanks, Rob! Without /., my work life would soooo much more productive, and that's the worst thing I can imagine. Good luck, and hang in there.
"shop smart:shop s-mart" ash
I, for one, -don't- think you're any kind of censorship nazi and a pox on anybody who does think that. You do extremely valuable work, put in an enormous amount of time & effort, and I certainly appreciate the results!
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
we appreciate it.
A good one.
http://gabrielcain.com/
I'd click on a message, and it wouldn't show the body of the messages...just links to the people who've replied to it.
Anyways, thanks for fixing.
<tim><
Nope, I saw this one ok. It only happens sometimes. Other times, it's just a grey screen (but everything has loaded up). I just resize the screen to force things to regenerate and the page then appears fine.
<tim><
You're doing a great job for people who appreciate it.
Maybe we should organize a stroke-a-thon -- have slashdotters clap if they like Rob...
while the internet community of hackers fixes the database problem.
What a great group of people that Slashdot attracts. I would look at "Rob's problem" if the _current_ source was available. I bet that many others with more time than I have would also be happy to pitch in and help.
Cheers,
smithdog
periodically creating static HTML files from a dynamic database.
A fully-dynamic system has to read the DB and spew crap out for every user (could be many requests per second!) The theory is that this system is the most "live" (i.e. changes are visible to everyone immediately). The reality is that the hardware requirements of this system are very high for a moderate to high traffic site (Slashdot surely qualifies).
A periodically updated system has to read the DB and spew data (this time to files, not users) only once every 30 seconds, 1 minute, 5 minutes, or whatever seems like a reasonable interval. Compared to running many times per second, this starts to look pretty nice. And you'd be surprised how large you can make the intervals while still maintaining a good user experience. An interval of 2 minutes would be almost indistinguishable from a truly live, dynamic system to most users, and 5 minutes is still completly within the realm of acceptability. The final optimization is to update the bare minimum files that need to be updated on each cycle. That's the real trick, but done right, it can be very efficient (thus allowing you to decrease the update interval even more). Think about it :-)
Oh, and as for corrupted DBs, what DB server are you using?
isamchk
Read all about it...
(Sorry if this is a duplicate)
I think the original query wasn't questioning the use of efficient access procedures so much as use of an RDBMS, because of all the other baggage that comes with the territory.
Whether it is a good idea or not depends on whether the disadvantages of complexity and lack of transparency outweigh the advantages of clean access API and high flexibility. The answer does not always favour RDBMSs because high flexibility can be irrelevant in many cases, especially in an informal web environment. A combination of DBM files and directory-based structuring can often provide both rapid access and a lot of flexibility while keeping complexity very low and transparency very high. Furthermore, this can interface trivially with standard file distribution mechanisms to scale the whole thing up to thousands of servers if you feel the urge, and without introducing single points of failure that are almost inevitable with commit-based solutions.
It is a mistake to head blindly down the RDBMS route without seriously considering that there are strings attached. Use them where it is *necessary*, not where the tempting illusion of a simple SQL API makes it appear merely convenient.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
Chill out Dude!
Great service you have here,for the people!
Keep up the good work! Thanks!
if the source for v0.3beta were actually out there then this coveted 'bazaar' model might actually be helping slashdot's woes. same for adfu, it doesn't crash as much as it used to, but, everysooften, i do not see an ad, and instead see a 404 error overwritting the slashdot header.
again, where's the code? where's the remote cvs?
Obviously it's due to someone just like me (except for their being talented, experienced, and knowledgeable) who is also frustrated by growing "things to read on slashdot/hours in the day" ratio.
How 'bout a Reader's Digest Condensed version?
Squishdot?
I see even classic Slashdot is now pretty much unusable on dial up anymore.
I don't post much, but I just thought I'd say I appreciate your hard work, Rob. This is the only site I hit on a (many-times-during) daily basis. Thank-you for Slashdot. We'll be patient.
I think /. already does this. the db keeps the articles and everything, and a daemon rewrites the pages every few minutes. that's why comments dont appear immediately.
Everyone seems to forget that Slashdot has been growing by leaps and bounds lately. Don't be suprised that there are a few problems. Be suprised that it hasn't crashed and burned long ago.
Why? I'd guess that it probably has something to do with the fact that Rob et al have been busting their butts to keep it running so all the Slashdot Longhairs can make them feel appreciated by telling them how much Slashdot sucks.
I know I'm kicking a dead horse, but it seems that most forget that this is a free site, run by volunteers who care deeply about it. I'm not saying constructive criticism is bad, but there's a reason they call it constructive.
Rob, thanks for running the best site on the Web. Keep up the awesome work. (And don't OD on those caffienated penguins!)
Chris
"Wherever you go, there you are."
Have a cron job do a mysqldump of the database every hour - that way you at least won't lose as many articles. You could write a quick program to integrate in any newer comments using the unique IDs.
I hope you've paid the $ 200 for support and asked the mySQL folks about this. I run a much lower volume web site with mySQL, and I've never had any problems. But in case my sites become higher volume, I'd sleep a lot better at night if I saw a resolution of some sort to these problems.
D
Jeezus, calm down buddy. Rebooting is not a solution. There is still a problem, it might just be hidden away for a while longer, or it might not come back up at all. Give CT a break, he's working at it, I'm confident that he will fix it, just give him some time.
-matt
I'd just like to say that I'm behind the entire /. group and that I don't think you should switcht the database unless you want to. Take your time, find out what the problem is, and fix it. If it takes 2 weeks, fine. If it takes 2 years, fine. I would like to start a fund to donate Jolt and Pizza to feed our nice /. hackers so they can get the job done and get it done right. Everyone knows that real computer nerds do their best work while loaded up on caffine. In short, if you feel like yelling because /. is broken, STFU. You didn't do anything to deserve this site and you certainly have no right to demand anything. I'd like to say thank you to /. and that I'm praying to Linus for you all.
-matt
mysql saved my life. Msql is in the shitbin.
Hey, all of my invisible friends have dissappeared, and the voices don't know where they are. Has anyone seen my invisible friends? Anyway , Rob, me and all of the voices in my head wish you the best!
:)
Keep up the Good work, Metaphor
We're with you, Chaos
Don't let it get you down, Reality
We feel your pain, Master Switch
I am possessed by daemons.
-Master INIT
-Master Switch, one more element in the machine
Ah, shrug 'em off, Commander. If anyone thinks you're a Net Nazi, well, they've never seen a real BOFH then.
Keep up the good work, and you might consider a vacation, if you want to train a lackey. No, I'm not volunteering; I'm a greeeeen Linux newbie!
Believe nothing, not even if I say it, if it violates your sense of reason -- Buddha
Swear to god, I only eat like 20 a day, and I spend half my day in the bathroom, pissing.
No.
...
You don't suppose these database problems could be a symptom of a Y2K compatibility problem..?
Just a thought..
Whoa...what's going on here? The guy (Rob) works full time to provide geeks like us with a helluva place to get news, and then gets flamed when it's not perfect? That's not cool at all. If you guys can't deal with a few missing stories (even though they're all out on the web, and just linked to from here), then you have some serious issues
In the end-thanks Rob, you're doing a great job, and good luck getting everything back to normal
Juiced? Or Not?
It would be quite amusing (I might even hazard ironic) if this particular article fell victim to the self corrupting Database.
Please know that you are a kind of a god for many readers. I don't know what I would do with 24 hours in a day if I didn't have /. to read in my spare moments. Great work, Rob. I know what you are up against (though my site only sees about 1/100000th of the traffic :-)
-Signed Me
-- Infatuated with the fact that they can, software developers usually don't stop to think if they should.
Rob,
You are indeed doing a great job with Slashdot, but it definitely seems like it's time change the database. You shouldn't use a product just because it's Open Source.. if it doesn't work, dump it. We all criticize Microsoft and other companies for writing buggy software that breaks. We should be just as demanding of the Open Source projects.
Good luck with everything...