Slashdot Mirror
Large Scale Web Apps Built on Open Source
prostoalex writes "Brad Fitzpatrick presented at OSCON with on overview of his little project. Interesting facts about the evolution of the Livejournal back-end architecture."
← Back to Stories (view on slashdot.org)
It's all LAMP.
you can do allot with Lamp, just look at....SLASHDOT!
CB$@#--C
free ipod and free gmail!
LiveJournal? Not anymore...
OMG! Today I had CEREAL!!!!!
With MILK!!!! OMG!!
My companies backend is mostly Java.
:)
We are using Oracle as the database, and Solaris as the UNIX, but we could be using MySQL and Linux.
In fact, we are investigating that right now
comment directly in my journal
Uh, like, you mean the Web itself? That's large scale, certainly was built, and is most certainly built on open source.
So, yeah, I reckon it can be done. I'm using the proof-of-concept to submit this comment.
Anyone know what that document format is since it's roughly half the size of the pdf?
Karma: Chameleon (mostly due to the fact that you come and go).
Why is there a password on this sxi file (star office presentation)... is the file not open source?
...right here.
It's powered by GForge, so it's backed by PHP and PostgreSQL.
There are a bunch of other sites running GForge listed here...
The Army reading list
Yes, the empty string.
It's better to be the foot on the boot than the face on the pavement. ~~ tkx Kadin2048
Maypole is a Perl framework for MVC-oriented web applications, similar to Jakarta's Struts. Maypole is designed to minimize coding requirements for creating simple web interfaces to databases, while remaining flexible enough to support enterprise web applications.
Ok, so most of the Journals lack even a scrap of entertainment value... but the data feeds are normally fun. Is there anyone left that hasn't wasted a few bytes on the following url?
http://www.livejournal.com/stats/latest-img.bml
Hint - its a constantly updating list of all the new images posted to journals. After a while you give up waiting for a hot chick to post and decide crazy survey graphics are as good as it gets. And then some hot chick posts her birthday party pictures, but she's only 14 and suddenly you wish you'd spent the day doing something else.
0daymeme.com: Great stuff.
Back in the .com days, I worked at a huge (now defunct) porn site. We had about 50,000 active hosted sites, 500,000 hit counters and a bunch of other stuff. We were getting tens of millions of page views daily, maxing out two 100 megabit circuits at times. It was all FreeBSD, a little Redhat, Perl, mysql, squid, apache, mod_perl and C. The only real closed stuff we used were BigIPs and traffic monitoring software.
The web is really a mixed bag that allows a mix of open standards, and proprietary software. To claim it is all open source is misleading. It is a dynamic network that allows development on multiple layers.
The most important aspect of the web is that the interface of the different layers were well defined and exposed...not that each line of code in the different layers is exposed.
Reading this provided a laugh, until I realized its a pervasive belief among the world. At what point does a person think, gee i've already got 5 servers running my site, adding more doesn't really seem to improve much; perhaps its time to write the site better?
It just goes to show that this person knew little about web site development and architecture - if they did they wouldn't have gone to this point. It also goes to show the failure of the technologies. Give it up slashdot crowd. mod_perl is not a valid technology for a large scale website! Perl was designed for a task, and that task was NOT enterprise application development.
A properly designed website with n-tier sepperation will be able to handle a large load and scale infinitly. You'll note that large websites who actually do real things besides logging people's daily problems don't use mod_perl and a thousand servers. There's a reason for this.
GForge really is great. We're using it internally at my workplace for request tracking and project management. Now, if only 4.0 would come out soon... :)
Who said Freedom was Fair?
About a month or so ago, slashdot was regularly dying while fetching pages. Anybody know what was actually causing the problem? I suspected it was Mysql, but don't know.
In any case, it seems to have quieted down some.
and yet it still can't withstand a good ole' slashdotting...
How is this "large scale?" Maybe it's medium-scale as far as the web goes, but otherwise, it's very much a lightweight app. From livejournal.org:
Per Hour: 6818
Per Minute: 114
That's 2 inserts a second, and maybe a hundred queries a second. Quite honestly, that could be handled by MySQL & PHP. Definitely not what I'd call "large scale".
I don't respond to AC's.
If you are looking for scalable OSS solutions, also look into Zope with Zope Enterprise Objects (ZEO).
Of course this got a troll mod but its what I've been thinking about Apple for a long time. If Apple had never gone to BSD or whatever, there wouldn't be as many Apple Nazi's all over this website and that article probably wouldn't have gotten modded a troll.
People, people, people. Chill out. What I said is true, the Web is built on open source. Not all of it but quite a bit. It was originally built on open source principles, long before the corporations even heard of it. It grew mostly through open source principles. And no, my comment wasn't meannt to be taken so seriously. I was using humor to illustrate a point. Too subtle humor, evidently.
I see your link to Java.net (running on mod_perl, by the way). Perhaps you like J2EE? The original goal of Java (err, Oak) was to program embedded devices. What does that have to do with anything?
I've come to the conclusion that people who say things like "____ doesn't scale" are merely engaging in vigorous and pedantic platform self-congratulation.
Yes, that's a euphemism.
how to invest, a novice's guide
Milk
But I think their 'Ballad of Michael Hutchens' off Bigger Than The Devil was the absolute best.
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
Not large enough scale to survive a Slashdotting...
I have used livejournal for some time to communicate and record various things with the lady in my life and I think it is very valuable as an imperfect effort to learn from.
When I initially started using it I found it to be relatively responsive, but over the past year years things seem to be getting slower and slower.
It is clear that his design isn't scaling well without reading the presentation, but after reading it I know see it as a sort of 'case study' to learn from.
But beyond that it has reminded me why any blog you actually want people to read should be elsewhere. Then again the quality of LJ blogs infamous...
OMG! He's got a goat link right on the front page.
Hey, Windows users, there is no such thing as "forward" slash, there is only slash and backslash.
You know the economy sucks when even the porn web sites are going out of business. DAMN YOU BUSH!!!
BTW, who got the unpleasant task of cleaning the keyboards there?
If you fixed that non-threaded code I hope you sent in a patch to the relevant people!
As a paying subscriber of Livejournal, I can say the only reason I even have an account is because of the friends that I have who use it. I would never use it as a case study for any technology. It's got huge performance problems, data loss issues, and usability issues. This may not be the fault of using OSS, but it definitely doesn't help it look good.
There is no longer anything that can be done with computers that is nontrivial and clearly legal. -- Paul Phillips
Is there something interesing here?
A firewall can not protect you from yourself. Turn off what you do not need. Do not use the firewall to do your work.
$10m/year is moderately budgeted? You must work for the government! Or 1999!
http://shit.slashdot.org/article.pl?sid=04/09/21/1 833246
Some may find it interesting that Wikipedia (covered earlier today on Slashdot) uses some code that came out of LiveJournal for caching: memcached.
Simpy
It's somewhat amusing that in the first load balancing example, one of the points of failure was Kenny. Especially since Kenny ALWAYS DIES.
Karma: It's all a bunch of tree-huggin' hippy crap!
Inside LiveJournal's Backend or, "holy hell that's a lot of hits!"
Believe me, it is taking all my strength to avoid making a certain obvious joke about this title..
exactly. try using it at 7 pm on a weeknight, its all but useless.
The way I see it, if you want to work for free, go for it. If you call it a hobby or whatever, great. But you're not a pro, and you can't be, not for free. Imagine CPAs, EEs, POLITICIANS! doing work, unpaid, that others gain from, one way or another. This isn't a profession by any stretch. It's a hobby, easily, but that's all. You can leave at any time, and hopefully someone else that has lots of spare time will take over... your hobby.
The P in LAMP refers to any of the following: Perl, Python, PHP.
The one that you are refering to uses java.
Moderators, please verify your sources before you mod informative.
;-)
It's kind of sad that moderators tend to be so biased at times on /. that it can be so easy to be a karma whore. Anyways, on to what I have to say...
Her explanation, of course, is not that she has a greatly inflated opinion of her abilities but that he teacher is anti-Christian.
Yes, this young lady may be on the ditzy side, as are most teens these days, regardless of their beliefs. I've seen the same type of drivel from a Marilyn Manson worshipping, black makeup-wearing gay boy as I have from the bubble-headed, bible-thumping cheerleader, except that the teacher was "a homophobic Christian bigot" instead of ANTI Christian.
At any rate, the little Christian cheerleader is right about one thing: a lot of teachers ARE biased against students--particularly in the humanities (English, and in Canada high school Social Studies in particular). I experienced this first hand in my senior year of high school. My beliefs tend towards libertarian ideology and conservative/free market economics. Social Studies teachers tend to be more socialist. I wrote a position paper in support of reducing government welfare programs to a minimum (whether it be corporate or personal). The resulting mark was 78 percent if I remember right. The highest mark I ever received from that teacher was 83 percent.
At the end of high school where I live, final exams are standardised, government-issued tests marked by a panel of teachers independent of the local high school--your teacher cannot influence your grade on the exam (I believe they aren't even permitted to see your completed test before it is marked). By chance, I could write about the same subject as the above-mentioned paper (you had a choice of three topics). Of course I couldn't remember the papaer word for word, however I used the same arguments, in close to the same order, as I did on the paper that originally scored 78 percent. Later I learned that I scored NINETY PERCENT on the essay (and 95 n the multiple choice/short answer...woo hoo!).
I think it's fairly save to say that a twelve percent difference indicates that there is quite a lot of bias and subjectivity in grading there...
You continue on stating opinion without making any strong argument by saying:
these zealots will continue to try to take control of this country
I've heard almost the same exact statement made a couple times before. One time it was coming out of the mouth of a hooded, cross-burning man to a news reporter in reference to Jewish people. The other time was quite recently, except the insult wasn't "zealot"--it was "pervert". That was from a demonstrator holding a cross and marching in a demonstration against gay rights. Fact is, most evangelical Christians are not zealots that want to toake control of their country--they just want to live their lives free of persecution and with the respect of others around them. This is no different from Muslims, or Jews or even athiests or gay couples who wish to have their relationships acknowledged by the state. Certainly, within ALL of those groups, there are funamentalist/extreme minority factions that would indeed love to take control.
You are free to state whatever beliefs you may have and I'll go to my grave to defend your right to do so, however I'd like to give you some advice: think a bit before making a blanket statement about a large group of people, whether it be positive or negative. You are likely to come across as closed-minded and even offensive to more than a few people.
Hmmm...large scale community web site with user journals that is feature-rich, skinnable and has minimal downtime (503 Service Unavailable? What's that?) - built using Open Source Software. Who knew?
Something the Slashdot coders could learn from, perhaps?
It was all FreeBSD, a little Redhat, Perl, mysql, squid, apache, mod_perl and C. The only real closed stuff we used were BigIPs and traffic monitoring software.
I worked HotMail a few years ago, and funny enough, we had a very similar setup. 10k FreeBSD servers, huge DB (coded internally), perl, squid, apache...
The only proprietary stuff was the network equipment, BGPs, etc. And this equipment served millions of users per hour, well over 10 million a day.
This article was bad timing. LJ is currently suffering growing pains from a missing hardware delivery and was slow for non-logged-in users before slashdot arrives. LJ serves around four times as much as slashdot all the time, so slashdot just added a few extra seconds to the ten second delay.
About the only thing you're right about is that the hashing is flat. I imagine if that becomes a problem it'll be remedied, and if someone else hits it first then they can just hack it themselves or bitch on the mailing list. Yay for open source.
Automatic expiration is there, as is explicit deletion. Explicit deletion is used when the application (LiveJournal) knows that some data in memcache is going to become stale.
The cache going away does not lead to data loss. It does lead to really shitty performance while the cache repopulates, but all of the data will still be in the database which is completely separate from the cache. If it was considered necessary, it wouldn't really be hard to load up a bunch of key objects into the cache from a script but that would be guessing which objects are going to be needed while just letting it repopulate and suffering some slowness for a few hours gets the right objects into the cache. Different applications have different needs.
I don't know what HA-NFS and AFS are, but I know that using Squid (assuming you're talking about the HTTP proxy) would be caching at the wrong level. Caching constructed pages is pointless because most pages are completely different for each logged in user. memcache caches the atoms of data necessary to build the page, such as information about users and journal templates.
Brad evaluated other solutions before starting memcached and found none to serve the need. memcached solves the problem that needed to be solved, and ended up being useful for other people too. CowboyNeal, for example, is tracking memcached and slowly implementing it for different parts of slashdot. I've not been paying enough attention to know exactly what's using memcached right now, but I know that some parts of slashdot are using it.
(I'm not an LJ marketing drone, just an interested third party.)
LiveJournal wasn't adequately planned from a business perspective either. Like many .com era companies, they went for massive uncontrolled growth.
Because of the ballooning user population, they've ended up in a situation where they've had to install a bunch of anonymous moderators to "control" abusive users, apparently using inadequate tools for the task and with little guidance. And as everyone knows, anonymity + power + no oversight = abusive behavior. See my signature link.
Brad admits he basically has no idea what the "abuse" team is doing, so the whole LJ organization is dysfunctional.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
I didn't realize that it had automatic expiration. I must have missed that somewhere while reading the documentation.
The cache going away does not lead to data loss. It does lead to really shitty performance while the cache repopulates, but all of the data will still be in the database which is completely separate from the cache. If it was considered necessary, it wouldn't really be hard to load up a bunch of key objects into the cache from a script but that would be guessing which objects are going to be needed while just letting it repopulate and suffering some slowness for a few hours gets the right objects into the cache. Different applications have different needs.
Don't feed me bullshit. memcached dies and so does your entire cache. That's significant data loss no matter how you want to spin it.
I don't know what HA-NFS and AFS are, but I know that using Squid (assuming you're talking about the HTTP proxy) would be caching at the wrong level. Caching constructed pages is pointless because most pages are completely different for each logged in user. memcache caches the atoms of data necessary to build the page, such as information about users and journal templates.
Squid doesn't just cache pages, you know. I can cache a wide range of data that's served over http. Sound familiar? If you've read the memcached protocol documentation, it should.
As for the others: OpenAFS and HA-NFS. So much for "evaluated other solutions". These are both lightning fast high-availability NFS replacements - AFS sports numerous features such as client-side caches. And yes, they are open source.
Whoop de doo. Slashdot is looking at memcached. Their DBMS is notorious for corrupting itself, so that tells me quite a bit about their availability concerns.
Like I said - this may work great for LJ and Slashdot, but there are enormous e-commerce sites (that believe it or not, use a heckuva lot of OSS) that have a little more to worry about than losing ad revenue for the 10 minutes it takes to repopulate memcached. Having that kind of downtime simply is not possible. You not only lose sales, depending on your caching strategy, you can get unrecoverable orders, or just outright lose customers because your site is slow. It's not uncommon, either, it's pretty much a guarantee if your site gets slow or goes down for any extended period of time - your full-service uptime directly correlates to sales for sometimes several months, and god knows you're fucked if it happens during the christmas season.
Hehehe...Karma still "Excellent."
:D
See ya in the M2 Buddy
WTF?!? I got modbombed!!!
***WHINE***