Slashdot Mirror

Who still uses tape? Seriously, no data centric company on the planet still uses tape, its easier and cheaper to throw a bunch of large drives and a big fat pipe to offsite storage than deal with a tape robot.

People still using tape are doing so because they haven't moved on and like pain or are just ignorant of the alternatives.

Google, probably the most "data centric" company on earth, that's who!

http://highscalability.com/blo...

http://www.theregister.co.uk/2...

You're misunderstanding what's been written in that article. This is exactly the scenario that banks *have* to prevent before and as it happens.

These excerpts from one of Brewer's talks seem to substantiate my "misunderstanding": Eric Brewer on Why Banks are BASE Not ACID - Availability Is Revenue

segedunum:

Chasing around for compensation later cannot be an option in many cases because it is going to be abused.

When the system is functioning normally, the difference between strong consistency and eventual consistency is on the order of a few milliseconds. I don't think that leaves much of a window for abuse. The fundamental question is what do you do when there is partitioning? Or as you call it, system degradation. If you take an ACID approach then you shut down everything until the partitioning has been repaired. If you take a BASE approach then you still provide at least some functionality by sacrificing strong consistency. The CAP theorem says you cannot have both strong consistency and availability when there is partitioning.

Whatever system you use locally will be checked live, usually with a mainframe based system that is ACID compliant. If that isn't possible then you have a gradual system degradation where only certain types of transactions are processed.

The fact that you have any functionality at all when there is non-trivial degradation is due to using an overall BASE strategy instead of an ACID strategy. I have no doubt that one or more ACID databases are used as parts of the system but an overall BASE strategy is used by banks when there is partitioning (system degradation).

Remember, this thread started with an AC claiming that you would have to be an idiot to use anything other than ACID for storing data. People responded by saying there is also a place for BASE systems and that the banking industry uses an overall BASE strategy. Perhaps I misunderstand what you are saying but it seems like you are saying that as long as an ACID database is part of the system (or a central part of the system) then the overall system must be ACID which makes little sense to me.

I don't think anyone here is suggesting:

the article is [...] a carte blanche to justify NoSQL systems or to do away with any core systems that compromise ACID at their heart.

The point I've been trying to make is that just like there is a place for ACID systems there is also a place for BASE systems. In addition, as the data sets become larger and more complex and more spread out, the ACID approach becomes more and more untenable due to the CAP theorem. For most (but not all) cases, high-availability and eventual consistency will trump strong consistency.

There clearly seems to be a failure of communication here. Since you did not like my dumbed down explanation, perhaps you would prefer to hear what Eric Brewer has to say. He seems to have gotten a whole lot of awards for someone who is a "NoSQL nutter".

Eric Brewer on Why Banks are BASE Not ACID - Availability Is Revenue:

Myth: Money is important, so banks must use transactions to keep money safe and consistent, right?

Reality: Banking transactions are inconsistent, particularly for ATMs. ATMs are designed to have a normal case behaviour and a partition mode behaviour. In partition mode Availability is chosen over Consistency.

There are more details here and in many other places.

Acquainting a traditional RDBMS with a phrase like 'lower availability' just highlights to kind of twilight zone you start getting into when talking to any of the NoSQL crowd.

Are you saying you think the CAP theorem is false? I'm assuming large distributed data sets so partitioning is inevitable. According to CAP this means there is a trade off between consistency and availability. RDBMS provide strong consistency so they cannot also provide high availability when there is partitioning.

You didn't work on Mt Gox's systems at any point did you?

Sarcastic ad hominem attacks are an extremely poor substitute for reasoned debate.

> How could these not be important for banking is beyond me.

It's not that they're not important, its just that they are not the *most* important thing. Banks care about making money for themselves more than they care about anything else:

http://highscalability.com/blo...

Tell us which bank you work for so we can avoid it.

If a bank doesn't care about ACID, which means it doesn't care about losing completed transactions, which means losing track *OUR* money so they can get more profit.

I don't know where you live, but where I live, our regulators won't let banks off the hook for losing track of customers' money.

> How could these not be important for banking is beyond me.

It's not that they're not important, its just that they are not the *most* important thing. Banks care about making money for themselves more than they care about anything else:

http://highscalability.com/blo...

Banking transactions are generally not ACID. I'm sure the multi-trillion dollar banking industry are all complete idiots compared to the AC on /.

http://highscalability.com/blo...

If you're running a discussion forum that you share with 50 friends, sure, it can be in the first category and you can do it for peanuts and enjoy all the high quality interaction you like.

I disagree, let's have a look at Disqus, who (according to here) have around 100 servers total for serving (in 2011) "500 000 sites" with "15 millions of registered users" and "17 thousands of requests per second" for "250 million visitors (for August 2010.)". A later blog-post from Disqus itself puts that in perspective.

[Disqus] Recently hit two million concurrent users with 5 servers. Hit peaks of ~950K subscribers per machine and 40 MBytes/second per machine with the CPU usage under 15%. [source]

Let me re-iterate: they're handling 2M concurrent users with 5 servers!!
Granted, they don't use VPS' for 5 bucks (and they use more than those 5 servers), but what they're paying could be considered 5 bucks if they were playing in "our" ballpark here. Another interesting tidbit from there:

5 push stream servers were required because of network memory limitations in the kernel. [...] Otherwise could run on 3 servers, including redundancy. [same source as above]

Ergo, a lot is possible, if the architecture is right. (If you're running slashcode, of course, then... well...)

I also run some commercial sites, aiming at a wider audience, charging real money for signing up. [emphasis added]

Great! So you've got users who pay for the extra effort. My post was referring to that guy who was arguing that the internet needs ads, because everything is so expensive, which I still think is utter BS.

But running a significant news or social networking site with thousands of participants? Not even close.

I don't know about social networking, but news sites can be made static, cached, and hosted cheaply. I don't have metrics here, but I think it's safe to assume that if Disqus can serve 250m visitors on 100 servers, you'll be able to serve a million and more on one; especially for a static site.

Also, I've implied a counter argument to MojoKid's statement that "The internet is no different than any other media," in the sense that you don't need to buy several Heidelberger's for a couple hundred thousand or more to start a news site. The upfront investment is almost totally negligible and a small percentage of subscribers is enough, once you hit the limits of your initial infrastructure. And when you do hit the limits, in most cases [educated guess], you will have some users willing to pay.

Yes Yes yes! SSDs are a God-send for performance issues!

Database servers? Check!

Caches? Check!

Session Managers? Check!

Load Balancers? Check!

As stated by Reddit: Think of SSDs as cheap RAM, not expensive disk.

Have you ever used Stackoverflow.com ? Congrats, you just used IIS/Exchange/Windows. Oh and it scales really well and is used by a lot of popular web sites.

http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html

Scalablity is always an issue, and no test can accurately predict what is going to happen when 10, 000, 000 logins drop by.

http://highscalability.com/blog/2007/11/13/friendster-lost-lead-because-of-a-failure-to-scale.html

http://www.stackless.com/

They are using Python 2.7:
http://community.eveonline.com/news/dev-blogs/stackless-python-2.7/

Great discussion of pros and cons of Stackless:
http://stackoverflow.com/questions/588958/what-are-the-drawbacks-of-stackless-python

Here's an interesting page with a few nuggets of info. In the discussion section, some people claim that the game used to crash with space battles as small as 100 ships. Clearly the game has been improved since then.
http://highscalability.com/eve-online-architecture

If you are really interested, here's a talk from PyCon 2009 that goes into some detail on what they do with Stackless. They had some problems that only showed up on the crazy load of a real system, so they had to go live with some code to test it!
http://blip.tv/pycon-us-videos-2009-2010-2011/stackless-python-in-eve-pt-2-1959372

P.S. A couple of good trailers:

http://www.youtube.com/watch?v=HrrVDV_NsNo

This one bored me at first but then got much better as the music got going.
http://www.youtube.com/watch?v=euMjOHgb9A8

LinkedIn Moved from Rails to Node: 27 Servers Cut and Up to 20x Faster http://highscalability.com/blog/2012/10/4/linkedin-moved-from-rails-to-node-27-servers-cut-and-up-to-2.html

Having said that, I've started using Node in a limited way and it is obviously immature. NPM is a mess. There are far too many almost the same packages, and no obvious way to choose between them without reviewing the code. A package may not be compatible with the latest release, but there is no way to tell without installing it to try it out. These are signs of a still evolving ecosystem.

Still, it's only a matter of time until the rough edges are smoothed out and Node becomes accepted as a legitimate server side alternative.

5% actually makes a huge difference. "Latency matters. Amazon found every 100ms of latency cost them 1% in sales. Google found an extra .5 seconds in search page generation time dropped traffic by 20%." [link]. These statistics would not be true if the average Joe would not notice them. He notices, he just wouldn't phrase it as "this site was 100ms slower than usual so I didn't buy from it."

I'm going to add a couple of articles I liked for your consideration. The articles and some of the technology are old, but the ideas are probably still sound.

http://highscalability.com/amazon-architecture

http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster

http://queue.acm.org/detail.cfm?id=1142065

http://www.webperformancematters.com/journal/2007/8/21/asynchronous-architectures-4.html

I'd be curious to see responses to these since these articles are old. I haven't been to these websites in a while, so maybe they have some other interesting and more up to date articles.

As for PostgreSQL, I'd recommend this book. The first few chapters apply to any database. The next relate to PostgreSQL specifically. If you're a code head (like me), this book may be better suited for the DBA, but we don't know your specific needs.

I'm going to add a couple of articles I liked for your consideration. The articles and some of the technology are old, but the ideas are probably still sound.

http://highscalability.com/amazon-architecture

http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster

http://queue.acm.org/detail.cfm?id=1142065

http://www.webperformancematters.com/journal/2007/8/21/asynchronous-architectures-4.html

I'd be curious to see responses to these since these articles are old. I haven't been to these websites in a while, so maybe they have some other interesting and more up to date articles.

As for PostgreSQL, I'd recommend this book. The first few chapters apply to any database. The next relate to PostgreSQL specifically. If you're a code head (like me), this book may be better suited for the DBA, but we don't know your specific needs.

Agreed. This guy doesn't really understand scalability.

The OP needs to read how Plenty of Fish started off:
http://highscalability.com/plentyoffish-architecture

* PlentyOfFish (POF) gets 1.2 billion page views/month, and 500,000 average unique logins per day. The peak season is January, when it will grow 30 percent.
POF has one single employee: the founder and CEO Markus Frind.
* 30+ Million Hits a Day (500 - 600 pages per second).
* 1.1 billion page views and 45 million visitors a month.
* Has 5-10 times the click through rate of Facebook.
* 2 load balanced web servers with 2 Quad Core Intel Xeon X5355 @ 2.66Ghz), 8 Gigs of RAM (using about 800 MBs), 2 hard drives, runs Windows x64 Server 2003.

And also about NginX:
http://www.aosabook.org/en/nginx.html

If you "need" multiple servers when you are first _starting_ out you're probably focusing on solving the wrong problems.

Oblg. :-)
http://highscalability.com/blog/2010/9/5/hilarious-video-relational-database-vs-nosql-fanbois.html

But your old fasioned DB isn't "Web Scale": http://highscalability.com/blog/2010/9/5/hilarious-video-relational-database-vs-nosql-fanbois.html

Sorry, I love this video..

Unfortunately, it didn't work out that well for Hitler either.

But your old fasioned DB isn't "Web Scale": http://highscalability.com/blog/2010/9/5/hilarious-video-relational-database-vs-nosql-fanbois.html

Sorry, I love this video..

A large part of Instagram's value exists in the experience and strategies that allowed an initial three employees to manage a scalable, distributed application serving 10s of millions of customers. If Facebook is able to successfully incorporate Instagram's knowhow into their current stack, they could see significant savings in operations and management. That, alone, is worth billions of dollars.

More on Instagram's architecture...

You might be interested in this site: http://highscalability.com/

Indeed - BerkleyDB was NoSQL before NoSQL was cool!

Sadly Joel Sprotsky the ex-Microsoftie still has a lot of work to do: choosing .NET and Windows server (!) was probably far from the smartest idea. All the biggest websites (eBay, Amazon, GMail, Google, Wikipedia, etc.) do NOT run on Windows servers for a reason.

Perhaps you should start by looking at how StackOverflow actually does it since a majority of their servers *aren't* Windows based. http://highscalability.com/blog/2011/3/3/stack-overflow-architecture-update-now-at-95-million-page-vi.html

Facebookâ(TM)s New Realtime Analytics System: HBase to Process 20 Billion Events Per Day

Via: High Scalability:

The need for such a high powered analytics system is driven by Facebook's brilliant plan for world wide web domination via the viral propagation of social plugins, all tying the non-Facebook web back into Facebook and the Facebook web back into the non-Facebook web. Basically anything that people can do is captured and fed back through Facebook and anything done on Facebook can be displayed on your website, building closer relations between the two.

So which large sites use Java to power their site ?

Ebay
Amazon uses it, among other things
google uses it, among other things (you name it, they probably use it somewhere)

Lots of corporate sites, intranet and extranet, use java. Java is extremely strong there. Some of these sites are small, some are huge.

Ultimately, most sites hide what language they're written in -- you have to go looking for it. To be more precise, they don't intentionally hide it, it's just that their sites tend to not make it obvious. (php is often more obvious than many others, however.)

php is used for more smaller sites it seems -- if you install your own blogging or forum package, odds are good it's php. java seems more popular with larger commercial sites. Certainly, these aren't hard and fast rules -- Facebook uses php very heavily (but it has some java in there too) and more than a few personal sites use java.

Which is used for more "major internet sites"? Got me -- but first you'd have to define that term. I would fully expect that php is used in far more sites total than java, however.

So which large sites use Java to power their site ?

Ebay
Amazon uses it, among other things
google uses it, among other things (you name it, they probably use it somewhere)

Lots of corporate sites, intranet and extranet, use java. Java is extremely strong there. Some of these sites are small, some are huge.

Ultimately, most sites hide what language they're written in -- you have to go looking for it. To be more precise, they don't intentionally hide it, it's just that their sites tend to not make it obvious. (php is often more obvious than many others, however.)

php is used for more smaller sites it seems -- if you install your own blogging or forum package, odds are good it's php. java seems more popular with larger commercial sites. Certainly, these aren't hard and fast rules -- Facebook uses php very heavily (but it has some java in there too) and more than a few personal sites use java.

Which is used for more "major internet sites"? Got me -- but first you'd have to define that term. I would fully expect that php is used in far more sites total than java, however.

So which large sites use Java to power their site ?

Ebay
Amazon uses it, among other things
google uses it, among other things (you name it, they probably use it somewhere)

Lots of corporate sites, intranet and extranet, use java. Java is extremely strong there. Some of these sites are small, some are huge.

Ultimately, most sites hide what language they're written in -- you have to go looking for it. To be more precise, they don't intentionally hide it, it's just that their sites tend to not make it obvious. (php is often more obvious than many others, however.)

php is used for more smaller sites it seems -- if you install your own blogging or forum package, odds are good it's php. java seems more popular with larger commercial sites. Certainly, these aren't hard and fast rules -- Facebook uses php very heavily (but it has some java in there too) and more than a few personal sites use java.

Which is used for more "major internet sites"? Got me -- but first you'd have to define that term. I would fully expect that php is used in far more sites total than java, however.

No, that's not the reason. Consider that PlentyOfFish was able to do a 30 million hit a day all dynamic website using "2 load balanced web servers with 2 Quad Core Intel Xeon X5355 @ 2.66Ghz), 8 Gigs of RAM (using about 800 MBs), 2 hard drives, runs Windows x64 Server 2003" for the web servers. There's a detailed writeup of PlentyOfFish's architecture and hardware here.

Oh man, don't you know!

We don't use relational databases anymore. They don't scale /sarcasm

Take a look at SQL alternatives. There's an interesting PostgreSQL use case which uses only open source tools to achieve a good horizontal scaling solution. This post tells a little about how they did it: http://highscalability.com/skype-plans-postgresql-scale-1-billion-users The post also says that there is a very similar approach using MySQL.

Are you trying to say MySQL or Postgresql are equal to Oracle in performance, reliability, or documentation?

I'm saying that oracle is becoming more and more irrelevant, as some high scale deployments of other software shows.

Some licensing schemes with oracle can wind up costing companies almost a million dollars per year. That equates to quite a few extra full time employed database administrators.. which would more than be able to make up for any perceived lack of documentation etc.

There really are very few scenarios left where you actually need oracle for your database.

Taking the 15x performance increase over the 1979 Cray, we find that there are about 4 doublings to get 15x (16x) meaning that the android phone roughly compares to a 1985 Supercomputer, which doesn't surprise me at all. My cheap, now antiquated WinMo smart phone easily plays 486-era DOS games in a virtual box emulator, despite being a radically different chipset. (Arm, not x86) So factor in approximately 50% cut in performance due to emulation, and you have my phone demonstrably comparing to (at least!) a midlevel Pentium, and that's a minimum.

Honestly, sometimes it's astounding to me just how much processing power we throw away because it's just so cheap. When you read just how much performance this guy gets out of a single-core Dothan it just blows the mind. Underscoring my point: did you know that Mailinator runs entirely on one, not-so-impressive 2 Ghz AMD Athlon and a whopping 1 GB of RAM?

It was documented at http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster that a language change would have provided a 10%-20% speed increase, while architectural changes that Ruby on Rails could easily accommodate would provide with them with increases of 10,000%.

Disclaimer: I work as a Java developer.

Java is freer than .NET, I'll grant you that.

However, Java manages to screw up language features, too.

Generics and the way they were implemented in Java 5 is the most obvious example. Rather than creating generics and adding full-blown support for them down the chain, they were implemented by erasing the generic type information at compile time, so that generics could be used with anything expecting non-generics.

Generics is one of the things that .NET 2.0 got right, with both compile-time and run-time support. Even if it did mean implementing a second set of collections.

Having said that, Java has also obsoleted some of their collections (Java doesn't deprecate classes, only makes them obsolete). For instance, the Vector and Stack classes are obsolete. The Dictionary interface and its Hashtable implementation are obsolete.

Now, to address the parts of your post that are flat our wrong:

Why is Java powering a huge part of Google (GMail, GWT, Android, etc.)

Android, check. GWT, kinda check (it's a Java library, so why you included it in this list, I'll never know). GMail... [Citation needed].

No, seriously. Google has the largest installation of Python in the world. To the point where, in 2005, they hired Guido van Rossum, the guy who wrote Python. He still works there today.

Since Python has been Google's primary language for the last decade or so, I'm not just going to take your word for it that GMail runs on Java.

Why is Java powering... Twitter

It isn't; Twitter uses Scala. So, while Twitter uses a JVM, they don't use the Java language. Also, Twitter's downtime was notorious in the past and isn't the example I would choose to use.

Used to be IBM blade servers, possibly still. Stackless Python for development.

We also use it extensively at Rackspace Email division. We generate about 200GB/day of logs from postfix and dovecot installs, and hadoop with mapreduce allows us to pull all sorts of metrics and diagnostic information in very short timeframes. It helps our customer facing support reps, as well as allows us to give more demanding customers the statistics and metrics that they want, plus it helps us with capacity planning and a bunch of other stuff.

And it's designed to run on commodity hardware.

http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data

~Wx

The reason MySQL is able to scale in that fashion is because it doesn't respect the integrity of your data. If you're building an application where occasional corruption is acceptable, that's all well and good, but if you need your database to be properly ACID compliant and still scale, you've got a more difficult task to deal with, one that MySQL doesn't even attempt. They made their compromises.

Skype seems to be having a lot of success scaling out PostgreSQL.

http://highscalability.com/skype-plans-postgresql-scale-1-billion-users

It's a shame that Monty has always been inclined to obfuscate these issues in the name of gathering market share for his little toy... personally, I don't trust him any further than I could throw him. He's got no integrity whatsoever...

Compare all of the above with a site like plenty of fish, which while handling only about 1/40th the traffic is using less than 1/200th the number of servers to do so, because they've chosen a compiled architecture.

What is a "compiled architecture"? They're running on the CLR but end up facing the same problems as everyone else.

Both Amazon and Facebook have a service orientated architecture and developers get to write services using a variety of programming language and runtimes. So while Amazon tie the frontend together using a C++ app, they're not using that language exclusively.

If it's spending most of its time waiting on service responses, a well designed C++ app scales out about the same as PHP. A PHP app server will need scaling up first but savings in development and maintenance costs more than make up for it. Adding 20 machines to your frontend cluster is still cheaper than the cost of employing an additonal developer.

What are you not understanding?

Compare all of the above with a site like plenty of fish, which while handling only about 1/40th the traffic is using less than 1/200th the number of servers to do so, because they've chosen a compiled architecture.

I've been trying to find stats for Amazon, which uses a primarily C++-based system, but haven't been able to dig up any info on how many servers they use, unfortunately.

Twitter.

Rails wasn't Twitter's problem, though there was some initial finger pointing in that direction. The problem was that Twitter initially had a simple synchronous architecture (which, sure, is the easiest thing to do with Rails, as with most frameworks) when it needed an asynchronous messaging architecture, and that they didn't do database optimizations that are out of scope for the web application framework. See, among other things, this article.

Actually, it uses a bunch of technologies, including C++, Java, Perl, and much more. Here are some specifics.

"PG however does not have any support for scaling".
Maybe you should go tell skype that.
They intend to scale it to handle 1 billion users.
http://highscalability.com/skype-plans-postgresql-scale-1-billion-users

Also, i have no idea of what you are talking about.
To scale up database solutions to huge mean to totally rethink how one design the database.
Using convenient stuff like stored procedures and even worse, triggers, in a large clustered system would be really stupid and a total performance killer. However, if one do design systems like that, one would need someone to blame..

So maybe thats why that isn't a very high priority(for continuent)? Things like that generally belong on application servers which don't mind iterating and algorithmisizing(well you go find a better word then..).

Databases are for set-based operations, and that becomes even more obvious the more the data grows.

Also, achieving a guaranteed zero loss of transactions kind of lessen the need from them in the first place..since the most important aspect of a database transaction is not to commit but its ability to lose itself..wouldn't you say? :-)

And you *always* need middleware to scale databases. It just isn't called middleware by salespeople.

This sure looks like a growth area for qualified developers. An audience poll at the Gartner Data Center conference in Las Vegas in November found that just 17 percent of attendees felt their developers are prepared for coding multi-core applications, compared to 64 percent who say they will need to train or hire developers for parallel processing. "We believe a minority of developers have the skills to write parallel code," said Gartner analyst Carl Claunch. I take the Gartner stuff with a grain of salt, but the audience poll was interesting.

McColl's blog is pretty interesting. He only recently started writing regularly again. High Scalability is another worthwhile resource in this area.

dude.. you don't need php. you need a clue: http://highscalability.com/

> What's tomorrow's postfix?

Maybe nginx? highscalability.com has an article about a popular Rails Facebook app that replaced a hardware load balancer with a dedicated nginx box. 10M hits a day, not bad.

But just ask the folks at Twitter how well its database access scales to meet demand - it almost put them out of business.

A lot of things that I read obout it remind me of Python in the early days - but I don't recall Guido van Rossum being so Great Leader-ish

A lot of things that I read obout it remind me of Python in the early days - but I don't recall Guido van Rossum being so Great Leader-ish, and there's also the difference that Python actually can be used for an amazing range of applications. Ruby can too; it's a beautiful language - until you cut its balls off with Rails.

Try here: http://highscalability.com/

These sites are pretty useful when it comes to planning high performance websites:

http://highscalability.com/
http://www.allthingsdistributed.com/

These are both decent starting points. Please report back if you find something good -- I'd be very interested.
http://highscalability.com/
http://www.allthingsdistributed.com/