Slashdot Mirror
RIM Server Crash Leaves Millions Without BBM
Several readers have sent word that "tens of millions of BlackBerry users in Europe, the Middle East and Africa have been unable to receive or send emails and messages through their phones, following an outage at the server systems of parent company Research In Motion." RIM has confirmed that they're aware of the problem and working to restore service. A former RIM employee said to The Guardian, "They didn't start looking at scalability until about 2007, when they had around 8M active devices. The attitude was, 'We're going to grow and grow but making sure our infrastructure can support it isn't a priority.' They have their own clunky infrastructure to do something that you don't really need a clunky infrastructure to do anymore."
Is that the last minute is either "next year" or "last week" depending on which side of the disaster you're on.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
If Blackerry was still popular.
Isn't iOS 5 going to feature a very similar "clunky infrastructure" feature?
...
Okay, why does a disgruntled ex-employee's rant about scalability and infrastructure come into play before we know that scalability or infrastructure was the cause of the break? Seriously, maybe the taco bell dog just chewed through fiber lines in NY and LA while on tour. Could happen to anybody.
-- IANAL, this isn't legal advice, and definitely isn't legal advice for you. Also, Squee!
... they just don't know it yet. We have 40 Blackberry's in our company, but we purchased our last one about six months ago. I hope BES dies a painful, painful death.
Android, here we come.
----- obSig
Thats the problem with blackberry devices depending on RIM for service...
Your mobile operator could fail, but theres more than one operator...
Your own email server could fail, but your in control of this yourself and can take steps to fix it... Plus, it only affects you and not anyone else, you have a choice of email providers and if you run the server yourself its your own fault if it fails.
You are stuck with RIM service if you want a blackberry handset, you don't have a choice unless you switch to a different type of handset.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
It's as if millions of souls cried out ... and were suddenly silenced.
The company books ONLY show how expensive providing good service is. They don't show the enormous losses when inadequate investment in support infrastructure collapses.
While all of these solutions are scalable they have terrible latency; especially when nodes are failing. RIM probably needs very low latency for it applications.
Nothing of one was the othered.
And, you know, SQL databases, too, since they scale very well.
BTW, why do you jump to the datastore as the scalability problem? It's not mentioned in the article. For all we know, it could be their internal Cisco routers that are overloaded or their WAN links or their 10-year-old Windows 2000 servers or something else.
Advice: on VPS providers
In four years I'll be starting a company based on the idea of having a device that stores all your photos, emails, and applications locally so you aren't tied to the cloud.
This is generally the decision upper management makes. Step #1 Make the infrastructure as cheaply as possible in a way that works for the situation right now. Step #2 Get as many users as possible. Any band-aids that are necessary is just support for our infrastructure, and everything will magically scale despite our single point of failure. The justification for this, is that hardware is cheaper than software engineers, so it is cheaper to scale by just paying for better hardware. So, scalability should not be a problem as long as their engineers were competent building the initial infrastructure. However, this event shows they obviously were not.
He's jumping to the datastore because he can't cunningly fit in a link to his presentation about NoSQL database software if he doesn't.
. . . to ruin my chances of snagging a big, beautiful man.
My sister opened a computer store in Hawaii. She sells C shells by the seashore.
One word: CLOUD!
And you wonder why the shit fails and brings everyone down. Its the nature of cloud service crap.
Current infrastructure will be more than adequate to support the number of RIM users left a few years from now...
In case you haven't heard, relational databases have a dickens of a time scaling horizontally.
The whole point of datastores is scalability.
So what? It's a free presentation--I have not placed any ads on the page. It's called context, as in why I have experience to say what I wrote.
Jeez everybody is so suspicious these days. You know, there are some people who don't have to monitize everything they touch. For that matter, all of the NoSQL databases I mentioned are free--nobody is making any coin off of them. My link was harmless.
Will your presentation be discussing how many NoSQL database systems often corrupt or outright lose data because they don't offer proper ACID support?
Will your presentation be discussing how, in the real world, "eventual consistency" really means "never consistent"?
Will your presentation be discussing how anything but the simplest queries become extremely painful to craft, especially when the only query language available is JavaScript?
Will your presentation be discussing how Cassandra caused significant problems at both reddit and Digg?
Will your presentation be discussing how all NoSQL functionality can easily be implemented using traditional relational database systems?
Will your presentation be discussing how many of the performance problems that NoSQLers have with relational databases are solely due to those NoSQLers not knowing what indexes are, never mind how to use them?
Will your presentation be discussing how many of the scalability problems that NoSQLers have with relational databases are solely due to those NoSQLers not knowing what replication is, never mind how to use it?
If you don't mention such things during your presentation, then you will be deceiving your audience.
> have been unable to receive or send emails and messages through their phones
Maybe the users just don't hold their phones correctly, like it happened with another vendor...
lucm, indeed.
A campaign against RIM has been running in the media for the last few months. Before the iPhone and Android devices there was the Blackberry and I bet RIM still have a few cards left up the sleeve.
In case you haven't heard, relational databases have a dickens of a time scaling horizontally.
I love it when you NoSQL advocates make blatantly false and incorrect claims like that, without even realizing how wrong you are. But I suppose it's not surprising, given that many of you guys haven't even heard of indexes, and have never used a relational database aside from MySQL.
ACID support is not the goal of NoSQL. So your question is like asking if dirt makes a good bologna sandwich.
Eventual consistency is actually known as Tuneable consistency. You pick how fast you want your data replicated. There is no free lunch.
You are referring to CouchDB methinks. Bit player...
No, my presentation will point out how Facebook to this day uses Cassandra for inbox search. Just because you chose a particular datastore does not mean your architect can check her brain at the door.
Relational databases still fill a need, but because they need to support locking, they are orders of magnitude slower. Cassandra blows away MySQL. Oracle costs a bundle. Relational databases do not scale well horizontally. NoSQL databases--especially Cassandra--do scale well horizontally. No Master Nodes.
So, you're blaming the NoSQL database because its users may not educate themselves on how to use it. Blame the victim much?
Finally, Larry, why oh why did you choose to post this as Anonymous Coward. We know that Oracle is miffed that no startup uses your software, Mr. Ellison, but did you really have to go off half cocked like that?
Larry, I really think you need to go back on your Lithium... your mania is starting to show.
Not true, Pal. My first DB was IMS with its hierarchical structure, awesome speed and vomit-inducing PSB blocks. IMS was invented to support the SaturnV rocket. Then I moved on to DB2, supporting a claims table that held 65-million rows and was partitioned up the wazoo. The, of course, I have been through versions 8-11 of Oracle and its awesome SEQUENCES. Then I was forced to use that puke (stolen from Sybase) known as SQLServer (lacking indexes and forcing us to use index-like tables.) I have used MySQL and did not enjoy much about it except for the price.
No, Anonymous-Coward-weak-pussy, I have used all flavors of Relational databases. I know the joy of denormalization and partial dependencies and transitive dependencies. I know how triggers can kill a DB and how partitions can allow indexes to be rebuilt easily.
No, I understand all that--Anonymous-Coward-weak-pussy--and I know that there are simply some situations where relational DBs are just not right.
I think Facebook--which wrote Cassandra (and still uses it for inbox search but now uses HBase, another NoSQL)--knows a few things about global scaling.
So, moron AC-pussy, you are wrong.
When you consider the fact that most of Rim's internal tools don't scale well and there built off broken technology it was bound to hit the outside world. Rim's general outlook is have it crash first and blame the coops later. Very little of there info-structure is designed to handle loading and expansion. I'm surprised it took this long to happen.
The GP brings up some serious problems with NoSQL "databases" (that's not really the proper term, since their implementors don't seem to care/know about ACIDity), and you manage to ignore or sidestep every single one of them. Then you go off on some pointless tirade about Larry Ellison. Only somebody hyping NoSQL could consider making billions of dollars selling the relational database system that powers many of the most important software systems ever created to somehow be a "failure".
I hope that you show this entire thread of discussion to the people that you're presenting to. Let them read each and every comment. It'll do a good job of putting your presentation in context. At the very least, it'll inform your audience about some issues that you apparently don't fully understand, and thus seem reluctant to discuss.
Oh, chill, dude. Someone who saw the sarcasm tags rated it a +1, Funny.
Best of luck to you on your presentation.
I addressed each of his points in order. Can't you read? You say that NoSQL doesn't support ACID--when that's part of its design. There is no promise or intention of supporting ACID. If ACID is necessary--use a relational database! So, you think that Facebook is full of idiots? Why on earth would Facebook's engineers choose to base their entire site on NoSQL datastores if they were not useful!
Finally, the bit about Larry Ellision? That's called humor. You know, to laugh? Ha ha? If you don't find it funny that is your choice but obviously that part was intended in jest. I also notice that you--pussy--posted your comment as Anonymous Coward. That speaks volumes. If you felt such righteous indignation--why not stand behind your every opinion, as I have done? Pussy...
Wow, another Slashdot hatchet job on RIM. Heavy into Apple stock, are we?
RIM certainly has issues, and it may not survive. But it seems the Slashdot editorial staff wants to make sure.
Even when it's Google or RIM, an email server being unavailable isn't what I'd call "news".
I do not fail; I succeed at finding out what does not work.
That takes a lot of IBM printer paper. And the SaturnV supplied its own lift. Nice architectural separation.
This is one of the most unprofessional comments I've ever seen submitted to Slashdot. There's absolutely no need to resort to petty name-calling like you just did.
If I had some mod points for humor, they would all be yours... :)
Okay, Anonymous Coward. So, you can't even use your real name to express your indignation? Hah!
P-u-s-s-y!
Your statement was one of the most craven I've ever seen expressed on Slashdot.
Slashdot is not beanbag, pal. It's combat.
Maybe you should go here, where you feel less threatened
Oh wait, I don't care, I have an HTC Android phone. I gave up on RIM and their kiddie phones.
This would have been a first post if I had access to my Blackberry.
An even better idea would be to be able to sync things WITHOUT a damn iCloud at all! What a concept!
But you can ALSO do that. Without iCloud you are still perfectly free to sync media and applications onto the device, even if all you have is a computer and an iOS device and no network.
Why is having layers of possibilities not better than the alternative?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Wow, another Slashdot hatchet job on RIM.
In Slashdot's defense, there were already several hatchets in place by the time they got there, and they were pretty obviously self-hatcheted. Slashdot is just pointing out the obvious.
When you have a central point of failure you WILL HAVE FAILURES. It's like saying the sun will rise. I've never liked Blackberries for that reason long before any obvious decline began; I greatly prefer systems that lack such points of failure.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Forgetting that iOS 5 is all cloud-based,
It's not cloud based. It is, to put it in manager-speak "cloud enabled". The technical point behind the distinction is that iOS5 merely always for the cloud to provide a device with MORE data which it then stores locally. It's not cloud based at all, as content and media is locally based but then distributed.
that great new feature on the 4S, the voice assistant, is ALSO all cloud-based.
Really more service based, but yes that requires a data connection. But you can use the system without Siri, that's just a convenience. It's not like Siri going down prevents you from doing ANYTHING you could do otherwise.
Apptards find out that maybe a walled garden isn't the nicest place to live after all.
Do we get to call you an Andtard? Grow up and think rationally, as your desire to make fourth graders laugh has clouded your technical judgement. Then perhaps you will not make such embarrassing and glaring mistakes. No wonder you post AC, if I were to post something as lacking in substance and fact I'd try to cover my tracks as well.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
In case you haven't heard, relational databases have a dickens of a time scaling horizontally.
The whole point of datastores is scalability.
oh, you are THAT guy ...
http://www.youtube.com/watch?v=b2F-DItXtZs
Who logs in to gdm? Not I, said the duck.
Except it was off topic and thus astroturfing.
So, you think that Facebook is full of idiots?
Case dismissed!
The problem is not that system failures happen, but that businesses and people don't plan to deal with those failures.
Running your own email server will not prevent crashes. RIM crashes, Google crashes, a bazillion corporate email servers crash.
The trick is not to expect anyone to really deliver 24x7x365 uptime, because no one has ever actually done so. The closest they've come is playing word games with the service contracts and reason for outages so that they can still claim 5-nines uptime, even though their honest stats might be more like 3-nines at best.
I like a poster I read several years ago: "Failure to plan on your part does not constitute an emergency on my part."
I do not fail; I succeed at finding out what does not work.
on365 is a company that they hire and pay too much money for people to run their DC's next to a fixed BB-PM/Tech (which was me).
These are people that know how to rack a rail kit in in many ways and do it in the most incorrect way. People that don't know what cat5 is and how it works.
These people get payed in the UK more then 4000 pounds and they get hired by BB to work in NL, BE and FR. \
When I told them that they should hire fixed BB personal for DC's instead of people that cost more then managers and deliver less then a monkey, they were like.... Yes, we should, but we will not! ..... So I quit the job and waited for errors!
This is the second they made so far! ^^
While all of these solutions are scalable they have terrible latency; especially when nodes are failing. RIM probably needs very low latency for it applications.
When the endpoint of its applications is either receiving traffic from, or sending traffic to, a small underpowered device typically at the end of a cellular network connection, I'd have to say that the latency of its inter-datacenter connectivity for its redundant backend data storage should be the least of its concerns.
You're special forces then? That's great! I just love your olympics!
You don't know what you're talking about. 95% of Facebook runs in an in-memory layer with write-back to MySQL (pages are never generated from the database, or whatever datastore you're imagining). I work there.
Fucking brainless slashdork editors. Sometimes it feels like the people who say that blogs and other "user published material online" is killing journalism were right.
I don't believe you, Anonymous Coward. To start Facebook invented Cassandra. Are you saying they no longer use it for Inbox Search? Secondly, I have read loads of articles from people who sign their names to their articles, unlike you AC, saying that Facebook uses HBase for several places where they found Cassandra was up to the task. All of the textbooks (published in 2011) that I have studied list with precision how Facebook uses Cassandra and HBase. So, I will accept that Facebook may use MySQL but anytime someone uses a made-up statistic like "95%" I have to believe they are exaggerating. Also, anytime someone posts as Anonymous Coward, I question their veracity. That would include you. If you speak the truth, why are you hiding behind Anonymous Coward?
Whether it's constant or intermittent makes no difference, if access to those servers isn't available when you need it then you can't use those features.
If Siri is down you just use google or make an appointment yourself.
If iCloud is down you just listen to the music already synced.
The operation of the device does not RELY on them being up any one moment. The benefit that comes from iCloud would be just as good even if the servers operated only 50% of the time.
The point is that Apple is relying much less on the network nature of the cloud than other devices or systems.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
In many apps those DB solutions are providing the front end traffic and not everyone is on a cell phone. Especially with the rates and bandwidth constraints being currently charged in the US.
if icloud goes down then you lose all of them
Nope, I can still play music, or see videos, or use apps that have had data synced by iCloud.
For example 'Find my friends' when icloud goes down?
Yes, that is not part of iCloud.
There is not a feature of iCloud I can think of that stops you from using any aspect of it directly when down, the advantage of the local cache model.
"There is more worth loving than we have strength to love." - Brian Jay Stanley