Blackberry Network is Down

Brian writes "According to CNET and others, "A system failure at Research In Motion has affected BlackBerry users in the Western Hemisphere, a news channel reported on its Web site late on Tuesday. The infrastructure failed on Tuesday night, and e-mails were not being delivered to the handheld devices.""

Comments

must resist Great Disturbance In The Force joke...

[ScentCone]

Because millions of voices really ARE crying out in pain. Man, the cooler we make things, and the more that entire business cultures get built around this stuff, the more fragile it is. Just think of the war stories we're going to hear about people who've come to utterly depends on their Crackberries having lost a deal, not heard that a critical server was down, not realized that a surgery had been rescheduled, and so on. I wouldn't make a living if people didn't depend on fancy networked technologies, but it sure does feel like a house of cards, some days.

Not very reassuring.

[Dan East]

A single point of failure can bring down the entire network? Not very reassuring, especially considering Blackberry is predominately a business tool.

Dan East

Re:Not very reassuring.

[lucabrasi999]

especially considering Blackberry is predominately a CRITICAL business tool.

Contrary to popular belief. E-mail is NOT a critical business tool. You don't believe me? Then compare e-mail with the following:

Supply Chain Systems? Critical.

Customer Order/Customer Relationship Systems? Critical.

Manufacturing systems? Critical.

Payroll? Absolutely the single MOST critical application at most enterprises (especially during pay runs).

But, e-mail is not critical. When I've been involved in storage meetings, e-mail is always a Tier Three application. In other words, while it is a useful tool, there are other ways of contacting people in an emergency. You know, like by telephone.

Re:Not very reassuring.

[fabs64]

This is a ridiculous statement, quite often those things you stated rely on email, as it is a means of COMMUNICATION, and in quite a few businesses, the primary one. Supply chain is worthless if crm can't communicate with them, ditto for manufacturing, payroll is worthless if they can't communicate with accounting or HR. Also note, for those things the phone is not an option, it must be in writing. Try to keep in mind that many, many businesses do not reside in one office, or one building, or one postcode, or one country, before you make a silly comment like "e-mail is not critical".

And even putting aside all that, it depends on the business, for an IT consultancy, network admin, software dev house, e-mail is pretty fucking critical.

Re:Not very reassuring.

[lucabrasi999]

Wow, you must be one of the users that is freaking out over Blackberry today.

Supply chain is worthless if crm can't communicate with them, ditto for manufacturing, payroll is worthless if they can't communicate with accounting or HR.

Most of the systems I describe have a DR and/or failover capability because they are considered critical to the operations of the business. Personally, I have NEVER seen an e-mail system that has had DR or Failover. Beyond DR, ask an even more important question: In the event of disaster, how quickly does e-mail come back up on-line? Is it the first system recovered? Or are there 35 other systems that are recovered before e-mail? That should tell you how important e-mail is in comparison to your company's other systems.

Now, let's talk about using e-mail as an interface: Does the communication you describe have to occur via e-mail? I have been working in IT for 15 years and I have never seen a place that depends upon an e-mail to interface between any of the systems I described. For example, if I am running a Supply Chain system, I can keep the supply chain moving for hours without worrying about ANY e-mail communication with external systems. Well-designed Supply Chain systems use EDI (not e-mail) for communication. And, EDI is usually built with redundancy and DR capability If they don't use EDI, they depend upon someone manually entering the information into a screen, or even a flat file. If I absolutely MUST contact someone, I just call their phone. As for Payroll, payroll runs at even small companies are not dependent upon e-mail. The timekeeping systems interact with the Payroll systems via EDI or some sort of file transfer that is NOT dependent upon Exchange/Domino.

Hell, even if I am in hardware support and my e-mail server crashes, I should be smart enough to log directly into my messaging console to keep an eye on the status of my other servers. Most monitoring tools use e-mail to send messages to Operations. My operations department had better be able to log into their consoles to check the systems real-time and not be waiting for the e-mail system to recover.

Also note, for those things the phone is not an option, it must be in writing.

There are a few options here: If you are running a normal PO approval process, then there is always an expected time-lag in approvals. My manager cannot be expected to drop everything to approve my new business-card order system. And, if there is an emergency, well-designed systems have an override that allows for exceptions.

Try to keep in mind that many, many businesses do not reside in one office, or one building, or one postcode, or one country, before you make a silly comment like "e-mail is not critical".

Firstly, I work for a Fortune 500 company. We are spread all over the world.

Secondly, There is nothing silly about my statement. E-mail is NOT critical. People THINK it's critical because it is the system they have the most 'face-time' with. But, face-time is not equal to criticality. If your organization has built interfaces or business processes that depend upon someone approving something via e-mail, then you have a very poorly designed system. You can easily have interfaces move directly between systems without ANY human interaction. You can build custom web-pages to allow approvers the ability to make their approval directly in the purchasing system.

Cold-turkey for Crackberry-heads

[redelm]

Well, it was bound to happen with a centralized/heirarchical service model. Perhaps RIM will learn and go decentralized. Perhaps not.

More interesting will be the addicts reaction. Some people really hang on the devices and get addicted to their Crackberries. I wonder how they will adjust (most people will do just fine) and what lawsuits will result. Or if the plantiffs are too worred about simply having their service cancelled!

Re:Cold-turkey for Crackberry-heads

[Dr. Eggman]

Lawsuits? Surely any network-based buisness with a legal department worth a lick of salt would include provision in their terms and agreements of services that cover such instances of blackouts, loss of service, or even financial collapse of the company leaving addicts with their network device without a network.

Re:Better headline needed

[Anonymous Coward]

COME ON! Two words. Two very short, very simple words. Even the vast majority of non-English speakers get this one correct. What possible excuse can you have? I'll accept dyslexia as the only valid answer.

Uh, system reset...

[OnlineAlias]

"Officials with RIM said they are trying to reset the system and told NewsChannel4 that they are concerned that the backlog of data, which will rush through when it comes back on line, could cause a bigger problem"

When in doubt, reboot!

It IS a house of cards

[Opportunist]

Reason? It's cheaper than bricks.

Seriously, that's what's gonna break our neck sooner or later. We strip systems of their redundancies to make them cost less, we use cheaper components and the lowest bidder, we downgrade specs to the bare minimum because price makes right.

Technology already starts learning from nature, copying structures and models from millenia proven concepts. I think business could learn a thing from them too. Because nature has down what business wants to achive: Maximum output for minimum input. There is no such thing as waste and surplus in nature's makeup, if there was, it would be used for more output instead. So why do we have 2 kidneys, why is our brain able to adapt to damage, if it wasn't for the simple fact that this proved to be the more successful way in the long run.

But as long as companies are run by managers who care more about their next quarter report than the company itself, this won't fly.

Re:It IS a house of cards

[TrippTDF]

Disclaimer: I am 27.

You do have a good point about things going out when you get older, but that's not nature's fault. Because if it were not for all our complicated medicines, we'd still have about a 45 year lifespan, and we'd be dead before a lot of those things started to happen. So Nature does have it worked out to the extent that it thinks we should be living... all the other things that happen stem from our ability to extend life through knowledge.

Re:It IS a house of cards

[Illserve]

So why do we have 2 kidneys, why is our brain able to adapt to damage, if it wasn't for the simple fact that this proved to be the more successful way in the long run.

Actually, evolution puts our design under the exact same pressures: do more with less. That is, in fact, the definition of natural selection.

Reliability and redundancy are just one aspect of our optimality, and it's not something that's always at the top of the priority list. One might counter your point about 2 kidneys by asking why we don't have two hearts. We sure as hell should have two hearts more than two kidneys. You can go a day or two without a kidney without even noticing, but 30 seconds without a heartbeat is a bad way to start your day.

I think our internet is about as redundant as we should like and that it doesn't seem likely that these various gloom and doom scenarios are going to play out. Yes there are periodic problems, but they are rarely systemic. Even under various forms of DOS attack and the horrendous inefficiency of spam, when I wake up in the morning, I can visit slashdot so long as my local connection is up.

And that's pretty impressive given the largely unstructured way in which the net has been thrown together. Let's not take what we've got for granted.

Re:It IS a house of cards

[Opportunist]

The human body wasn't meant to last for 80 years. That's way past warranty, and the menopause certainly stresses that. After you're no longer able to procreate, you are no longer "valuable" in the eyes of mommy nature.

Yeah, she's a real bitch. Never said she's nice.

We're actually more built for about 40-50 years, tops. The rest is human work. Also, we weren't meant to eat trans fat and spend our life in front of a TV inhaling the thermal waste of some plant. We were actually meant to spend it hunting and gathering.

Don't call it nature's bad design when you use said design off spec. If you use your screwdriver as a chisel, do you complain to the manufacturer when it breaks? It wasn't meant to be used as such.

Damnit man, I need details!

[Darth_brooks]

It'll be interesting to see just how much detail we get concerning what exactly failed and why the current level of redundancy didn't kick in.

I mean, it's not like the power supply failed on an NT 4 server (you know, the one with the post-it taped in place that says "East Coast B-berry server, DO NOT POWER OFF!!!"), it's not like somebody accidentally drove a nail trough some coax in the wall at RIM's HQ, it's not like somebody accidentally typed "rm -r *" at the wrong prompt. There has to be some serious "Thank God I'm not the one stuck cleaning up that mess" stuff going on here.

Funny unrelated story. We had an exec looking at one of the blackberry's. He put in back in the hard case and was fumbling around with it and saw the "RIM" on the back. Then he asked, in the innocence that only an exec can have, "So, how does one go about getting one of these 'RIM' jobs anyway?" When I am canonized as a saint, one of my miracles will be "not laughing at the VP who asked how to get a rim job."

For a communications company

[philo_enyce]

RIM is pretty bad at communicating with their end users. They do not have a network status page. When you call in to TSupport, there is no message indicating a general interruption in service. This is totally unacceptable. I spent two hours on hold last night after checking over a client's BES and not finding any issues locally. That hold time could have easily been reduced to a couple of minutes if RIM had a system in place to notify users of problems. Even Time Warner, which has terrible customer service, has service outage notices as part of the call tree when you call in for support. I would guess that this is a conscious choice on RIM's part, to try and appear as if they never have problems. Bad idea. Customer frustration sets in big time when you don't talk to them during outages.

RIM: Get better at communicating with your client base or they will go elsewhere.

philo

Re:Crap!

[somersault]

Regardless, I'm kind of thinking that they're a victim of their own success, and every time my users have been having delays I'm thinking that RIM are doing a poor job of keeping up with demand. It could just be problems with the local telcos of course, but here we see that the whole network has gone down, and I really am not surprised after the Blackberry issues we've had for the last couple of months.