Slashdot Mirror
Server Failure Destroys Sidekick Users' Backup Data
Expanding on the T-Mobile data loss mentioned in an update to an earlier story, reader stigmato writes "T-Mobile's popular Sidekick brand of devices and their users are facing a data loss crisis. According to the T-Mobile community forums, Microsoft/Danger has suffered a catastrophic server failure that has resulted in the loss of all personal data not stored on the phones. They are advising users not to turn off their phones, reset them or let the batteries die in them for fear of losing what data remains on the devices. Microsoft/Danger has stated that they cannot recover the data but are still trying. Already people are clamoring for a lawsuit. Should we continue to trust cloud computing content providers with our personal information? Perhaps they should have used ZFS or btrfs for their servers."
homemade cell phone porn videos cried out and then were silenced.
This seems a rather silly point to make. I know this is Slashdot and we have to suggest Open Source alternatives but throwing out random file systems as a suggestion to fix poor management and HARDWARE issues is some place between ignorant and silly.
Perhaps they should have had at least mirrored or stripped raid, with an off-site backup every week or so?
A server failure caused all of the data to be lost?
No backups? Not even a spare server with a mirror of the data? No servers in different places? No off-site backup strategy?
As an aside, why would that data be stored in volatile non-battery backed up ram? All of my graphing calculators have a special battery to keep the ram, and they aren't even supposed to store important stuff. Flash is cheap enough these days, why should simply removing the battery cause important data to be lost?
If I have nothing to hide, don't search me
I mean, having the data backed up in the net may be nice, but not having it stored on permanent memory in the phone IMHO is silly. Even without a server failure: What if you don't have net access, and your phone's battery gets empty? I also expect to be able to switch off my phone any time I want without data loss, no matter whether I currently have net access, and no matter whether I have actually changed and data since the last time having net access.
The Tao of math: The numbers you can count are not the real numbers.
So are we saying microsoft didn't have a backup? what about a offsite backup? Who wants to bet they were using their own backup solution? if they had a decent storage array they could have had snapshots and offsite replica's to restore from
shit, is that TSR still hanging around? goodness!
If the above means anything to you, "apt-get install joe mc" will make you smile as well.
~.~
I'm a peripheral visionary.
Either this is a really, really serious meltdown which completely killed not only the server but all their backups as well (and what're the chances of that?), or their IT guys have been really, really slack and just didn't make any backups...
Guess they should have used a better smartphone, like *anything* else on the market... Even the cloud-centric Pre will still work if you don't have access to the Cloud - even if Google and/or Palm dies, you'll still have all your information on your phone! Jesus... Doesn't inspire confidence...
'nuff said.
Microsoft/Danger has stated that they cannot recover the data but are still trying.
CC.
TaijiQuan (Huang, 5 loosenings)
This is an issue of irresponsibility. Plain and Simple. The company responsible for maintaining the data should -- at the very least -- have had some full system backup from last month. If they had some old backup somewhere at least you could chalk it up to systems failure or bad backup tape or bad admin or something.
But the fact that there is no backup anywhere indicates brazen negligence on the part of everyone responsible for the data. Everyone who had a part in designing the system and managing the system is culpable. The most ridiculous part of this is the over-reliance on server-side data storage by the sidekick designers.
Notes From Under *nix: blas.phemo.us
Because the entire Sidekick architecture is very client-serverish, not transparent as with ordinary phones (GPRS/EDGE/UMTS/etc. through a NAT to internet at large); the server is supposed to be responsible for all that data, and the phone is just caching it. Given that architecture, asking why the local copy is on volatile RAM is analogous to asking why your CPU doesn't have a battery backup for system RAM, or even L2 cache.
That's one of the big reasons I didn't go with a sidekick, even though they have (or had, last I was shopping around) basically the cheapest internet plans available; they push all sorts of stuff that's handled by the phone in any other system off to the Danger servers,. While that does expose you to other people losing your data, as seen here, I didn't even consider that. I just like having a direct internet pipe, so I can run whatever software I want locally.
That said, there are plain benefits to the Sidekick model, for some people. Basically, if you don't want to do funny stuff on your phone, and if you're no less incompetent than the MS/Danger sysadmins, it's better. After all, if you drop your sidekick in a toilet, run over it with a truck, and vaporise it with a plasgun, you can just get a new one and have all your data back -- which is good, since if you're 95% of people, you've _never_ backed up your phone's data. But it's not for me, and given your desire to have your phone work as a PDA even if you power-cycle it in a wilderness/cave/other net-less place, it's not for you either.
Right feature, wrong server? MS understands the need for a "Rose Mary Stretch" default setting.
The congress critters have learned a lot from the "terrible mistake" of email backups.
From cute page boys to Iran contra, MS can market this as a feature.
Domestic spying is now "Benign Information Gathering"
is one of the reasons I don't like cloud computing.
The other is that you need internet to get to your stuff. I've had very negative experiences with my previous ISP (sometimes two weeks without more than 5 minutes of internet) so now I don't trust anything that requires me to be online.
There are 3rd party apps out there that will let you "backup" your phone data yourself. I personally use a program called bitpim www.bitpim.org (make sure you d/l latest version). It works with many different phone models and I have used it several times to "restore" my phone data (had 2 phones with hardware issues). It restored my calendar, notes, phone book and rings tones (that last one can save you d/l $$$). It is easy enough to install and use, you do not have to be a total geek to make it functional (but having one available to help you set up backups would probably help). Been working in the IT industry too long to rely on someone else backing up my data for me, and I will not encourage Murphy to have a party in my honor!
This is unbelievably bad. The real problem is : why aren't there incremental off site backups to another server farm? A weekly binary difference snapshot would have made this failure less catastrophic.
Ultimately, with a complex application like this, you can't guarantee 100% that the code doesn't have a bug in it that could result in loss of user data. You can be ALMOST sure it won't, but 100% is not possible with current analysis techniques. (even a mathematical proof of correctness wouldn't protect you from a hacker)
But a properly done set of OFFLINE backups, stored on racks of tapes or hard disks in a separate physical facility : you can be pretty sure that data isn't going anywhere.
you gotta do it yourself. You're the only one who knows how valuable your data is and you're the one who will be affected by its loss. Backup your own damn data.
Now is the opportunity for opensource to show what it's good for. Someone whip together a small app to extract all info from the Sidekick, put it up on sourceforge for FREE and you have tons of goodwill for OSS. Of course, the app should be Linux-only, thus forcing all Sidekick users to install Ubuntu...
http://i.zdnet.com/blogs/dilbert_disaster_recovery_plan.jpg
Forget all the speculation and semi-random after-the-fact suggestions, I am waiting for the write-up to discover how this monumental cock-up occurred. I hope I don't just learn that 'backups would have been a good idea'.
AT&ROFLMAO
It's like being kicked in the side.
All your data are lost by us.
"We live in a global world" - Harvey Pitt, former Securities and Exchange Commission Chairman
Microsoft/Danger
"The hands that help are better far than lips that pray." - Robert Ingersoll (1833-1899)
Have ZFS/btrfs developed tools to undelete or rescue files? It is pretty hopeless for ext[234] in my experience.
NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
Reportedly sidekicks are thin clients, other than making phone calls, everything on the phone is saved on the server side. Which is a special kind of retarded
Isn't that also how Android works?
I mean sure, the apps and such are on internal flash, but it's a different story for your "important" data such as email or contacts list. Heck, as I've learned, one can't even read one's existing ("synced") email without a working web connection. How they can call that "syncing", and what it's doing besides simple header indexing, is beyond me.
This is another reason I am loath to trust "the cloud" -- if I know I can be self-sufficient (in a data accessibility context), that's going to be much better than storing things on a corporate server and hope that said corporation is not going to, um, fall from the sky.
"Good news, everyone!"
Netcraft says www.danger.com uses freeBSD+Apache+PHP http://uptime.netcraft.com/up/graph?site=www.danger.com
With all the competition in the smartphone market today, this is probably an unrecoverable error. If they manage to recover the data then they will come off as heroes for having the courage to tell their customers promptly. Otherwise they just look like they are: incompetent. No great loss, though.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Why trust something that's named "Danger" to begin with?
HOW THE HELL DO THEY NOT HAVE OFF-SITE TAPE BACKUPS????
So essentially, everybody's Sidekick backup data, which is apparently critical should they ever lose power, was all concentrated on A SINGLE SERVER? I hope they at least say their tape backups caught fire and their replicated server died on the same day too...
Their retentions lines are going to be hot this Columbus Day weekend! The iPhone is getting cheaper...
According to a very long article on AppleInsider:
http://www.appleinsider.com/articles/09/10/09/exclusive_pink_danger_leaks_from_microsofts_windows_phone.html&page=3
MS was misleading T-Mobile about the state of Sidekick support, and apparently charging hundreds of millions every year for, and I quote "a handful of people in Palo Alto managing some contractors in Romania, Ukraine, etc". This is apparently because most of the Sidekick devs had either moved to Pink or quit out of disgust.
"incremental..."weekly binary difference"
Uh, those would do nothing in this case, where it appears the entire DB has been lost. You need a regular full backup, or diffs and incrementals are just cruft. It appears they don't even have that, since there's no talk of restoring to month (or ?) old data.
"National Security is the chief cause of national insecurity." - Celine's First Law
He also hopes that you are not going to learn only now, that backups would have been a good idea.
You SHOULD have said, I hope THEY don't just learn, 'backups would have been a good idea.'
Your boss again, this is what your meant right?
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
Danger?
MS's branding is actually a pleasant twist in advertising. Instead of spitting in our faces and telling us it's raining, MS now has the ballmers to tell it like it is.
Interesting article about the Microsoft/Pink/Danger/Sidekick relationship and leaks indicating that Microsoft are trying to kill Sidekick without telling the partners. Microsoft would never do such a thing of course ...
Rich.
libguestfs - tools for accessing and modifying virtual machine disk images
It is development dome.
Two companies enter, MS comes out, slightly fatter.
If you do business with MS, you are riding a tiger with the brains to realize that lunch is only a roll on the ground away.
MS really should be renamed to BubbaSoft. Get into the shower with BubbaSoft and you know what is going to happen.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
Haha!
Divide a cake by zero. Is it still a cake?
There's a reason why they call it "Danger"...
After all, if you drop your sidekick in a toilet, run over it with a truck, and vaporise it with a plasgun, you can just get a new one and have all your data back -- which is good, since if you're 95% of people, you've _never_ backed up your phone's data.
With services like MobileMe for the iPhone, that's no longer a unique advantage.
HOW THE HELL DO THE CUSTOMERS NOT TAKE BACKUPS THEMSELVES
If the data is important, then the only way to make sure it doesn't get lost is to take some responsibility for your actions and do back-ups yourself. If your service provider does not let you then don't use the service, it's that simple.
I never even heard of the sidekick, until a story in the news about one belonging to Paris Hilton being hacked, and all her personal information being spread on the net. Damn I thought, I'm never buying one of those pieces of shit. Everyone else said WOW check out Paris Hiltons fabulous new inexpensive texting device targeted at the 12-14 teen market, Lets all get one of those! My data is insecure? my data is unsafe? my data is stored out side my phone? Who cares, Paris Hiltons got one, she knows everything about informaton technology and mobile telecommunications after all.
A) The Sidekick apparently doesn't store anything, so customers can't make backups that easily, even if they wanted to, and
B) Danger designed this phone to store everything server-side. It is incomprehensibly foolish to not include a SUPER SOLID backup strategy as well. This problem has been ongoing for several days now; I don't know if the data was fine on the onset of this problem, but the infuriated customers have all the right to demand everything AND the kitchen sink for losing practically everything they had.
.. might be a fantastic file system, but it is not ready for production use just yet. Please don't deploy it on your servers.
Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.
Suddenly,
There's not half the files there used to be,
And there's a milestone hanging over me
The system crashed so suddenly.
I pushed something wrong
What it was I could not say.
Now all my data's gone and I long for yesterday-ay-ay-ay.
Yesterday,
Need for backup seemed so far away.
Seemed my data were all here to stay,
Now I believe in yesterday.
Anonymous
This is exactly my point. The irresponsibility of epic proportions is on the side of the users who didn't realise that the design of the system invites them to be fucked over by a data loss incident. They have no one to blame but themselves.
...as everyone neurotically signs into their servers to verify the status of their backup jobs and MD5 checks... only a few will go the distance to try uncompressing/unencrypting their archive files and browse around inside the restore area randomly checking files to ensure they look good.. what? you're doing backups on a machine without ECC RAM and writing the image files across a USB to a consumer-grade hard-drive or burner without checking the MD5s --and on that type of hardware you're doing compression/encryption without turning around and automatically decrypting/decompressing and byte-verifying the files against the backup source BEFORE calculating the MD5 on the image file? (And you can trust that MD5 calculation because of why?) Oh yeah, see how much you are "saving" by repurposing old non-ECC equipped hardware to run your backups or buying consumer-grade SANs from Fry's which use whoknowswhat? All those backups they ever created may have some serious problems just waiting for you to discover at the worst possible time...
Cloud computing?
That ain't no cloud. That's the fog obscuring the view of sanity.
IT has been trying this crap ever since the emergence of personal computers.
Fata viam invenient.
The title is a reference to this being an old fucking troll from the start of this year.
Apparently, it's new to you. Don't feed the trolls, dumbass.
Clerk: Danger Powers personal effects [shows box of off-site tapes and such]
MS: Actually my name is Microsoft Powers...
Clerk: It says here - name: Danger Powers
MS: No no no no no... Danger is my middle name
Clerk: Okay, Microsoft Danger Powers...
df -h
That we gather. However the weakness isn't the model but who's doing the serving. Why can't the Sidekick like the Blackberry use an independent server?
So is Ballmer tossing chairs about?? I think not. Probably sitting back with a smile on his face.
Conservative, mod down for violating
I believe that the WTC towers were collapsed by detonation, for example
Ah, ignorance at it's finest...
http://www.debunking911.com/index.html
Lesbian Nazi Hookers Abducted by UFOs and Forced Into Weight Loss Programs - -all next week on Town Talk.
I work in telecom at a different provider. SAN upgrades are performed by the SAN vendor and, IME, they always demand a complete backup prior to starting any work unless the customer demands otherwise. If the customer doesn't want the backup, we always had to get a Sr VP to sign off. There were about 10 Sr VPs in the company - not like at a bank where everyone is a VP.
Usually, we would perform firmware upgrades only when migrating from old SAN equipment into new. The old equipment would be upgraded and used to upgrade either lower performing SAN or directly attached disk arrays that had been neglected for 5+ years. Being out of warranty was avoided. Most data is too important to risk that.
BTW, we measured storage in petabytes and our storage team was **never** on the cutting edge. We were always 2+ years behind other BIG companies. Our labs may have this quarters' latest and greatest, but it would take years to get from the lab into production service. That drove some vendors nuts, but not the "names you know."
I saw where someone above said they randomly verified recovery quarterly. What a joke. On my systems (Sr Tech Arch), we deployed with redundant systems at least 500 miles apart. Many systems did have instant fail over, but if instant fail over was not possible due to the amount of data, **never** would we lose more than 24 hours worth of data. Between, RAID-10, near disk backups, tape backups, remote replication and backups at the alternate location, we had the data. Further, to verify the alternate system worked, we swapped primary production locations every week. I and my internal customer slept very well, thank you.
I have a good friend who works at T-Mobile in their architecture design team. It will be interesting to see whether this subcontractor had anything to do with the issues. I called T-Mobile for an unrelated personal item on Tuesday, they were already swamped with calls and said that a sub to Microsoft was working the issue. I'm thinking MS outsourced/bought the provider and the garage shop team was still running things - but I don't know. I do know that Microsoft has excellent engineers for systems like this and they are more cautious than google with their upgrades and deployed systems. Over the years, I've had to deploy a few Windows-Server-based solutions - usually for voice response systems. I was never really happy doing it. I don't trust backup systems much unless it is really a mirror that I can get to 1 file from 3 weeks ago easily.
Ok, back to upgrading the company email servers. A system version upgrade will impact users for less than 10 minutes - probably under 3 minutes, but we like to under promise and over deliver.
Sadly it comes to pass that every generation the Tao of Backup is forgotten and must be relearned through such trial by fire. http://www.taobackup.com/
With mysql.
Weird file I/O problems and file system corruption. Went to backup & tried to restore.
Restore failed - more filesystem corruption.
previous backup - restore failed and corrupted the filesystem yet again...
repeat many times etc etc
eventually a very old backup restored successfully, but there was data loss.
The bug was in the odirect handling within xfs filesystem under several linux kernel versions. The DB server would trigger it under high I/O, and the restore would also trigger it as well.
These will be big systems and i'll bet they are hitting a software bug on their primary and secondary (running the same software) systems.
Now I run an automated dump & restore. The secondary servers are always running yesterday's backup.
Ahh. Last week I picked out a new phone for my T-Mobile service. Sidekicks were offered. When I looked up the details and saw it ran an MS OS I moved on.
It is amazing how often not choosing Microsoft pays off.
"Epic fail" doesn't begin to describe this one.
Just wanted to let you know that the patent you mention in your signature has been cancelled. (warning, your toddlers might violate a patent)
See link below, scroll to the end:
http://www.google.com/patents?id=T2QKAAAAEBAJ&printsec=abstract&zoom=4&source=gbs_overview_r&cad=0#v=onepage&q=&f=false
Nature journal lied in Britannica vs Wikipedia Ask to retrac
Not a troll, I swear. Just someone interested in the works of Steven Levitt...
I've read many racially slanted comments elsewhere that seem to indicate a general opinion that the majority of Sidekick users are african american or hispanic. Someone responded at one point that there would be a huge drop in the number of drug deals that occur because everybody will lose their dealers number.
Now, the nerd in me started wondering...what if there actually was a connection!? Or not even something drug-related...could there be other connections that occur as a result of this data failure? Just food for thought...
Buy Steampunk Clothing Online!
Looking at the timeframe that Danger was acquired by MSFT and that the Danger OS was likely based on NetBSD (http://en.wikipedia.org/wiki/Danger_Hiptop), it's more likely that Danger was still using NetBSD as their Server Software and this was merely a process issue. Blaming it on the "Microsoft Platform" without any real data is just spreading FUD.
Microsoft/Danger
The second part is kind of redundant / a synonym, isn't it?
It's amazing how many times the name "Microsoft" and the words "catastrophic failure" end up in the same headline.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
I'll admit to having one of the original (and second version) of the Sidekick (They were called the Hiptop everywhere else except the USA) and the idea of storing everything on the cloud seemed great at the time - through several device upgrades, warranty replacements, and other hardware changes everything just automagically restored to the new phone within 10-15 minutes of switching the SIM.
One should add that the devices themselves are designed to "Play dead" when the battery gets low and shut down while still maintaining enough power to ensure the volatile ram holding the devices local cache of data remains intact. It's only if the battery is fully exhausted to the point of not being able to accomplish this, or a critical error/OS crash (The dreaded "red X of death") is encountered is the volatile ram actually in danger of being erased.
Therefore all the warnings about not letting the phones go "dead" or turning them off are a bit misleading since, excluding one of the two above situations everything is actually safe, but it's not without warrant since I'm sure MS/Danger are going to try to "backwards restore" whatever is salvageable.
Furthermore, since the OS is locked down extremely tight there's no (to my understanding, admittedly a few years old now) method of locally backing up a Sidekicks data. Contacts stored on the device can be backed up to the SIM card one at a time (with only the basic name/phone data, all other extraneous data such as profile pics, etc will not be included) but it was tedious to accomplish (one contact at a time) and the average Sidekick user (read as teen/clueless) probably has no idea how to do it anyways.
...since if you're 95% of people, you've _never_ backed up your phone's data.
Unless of course, you're an iPhone user, in which case your data is backed up every time you plug in your phone into your computer to recharge ;-)
All that data was apparently in one data center... Not nearly diffuse enough to be called "cloud computing". Google's hundreds of distrubuted, redunant data centers... that's a cloud.
As MS is involved there is a good chance that Sidekick users can either recover the their data from Google cache or buy a backup copy from an underground/crackers/warez forum of their choice.
my guess, the crypto key pass phrase was lost
as in for PCI/HIPPA compliance something was encrypting data. When this server crashed, no one knew the pass phrase to start accessing the data again.
my $0.02
Microsofts products are repeatedly proven to be terrible, their services are repeatedly proven to be even worse. Yet still people pay inflated prices and put them in mission-critical places. Even in preference to better and more robust alternatives, including many that are free, they still choose Microsoft for some crazy reason.
When is this cloud of stupity going to lift from Microsoft users eyes?
What if you are a conference, lose your phone, drop it into the punch bowl, have it fall out of your pocket and run over by a taxi. Having some sort of remote backup seems a good idea.
Yes of course which is why most modern systems backup to your computer or the cloud or what have you - but ALSO have local permanent storage.
A sign of the wrongness of the approach is that if your sidekick battery dies the data is gone. That would not be true of any modern smartphone at this point, your data would be held in flash as well as whatever servers or other computers it might have backed up to.
Furthermore if Danger didn't offer a viable means of letting people back up data to their own computers, that's another failure too...
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Cloud computing?
That ain't no cloud. That's the fog obscuring the view of sanity.
+1 Insightful.
Microsoft named one of its products "Danger" and no one realized this might actually be a warning?
love the taste, hate the texture
that backups where not done, and no process existed to test the backups. What I would like to know, is what does Microsoft gain or T-Mobile gain by not having to do infrastructure improvements or upgrades if all of the data is suddenly lost.
I think it would prove interesting to know, if someone made the decision to destroy the data, and save potentially millions in infrastructure upgrades.
I find the whole thing rather improbable as many have pointed out here, that backups suddenly failed, or even the off site backups are bad.
I think it is more likely, they (T-Mobile) decided not to do infrastructure improvements and to dump the data due to the fact the management cannot compete in the mobile space, in order to show a profit at the end of the year.
-Hack
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
To the standby or testing system. Our staging/testing systems all run yesterday's production data, restored from the most recent backup.
if your backups don't work then neither will your test/staging server... Which will be noticed.
What do you get?
* Backups tested every day.
* A test/staging/standby system identical to the production.
* Something the business can run all the crappy queries they like against without affecting the production system.
Deleted
Hell no! You never should have in the first place.
shit, is that TSR still hanging around? goodness!
Dude, what part of "Stay Resident" did you not understand. It's not like selling your computer rids you of it.
That's why I never ran them, nor consorted with Deamons.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
At least Microsoft has finally decided to make the end merciful and swift for companies they acquire, instead of slowly suffocating them over thousands of years like a Sarlacc. We should rejoice!
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Sidekick to the balls, that is!
This one is Chuck Norris level.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Granted, this isn't cheap, but our data isn't either.
Microsoft bought Danger for half a billion dollars. Current estimates of the value of this data are roughly... half a billion dollars, plus a little. There's little doubt that in addition to destroying the entire value of the acquisition they've created a connection between "Microsoft", "Danger" and "data loss". In their release T-Mobile isn't being shy about tying those things together. Not good. That's going to have impacts even for some completely unrelated cloud-based products like Azure.
Somebody's about to get a really awkward performance review.
Help stamp out iliturcy.
Anyone else see this is as a way for MS to get WinMo 6.5 phones (which by all accounts are tanking) into more hands?
"We're sorry we hosed your data, but to attempt to make amends we'll give you/discount you this brand new fancy UI Windows Phone with our apologies."
This has everything to do business planning of funding and timely rollout of redundancy and backup systems (including staff). The tech, tools, good IT staff hires, procedures, and strategies are out there and are pretty well known. I don't have to know about their infrastructure or staffing to tell you that they didn't invest in the infrastructure and/or training for staff to prevent exactly this sort of disaster.
I can think of three possible reasons why they didn't have such an infrastructure in place:
1) When M$ bought Danger, they scaled back to maximize profits
2) The risk/cost analysis (odds of failure, cost, and cost to prevent) of such a disaster was out of date, incorrect, or simply accepted
3) Funding was available but rollout of redundancy/backup was taking longer than expected
I doubt #3 since Danger has been around for a while and their customer base probably isn't growing exceedingly quickly. #2 is mildly possible, IMHO, if they simply accepted the risk. But, a class-action after a disaster like this has to be really expensive unless they think they can dodge the legal obligation of backing up user data. #1 is fairly typical of take overs, especially if profit is more important than safeguarding user data.
We see a house's front door from the inside of the house. Doorbell rings. Angry-looking guy holding a useless Sidekick walks up, opens door and sees Catherine Zeta-Jones standing there.
"Hi," she says, "I'm here to apologize for what happened to with your Sidekick," then promptly drops out of frame and you hear the sound of a zipper being unzipped...
Cut to close-up of the guy's face transforming from anger to a big, relaxed smile.
Fade to T-Mobile logo.
Surely they didn't upgrade the SAN and the offsite backup at the same time?
Surely they had an offsite backup?
Right?
tl;dr
Sidekicks have very rudimentary backup features. You can save phone numbers and names to a SIM card and put contacts, bookmarks, notes and pictures into emails and send them off to yourself. That's about it. The latest version of the sidekick does not have a file manager to aid in backing anything up.
The system is most definitely a thin client architecture...and I believe would have been properly managed..and continued to work..for years had M$ kept their greedy, incompetent fingers out of it.
I don't see how a filesystem woukd do anything when, from all reports, they fucked up a SAN upgrade and didn't have backups.
Oh, my hard drive just died, and I can't access any data on it any more. Maybe I should have used ZFS? I don't think so...
Specialist Mac support for creative pros, Melbourne
It's a feature. Lose your phone? The guy with the phone gets nothing, and you get everything back in seconds. And it's protected on our super-servers, so you never have to worry about loss. They weren't stupid, just trusting. Though from your tone, you apparently think nice trusting people deserve to be screwed.
Learn to love Alaska
My 9 year old daughter has a sidekick.
Microsoft has made her cry.
I suppose the question should be asked at the architectural level. Why in the world doesn't the phone store the data locally and simply replicate it to the server. That gives 100% of the benefits should the phone be damaged but also makes sure it's all there if it has no connectivity or if, for example, a sysadmin disaster wipes the server out.
If that was done, recovering from this problem would be just a matter of pushing everything on the phone back to the server. It might have even been possible to do it without anyone having to know the server died.
I've heard and read of enough problems with restoring complex transactional data structures that I can imagine this situation is far more complex than many believe.
What I'd love to see is a full post-mortem, a lesser version of that the NTSB does for airplane crashes.
Google's been doing some of this for their (too frequent) outages, but they've very high level -- typically something about a system reconfig overloading a router. The Cloud user base needs a far greater level of error exposure.
It took openness and in depth analysis to make air travel safe.
The Cloud won't be safe until we learn the same kind of lessons, and apply what we learn in new and improved systems.
John Faughnan
jfaughnan@spamcop.net
Teeth will be gnashing, and the lawsuits will be flying.
Its a shame (but typical) that in wanting to kill off 1 particular piece of equipment that they don't like (the non MS OS SideKick) that MicroSoft gets to give a whole swath of tech like cloud computing a bad name. OTOH, maybe this will serve as a heads up to most people that cloud computing is years away from really dependable viability.
Suppose you were an idiot. And suppose you were a member of congress. But then I repeat myself. -- Mark Twain
And using the cable that came with the phone (or aftermarket) to backup the phone's data ??? I did it on my V360, and I still do it on my current phone (sue me, it's an iPhone, but at least it's jailbroken) It's so simple to backup the device on one's computer... If it's using WinMo, Active Sync will be able to backup, right?
I've got better things to do tonight than die.
The problem is the cell phone cartel rigs all the phones to be either difficult or impossible for customers to copy their own data. The royals make sure everyone is dependent upon them, and if they screw up or just decide to throw your stuff out, you cannot do much about it. Classic type of problem when doing business with psychopaths...
Somebody's about to get a really awkward performance review.
Apparently the reviewee's name is Roz Ho. We should all send her flowers. Thanks, Roz. Write if you find work.
Help stamp out iliturcy.
Backups and Disaster Recovery:
Two problems exist: Did the backup happen and is it limited to just one big tape ??
Has anyone restored from backups ? Application, Data, Personality ???
Backups need to be restored someplace else - no point otherwise. What if one site
has been hit ?
One tape backups are better than epoch and incrementals in general.
Restore issues are really things you need to address.
For example: I left an organization once and got called back by a friend when they had
trouble. I wanted to change things so I checked the backups PRIOR to making changes.
Oracle DB backup would have been impossible after the new system admin changed
backups I had written. I rang my old boss who was not pleased.
Backups taking a long time? Have you tried to recover any files ?? At one place
the backups create an index but it takes so long for backups that the restore window
aborts - just looking through the index. SIGH!!!!! The only time you can recover a file
would be after a disaster or if the file you want is found early in the index file.
Backups suddenly taking a lot less time or using up less tapes ?? Check the backup
procedure. I discovered a backup once where 6 sets of 5 tapes were in play.
However the logs indicated that for the last 5 sets the tapes appeared to fit onto only
3 tapes. I identified the problem - a new operator doing the wrong thing and the backup
software allowing it to happen. I ported a similar bit of backup code and fixed the speed
problem as well: 6 hours down to 2 hours 10 minutes.
SAN backups work well but need testing: have you backed up every machine you need ?
I wrote about 2700 lines of code once: I started backing up just one machine a WEB
system. It needed a DNS server and a Certificate server so I had to backup two other
machines. It can be a problem ? This WEB server had several WEB pages - I checked
them - Some of the applications actually were off-system references - so they needed
to be up. I created a BASELINE after I asked for unique WEB page titles - just to sort
this out.
Backup limitations need to be known: some programs that claim to backup can have
bugs or features that prevent a decent backup. I discovered a backup of a filesystem
that should have backed up every file in say "/home" but because MVFS was running
- it failed. You need to stop or quieten filesystem operations for some things and use
another different method of backup as well. Its a tough decision.
Filesystems: Sparce filesystems, Large files ?, Access control lists, inode info ??
All of these need to be considered !!
If it's using WinMo, Active Sync will be able to backup, right?
I went back to PalmOS after some bad experiences with Pocket PC, including the fact that ActiveSync does NOT make a useful backup of a PDA or phone.
* It doesn't back up applications. It does keep a copy of applications when you install them through ActiveSync, but not if you install them via a CAB file on the device. And that copy is only usable until you replace or reinstall your PC because it depends on a maze of twisty little registry keys.
* It doesn't back up application data that it didn't install. If your applications create data, they have to have software on your PC to back it up.
* It doesn't back up files you install yourself. Depending on how you install them, it may keep a copy of the file you installed, but don't expect it to keep any changes made on the device.
That's why by 2002 every Windows CE "partner" was including a little program to backup to flash card, because ActiveSync doesn't do it for you.
I think even if sidekick data were housed on geographically away sysplex IBM Z/OS Z10 mainframes, it would be lost somehow with such management. Host OS and filesystem is irrelevant.
Run the most basic backup application, e.g. the one MS gives for free with Windows, you will see it backs up and VALIDATES entire data. I haven't seen a single application, command designed for backup doesn't do it.
How's that ignorance? Did I say who did it? Did I get onto conspiracy mode?
Look kid. Welcome to the real world. Taped, archived and accesable interviews with eyewitnesses (fireman, and alike) from cable tv networks all around the world said that they heared explosions from underground when they where inside the building. That's court grade evidence piece number 1. Taped, archived and accesable interviews with a head of a German universaty said in an interview that they researched the 'soil'(sorry I am not a native English speaker) from the WTC and found an alarming number of phosphor. Evidence number 2. NASA posted infra red satalite images from days after the collapse showing alarming heat still comming from the WTC site. Evidence number 3. Pictures of the days after showing 21 meter long steel bars from the core structure cut in a pricise way show evidence from a demolition. Evidence piece number 4. Then we've got confirmed, on public-, cable television by the head of the WTC 7 tower that tower 7 was collapsed by a demolition. Evidence piece number 5. Want me to go on?
Notice that this is not theory, but actual court grade evidence? I don't know who are responsible for this. I don't know what the hell was going on. All I know is that this is not speculation and it's comming from highly credible sources and not John Doe dragon fighter xXx p0rn l33t 0wN3r's, AOL homepage. Kindergarten is someplace else.
Here be signatures
Continue to trust?
After learning this was likely caused by a failed single SAN upgrade by Hitachi, I have to think that the architecture built to support the Sidekick didn't have an adequate budget to be built right.
Budgets ultimately decide what we techs/admins get to work with. We can always ask for what we want. But someone else (procurement, finance, project management, architect) can shoot it down, resulting in Plan B. And in most cases the person(s) signing/approving the final purchase order hasn't got a clue. By the time a failure occurs, the parties responsible for the system in place have long gone to their next position to screw up.
How's that ignorance?
It's ignorance since you obviously have no clue how things happened in reality.
Did I say who did it? Did I get onto conspiracy mode?
Yes you did. Since the official record differs from your assumption, it would mean that you think that there is a conspiracy at work.
Look kid. Welcome to the real world. Taped, archived and accesable interviews with eyewitnesses (fireman, and alike) from cable tv networks all around the world said that they heared explosions from underground when they where inside the building.
Well grandpa, many of those interviews were taken out of context or they were made by people who had no idea what's going on. Just because some panickinggue tells that "OMG, I heard explosions!" does not mean that there were actual explosions.
phosphor
Is this the "there were traces of thermite!"-argument? That's debunked here:
http://www.911myths.com/html/traces_of_thermate_at_the_wtc.html
and here:
http://www.debunking911.com/thermite.htm
Evidence number 2. NASA posted infra red satalite images from days after the collapse showing alarming heat still comming from the WTC site.
And what does that prove? That there were fires among the rubble? Surely not!
Evidence number 3. Pictures of the days after showing 21 meter long steel bars from the core structure cut in a pricise way show evidence from a demolition.
Debunked here:
http://www.debunking911.com/thermite.htm
The cuts were done by the rescue-workers AFTER the collapse...
Evidence piece number 4. Then we've got confirmed, on public-, cable television by the head of the WTC 7 tower that tower 7 was collapsed by a demolition.
Debunked here:
http://www.debunking911.com/pull.htm
Evidence piece number 5. Want me to go on?
Please do, I love laughing at your retarded arguments.
Notice that this is not theory, but actual court grade evidence?
None of the things you listed would pass as "evidence".
All I know is that this is not speculation and it's comming from highly credible sources and not John Doe dragon fighter xXx p0rn l33t 0wN3r's, AOL homepage. Kindergarten is someplace else.
"Highly credible sources" indeed.
Lesbian Nazi Hookers Abducted by UFOs and Forced Into Weight Loss Programs - -all next week on Town Talk.