Marriott IT Exec Shares Network Horror Story

alphadogg writes "Neil Schubert is only partly kidding when he calls Marriott International's move toward a converged network a horror story. 'I'm here to tell you a terrifying tale of network design, support and administration,' he said at an IT conference in Boston, referring to a major bandwidth crunch caused by guests wielding Slingboxes and other network devices that overran the hotel chain's outdated network. 'One of the things we've learned about our guest networks is we have one of the most foreign, hostile environments known to man in the network administration world ... I can take 100,000 customers a night on that infrastructure and we actually have less incidents of harm than we do on our corporate back-office infrastructure.'"

Comments

Re:Where's the horror?

[dwater]

Of course that's not an accurate description. It was that he was not prepared for them to use it to the extent they did - he wasn't prepared for the degree of success that occurred.

On that note, I wonder what turns a healthy network into a broken one? Is there something different between a broken network and one that's just very slow?

It reminds me of the problems we had in Asia (I'm in Beijing) earlier this year due to the earthquake in Taiwan. Network congestion was so bad that we figured using a 56K modem would be faster, since at least then we're given a dedicated bandwidth (enough for a voice)...maybe. We never tried that out, so I don't know for sure...

Anybody have a "buzzword bingo" card for this?

[MonkeyBoyo]

I may have understood some of the article but it seems to have been mainly an exercise in trotting out what somebody thinks are the most trendy buzzwords.

Re:Unrealistic convergence plan

[Beatlebum]

>> Work with Slingbox to engineer channel aggregation with multicasting that bypasses the home units while transmitting the same programming

Oh sure, Slingbox is going to switch from being P2P to a multicast provider of copyrighted content in order to let Marriott go cheap on bandwidth.

>> Can an in-house video alternative be made as attractive as Slingbox? That's another solution.

This is non-trivial and does not solve the bandwidth problem completely.

Along the same lines I have a solution:

Cache the Internet on a local server and connected each room via a quad fiber ATM connection.

Re:One lesson from the article...

[Anonymous Coward]

i've met many CEO's in my time. without fail, they have all been weasles with no clue how their business runs. even the ones that founded it.

i currently work for a multi billion $ venture and the CEO is required to sign off on outside contractors HOTEL STAYS. i shit you not. he will usually approve anything you slap infront of him, but sometimes he must get this urge to interfer and ask a bunch of fucking stupid questions and attempt to derail it.

Microsoft update + Public Network = Instant DOS

[James_G]

While I was at the 2005 MysqlUC in Santa Clara, Microsoft put out one of their massive patch releases. Despite a large number of people running OS X or some Linux variant, there were still enough geeks in the hotel running Windows that they essentially DoS'ed the hotel's internet connection for about 2 days. I went down to reception at the time to find out if it was just the wireless, but the front desk people were similarly frustrated (they shared the same connection). Packet loss was at about 99%.

That was probably just a T1 or something, but still, pretty funny. I wonder if Microsoft realises the damage potential..

Re:Microsoft update + Public Network = Instant DOS

[Nezer]

What kind of fucking idiot updates their laptop during a conference? You wait to do that shit until you get back home in case it screws your machine.

Let's see... At a conference your computer is connected to hostile networks nearly all the time. Depending on the conference, there are potentially a LOT of people that know about 0-day exploits and might want to try something dumb.

I dunno. I can see your argument but there may be very good reasons to patch your system ASAP. I used to work in an environment where NOTHING got patched because they were afraid of fucking-up production services. I argued until I was blue in the face that we needed to do something and have a plan for deploying patches. I even went so far as to make proposals explaining the benefits, the risks, and the costs. No one would listen to me because I was a UNIX admin on a Windows team. Eventually I was let go and no one else took-up my cause (perhaps the cause was a large reason I was let go). No one on the team, except me, felt that there was any risk because the networks were "isolated" behind three layers of firewalls. About three months after I left some nasty work managed to find it's way into this "isolated" network and wreak much more havoc than we ever could have patching the damned servers.

I know that this isn't exactly the same thing as updating your laptop while on the road, but sometimes the updates are just worth the risk.

Perhaps the hotels should consider a caching proxy for just these sorts of events. Let the first user wait for the the download to come down the pipe and everyone else can leach from the proxy.

Ironically...

[EReidJ]

With some irony, I am reading this story from a Marriott hotel room at the Marriott Boca Raton. I've had mostly no problem with their services, but here are a few things I would call interesting: * My laptop can often see multiple nodes, some very fast, some blazingly slow. If you stay in a Marriott, try out the different nodes you can see. * Some Marriott properties give free Internet access, some cost $10/day. I wish Marriott would be consistent across all their properties. * If you stay in a Marriott that does charge for access, as for a low floor. This is because often there's a single wireless connection in the business center that is free, but other access points cost money. So if you can get a room near the business center, you'll be able to hook up to that one for free. * Finally, I've never had a problem with BitTorrent uploading at any Marriott property. I don't know if they leave all their ports open or what, but I traditionally leave my uploading port for BitTorrent open on 34567, and I've never had a problem with a torrent at a Marriott.

Re:overbooking/non-refundable

[evought]

That the reason they do it is because statistically a percentage of people don't make a given flight. Sometimes it ends up in conflicts, especially since they tend to err on the side of being full rather than no conflicts, but there is good reason to do it. If 100% of people who wanted on a flight showed up, they'd never overbook. However about 10-15% of people cancel their reservations or otherwise fail to show. Thus it makes sense to overbook the aircraft.

That might almost be an excuse except that they sell non-refundable "you die, you fly" tickets, supposedly for the exact same reason. Those empty seats are already paid for. They are trying to make additional money off of them at the cost of double booking. Like most businesses, they get you both ways and make you deal with the mistakes and inconvenience.