Marriott IT Exec Shares Network Horror Story

alphadogg writes "Neil Schubert is only partly kidding when he calls Marriott International's move toward a converged network a horror story. 'I'm here to tell you a terrifying tale of network design, support and administration,' he said at an IT conference in Boston, referring to a major bandwidth crunch caused by guests wielding Slingboxes and other network devices that overran the hotel chain's outdated network. 'One of the things we've learned about our guest networks is we have one of the most foreign, hostile environments known to man in the network administration world ... I can take 100,000 customers a night on that infrastructure and we actually have less incidents of harm than we do on our corporate back-office infrastructure.'"

Comments

Re:Where's the horror?

[luvirini]

It is a "horror story" because the network manager was not prepared for the customers to actually USE the service...

Come on!

[Anonymous Coward]

For running this slashvertisement, Zonk's getting one years free accommodation from the Marriott chain, so what's the problem?

Can't an "Editor" graft in peace?

hurm?

[spazmonkey]

So what was the point of that article again? I must have missed it. Perhaps the PR flak who subbed it could explain it to me. I want that two minutes of my life back now /.

Re:hurm?

[CrazyJim1]

And here I was hoping the comments would be exciting to make up for my boredom reading that article.

Re:hurm?

[WarlockD]

Agreed. Whats the freaking point? The article is a fluff piece as it doesn't even describe what, if any, the problems they had to overcome. I rather have it explain WHY the hotel customer network was safer than there internal network than just it be said:P

Marriott a sponsor of Slashdot?

Re:hurm?

[Midnight Thunder]

One of the points was that their current network implementation sucked. They have multiple networks, each handling a different type of medium, be it telephone, computer data and TV. This makes for huge complexity when administering the networks. Their current data network was not designed in an optimal manner and quickly got over run. Their plan is to provide one IP network for all data types (voice, fax, data and TV), instead of dedicated networks, and then have a box in each room which would provide an interface to allow to use phone, computers and TVs. This approach reduces cabling and makes it much easier to manage.

BTW The article could have been written better.

weilding slingboxes?

[ChiChiCuervo]

I just get this mental image I'm not going to be able to shake....

"Some call it a slingbox, I call it appleTV. nnnngggggghhhh"

Re:Where's the horror?

[dwater]

Of course that's not an accurate description. It was that he was not prepared for them to use it to the extent they did - he wasn't prepared for the degree of success that occurred.

On that note, I wonder what turns a healthy network into a broken one? Is there something different between a broken network and one that's just very slow?

It reminds me of the problems we had in Asia (I'm in Beijing) earlier this year due to the earthquake in Taiwan. Network congestion was so bad that we figured using a 56K modem would be faster, since at least then we're given a dedicated bandwidth (enough for a voice)...maybe. We never tried that out, so I don't know for sure...

One lesson from the article...

[BrowserCapsGuy]

There needs to be better coordination between marketing and IT. IT had no idea marketing was running commercials showing customers using all this high-bandwidth stuff so there's no way IT could be prepared for it. Imagine 160 customers just trying to view websites on one DSL line! I admire this guy for his honesty if nothing else. He'll probably catch hell for it from his superiors!

Re:One lesson from the article...

[timmarhy]

that's what marketing do dumby! they make outrageous claims then handball it to technical, and when technical can't make the impossible happen marketing make it look like the techies failed. marketing will then tweak their bullshit slightly to cover their own ass's and make it look like they saved the day.

Re:One lesson from the article...

[Kjella]

that's what marketing do dumby! they make outrageous claims then handball it to technical, and when technical can't make the impossible happen marketing make it look like the techies failed. marketing will then tweak their bullshit slightly to cover their own ass's and make it look like they saved the day

Marketing aren't half bad, they haven't got a clue what they're talking about and answer "yes" to most questions when they have no idea or not. Most have a fairly relaxed attitude to that and know that their job is to reel them in close so negotiations can start. They're great with powerpoints and talking points and all that, but noone would mistake them from a tech and particularly not a tech genius. That's why they usually only able to do limited damange, because the customers also want a more technical presentation.

The real bastards work in presales. The presales guys are very bright and very much so techs, but they have joined the dark side. Their domain are one of mock-ups, sales demonstrations and in no small part smoke and mirrors. They deal in hardcoded values and links, functionality that doesn't exist or doesn't work and graciously ignores all the difficulties of actually doing it. They're the people who *know* it won't work and still tell the customer they can do it and the customer will believe them. They're experts at closing sales, to then hand it over to an implementation team because clearly they're too important to actually do the impossible.

I actually have a better relationship to people that are a bit wishy-washy and hand-waving than those who know exactly what they're talking about - then lie about it.

This is more typical than horrifying to me...

[kungfoolery]

The business units of most organizations typically make promises to their customers without comprehending or even considering the IT implications. Account Executive to customer: "Sure! We can provide you and your thousands of users seamless B2B connections from your network to ours wirelessly from any global location!" Account Executive to IT department: "Ok, you guys figure out how to do that."

Re:This is more typical than horrifying to me...

[jd]

Yep, seen that at plenty of places. At one place, the IT department clearly spelled out what was possible and in what timeframe. Marketing oversold anyway. However, this isn't just IT, and that's the scary part of it. Airlines routinely overbook aircraft. Package holiday companies sell hotel rooms for hotels that haven't yet been built. You too can place advance orders for books that haven't been written or buy computers that have not yet been built. (One company I know has made substantial money off a computer for which even the prototype does not yet work.)

The problem with marketing is that it is not about selling what you have, it's about selling what the person wants to buy. If there's a discrepancy between the two, well, that's not your department. Complaints is three doors down, across the hall from Abuse.

Anybody have a "buzzword bingo" card for this?

[MonkeyBoyo]

I may have understood some of the article but it seems to have been mainly an exercise in trotting out what somebody thinks are the most trendy buzzwords.

Unrealistic convergence plan

[Bruce Perens]

Placing all access points in a single telecom closet for what are generally rather spacious properties requires that 2.4 GHz signals be carried through coaxial cable that is very lossy at that frequency - it might be fair to expect up to 90% of the signal to be lost in the wire. There is an FCC limit on the transmitted power, and even if you manage to boost that at the antenna you will be boosting noise as well. And this attenuation and noise will of course hurt receiving too. This is in general going to result in lower wireless quality than desired, much lower than possible.

Instead, get zero-management access points that do not do NAT, routing, etc, and treat them just like antennas once you set the SSID. Do the protocol processing in the telecom closet with a higher grade of hardware than consumer equipment. Cache DNS and web transfers there. Work with Slingbox to engineer channel aggregation with multicasting that bypasses the home units while transmitting the same programming, because so many of those folks are watching the same sports game. I can think of some interesting approaches to the possible legal issues with Slingbox aggregating channels, no doubt they can as well. Can an in-house video alternative be made as attractive as Slingbox? That's another solution.

Bruce

Re:Unrealistic convergence plan

[Leto-II]

Slingbox is not P2P at all. You stream content from your computer back home to wherever you want to watch it.

Re:Unrealistic convergence plan

[Franso6]

In that definition everything that's unicast is Peer to Peer.
The term is nowadays used for a form of content distribution that's based on using end-user-owned, non-specialised machines working collectively.
What you're referring to is a client-server model.. Usually considered as more or less the opposite of a P2P model.
I agree that the naming peer to peer is unfortunate, though.

Re:Unrealistic convergence plan

[Bruce Perens]

I wasn't talking about leaky coax, but TV coax - RG-6, -57, -59, etc. Some of the "antenna distribution systems" not only use that, they might even use the same wire that is used to distribute television around the building. Obviously a solution that does not require wire pulling is attractive to these properties, that's why so many of the hotel-room wired Ethernet devices are really a sort of short-haul DSL piggybacked on the room phone line. I bet they feed the WAPs with that pseudo-DSL, too.

When you said "zero-managment access points", were you referring to Cisco's LWAPP architecture

Cisco has a nice product in this niche, maybe with better reliability than consumer equipment and maybe that is worth the price. But most consumer equipment can be put into "LAN Bridge Mode" and would achieve the same thing.

Thanks

Bruce

Astroturf?

[UESMark]

This seems like a thinly veiled ad for Marriot internet access.

Biggest lesson learned......

[Nick Driver]

...is that the worst threats to your network do not necessarily come from outside.... they almost always come from your very own moronic employees.

Re:Where's the horror?

[Kyojin]

Whoever modded this "insightful" needs to get a clue. The article doesn't even have 3 pages, it has 2. Also the only mention of "Linux" is in the ads around the story, not in either of the two pages of the story.

Microsoft update + Public Network = Instant DOS

[James_G]

While I was at the 2005 MysqlUC in Santa Clara, Microsoft put out one of their massive patch releases. Despite a large number of people running OS X or some Linux variant, there were still enough geeks in the hotel running Windows that they essentially DoS'ed the hotel's internet connection for about 2 days. I went down to reception at the time to find out if it was just the wireless, but the front desk people were similarly frustrated (they shared the same connection). Packet loss was at about 99%.

That was probably just a T1 or something, but still, pretty funny. I wonder if Microsoft realises the damage potential..

Re:Microsoft update + Public Network = Instant DOS

[imac.usr]

What kind of fucking idiot updates their laptop during a conference? You wait to do that shit until you get back home in case it screws your machine.

Re:Microsoft update + Public Network = Instant DOS

[Nezer]

What kind of fucking idiot updates their laptop during a conference? You wait to do that shit until you get back home in case it screws your machine.

Let's see... At a conference your computer is connected to hostile networks nearly all the time. Depending on the conference, there are potentially a LOT of people that know about 0-day exploits and might want to try something dumb.

I dunno. I can see your argument but there may be very good reasons to patch your system ASAP. I used to work in an environment where NOTHING got patched because they were afraid of fucking-up production services. I argued until I was blue in the face that we needed to do something and have a plan for deploying patches. I even went so far as to make proposals explaining the benefits, the risks, and the costs. No one would listen to me because I was a UNIX admin on a Windows team. Eventually I was let go and no one else took-up my cause (perhaps the cause was a large reason I was let go). No one on the team, except me, felt that there was any risk because the networks were "isolated" behind three layers of firewalls. About three months after I left some nasty work managed to find it's way into this "isolated" network and wreak much more havoc than we ever could have patching the damned servers.

I know that this isn't exactly the same thing as updating your laptop while on the road, but sometimes the updates are just worth the risk.

Perhaps the hotels should consider a caching proxy for just these sorts of events. Let the first user wait for the the download to come down the pipe and everyone else can leach from the proxy.

Re:Where's the horror?

[Bastard of Subhumani]

You must be new here.

Re:The Horror

[eln]

Sounds like a lack of capacity planning. If the marketroids were planning a huge marketing push, the IT manager should have been given the resources to increase network capacity prior to that push. If the IT manager was given proper warning and funding prior to the marketing campaign, it's his fault for not effectively utilizing it. Either way, the problem could have been avoided with proper management.

Ironically...

[EReidJ]

With some irony, I am reading this story from a Marriott hotel room at the Marriott Boca Raton. I've had mostly no problem with their services, but here are a few things I would call interesting: * My laptop can often see multiple nodes, some very fast, some blazingly slow. If you stay in a Marriott, try out the different nodes you can see. * Some Marriott properties give free Internet access, some cost $10/day. I wish Marriott would be consistent across all their properties. * If you stay in a Marriott that does charge for access, as for a low floor. This is because often there's a single wireless connection in the business center that is free, but other access points cost money. So if you can get a room near the business center, you'll be able to hook up to that one for free. * Finally, I've never had a problem with BitTorrent uploading at any Marriott property. I don't know if they leave all their ports open or what, but I traditionally leave my uploading port for BitTorrent open on 34567, and I've never had a problem with a torrent at a Marriott.

Re:Where's the horror?

[dwater]

Well, my mum told me I shouldn't arrange to meet people off the net'...but, er, maybe...

Re:overbooking/non-refundable

[evought]

That the reason they do it is because statistically a percentage of people don't make a given flight. Sometimes it ends up in conflicts, especially since they tend to err on the side of being full rather than no conflicts, but there is good reason to do it. If 100% of people who wanted on a flight showed up, they'd never overbook. However about 10-15% of people cancel their reservations or otherwise fail to show. Thus it makes sense to overbook the aircraft.

That might almost be an excuse except that they sell non-refundable "you die, you fly" tickets, supposedly for the exact same reason. Those empty seats are already paid for. They are trying to make additional money off of them at the cost of double booking. Like most businesses, they get you both ways and make you deal with the mistakes and inconvenience.