BT Internet Outage Was Our Fault, Says Equinix (theregister.co.uk)

Kat Hall, reporting for The Register: Telecity's owner, Equinix, has 'fessed up to a "brief outage" which subsequently knocked 10 per cent of BT internet users offline this morning as well as a number of other providers. A spokesman from the group, which slurped up Telecity for 2.3bn euro last year, confirmed that the outage occurred at its LD8 site in the Docklands. The company has nine London sites which service more than 600 businesses.The outage occurred due to power failure, which lasted for around 75 minutes. ( Update: Some readers note that the outage lasted for as long as three hours. ) BT wasn't the only ISP that suffered an outage earlier this morning. All services have been restored, according to Ars Technica. Update: 07/20 14:57 GMT: It was apparently a faulty UPS that caused the outage.

Comments

Re:So

[fustakrakich]

Well, for one thing it shows how easy it is to pull the plug on a lot of people at once. A single service like this shouldn't have this kind of power. It illustrates the necessity of having alternate hookups that can *route around the damage*

This is why you need redundancy and backups.

[Ash-Fox]

This is why I have redundancy and backups. This morning I had connectivity issues on my 100Mbps BT line, so I switched to my 200Mbps Virgin Media line and all was well.

It does surprise me how many people that depend largely on the Internet for work don't get a second Internet connection (and do so on separate infrastructure) and then complain when they have down time and how some how they couldn't get 100% uptime all year.

Re:This is why you need redundancy and backups.

[110010001000]

Wow. Here in Seattle we are still using 9600 baud modems.

Re: This is why you need redundancy and backups.

[Anonymous Coward]

This sounds like when Mediacom lost a single router in Kansas City a few years ago. The redundant router was sitting in the data center but had never been powered up. When they did turn it on it took close to a week to get it working. That little mishap knocked out all of the US central plains.

Of course that was only one provider so not exactly the same scale. Same level of stupidity though.

Re:This is why you need redundancy and backups.

[known_coward_69]

i have a hot spot on my iphone. if my internet went out and i absolutely needed to vpn into work i'd use my phone and expense it later. or i'd call the people that control our company phones and ask them to enable the hot spot feature on my employer owned Samsung Galaxy. not a big deal and not worth a second monthly bill just to have a circuit i'll rarely use

Re:This is why you need redundancy and backups.

[AmiMoJo]

Virgin does offer "200Mb" in some areas. I was on it earlier this year, but was getting around 1.5Mb for most of the day so gave up trying to get them to fix it.

BT's "fibre" offering isn't really fibre, but it goes up to 80Mb over your copper phone line. They also offer even faster real fibre in a very few areas, and charge silly money for it. One of the reasons that the government is saying they aren't investing enough is their failure to roll fibre out to most of the UK, preferring to milk their crappy old copper network for as long as possible.

Re:This is why you need redundancy and backups.

[Ash-Fox]

I have VDSL2 in my area on BT which is part of the BT Infinity package, which I'm not using right now for obvious reasons, my Virgin Media connection however is FTTP part of Virgin Media's VIVID which does provide 200Mbps, here is my speed test to prove it:

http://www.speedtest.net/my-re...

Re:This is why you need redundancy and backups.

[Ash-Fox]

but it goes up to 80Mb over your copper phone line.

If you're lucky, you could be in an area where they're trialing VDSL2 (like mine) and it goes up to 120Mbps.

Re: This is why you need redundancy and backups.

[Ash-Fox]

If I'm lying, how did I fabricate my speed test?

http://www.speedtest.net/my-re...

Re:This is why you need redundancy and backups.

[AmiMoJo]

I wish they would stop pissing around with DSL and just lay in some fibre. Other countries started doing it years ago... In fact my ex in Japan had a symmetric 100/100 fibre connection back in 2005, eleven years ago, for less than I pay to get 50/15 VDSL now.

The government had a large fund for upgrading our broadband infrastructure, but BT pissed it away on upgrading its copper network. NEC offered to install fibre to the home everywhere, but because the government was chummy with BT's management they didn't get it.

Re:This is why you need redundancy and backups.

[petermgreen]

That depends on where you live and how much you are prepared to pay.

Afaict in most urban areas BT openreach "broadband" services top out at "up to 80Mbps" FTCC. In a few trial areas they have "up to 120Mbps" FTTC or "up to 330Mbps" FTTP.

There was supposed to be a product called FTTPoD which would allow people in FTTC areas to get FTTP if they paid a steep (usually thousands of pounds iirc) installation charge but new installations under that program have been suspended.

Virgin media are offering a 200Mbps cable package for home broadband customers and a 300Mbps cable package for buisness broadband customers.

If you move beyond the "broadband" products to the dedicated fiber products then it becomes a matter of how much you are prepared to pay.

Re:This is why you need redundancy and backups.

[Ash-Fox]

I wish they would stop pissing around with DSL and just lay in some fibre.

Honestly, I think BT's issues are more to do with the fact that BT openreach doesn't have a greater self autonomy, which lead to certain decisions that benefit BT more-so than everyone.

In fact my ex in Japan had a symmetric 100/100 fibre connection back in 2005, eleven years ago, for less than I pay to get 50/15 VDSL now.

I have friends in Japan that can't even get a landline and have to rely on mobile Internet, so, I would imagine your ex is where I am now, in a sweet spot for Internet connectivity. I don't even pay full price because I negotiate with the sales persons and sign yearly contracts. Symmetric lines though would be really nice, it's why I prefer my BT line (more upload).

NEC offered to install fibre to the home everywhere, but because the government was chummy with BT's management they didn't get it.

Didn't NEC want more money with their proposal?

Re:This is why you need redundancy and backups.

[Anonymous Coward]

I pay $100/m plus tax, no extra fees and free installation, for a 250/250 dedicated business fiber in the northern midwest. Playing games at home with 7ms pings to nearly every datacenter in Chicago. Less than 1ms of jitter to most datacenters around the world, including Australia and Japan. They're competent enough that I've emailed their support a trace route and they fixed a less than 20ms ping spike in under 30 minutes.

If you get any less service than that you're getting ripped off. My bill has not changed an iota in 5 years, but they gave me a free upgrade to the next tier a year ago. I should mention I don't even live in the city. Out in the woods. The ISP is over 100 years old and privately owned.

Re: This is why you need redundancy and backups.

[Xest]

Oh a link on the internet, well that's proven that then.

Besides, that link doesn't even make any sense, you claim you're on a 200Mbps connection, but that speed test shows over 200Mb/s.

To hit 203Mb/s as you claim via your speedtest, you'd actually need a 1.6Gbps connection.

Re: This is why you need redundancy and backups.

[Ash-Fox]

Oh a link on the internet, well that's proven that then.

A link to a known speed testing website.

Besides, that link doesn't even make any sense, you claim you're on a 200Mbps connection, but that speed test shows over 200Mb/s.

Like I'm going to complain if it's slightly more.

To hit 203Mb/s as you claim via your speedtest, you'd actually need a 1.6Gbps connection.

Anyone sensible these days referring to Internet connection speeds are talking about the speeds they can get on the Internet.

But if you want to know more about the Virgin Media aspect, I believe it's using EuroDOCSIS v3.0 currently, there is a nice little table on Wikipedia showing the connection through-puts on https://en.wikipedia.org/wiki/...

Re: This is why you need redundancy and backups.

[Xest]

"A link to a known speed testing website."

For which we have no way of linking the results to your actual connection. That link could be to anyone's results, or if they're you're results we have absolutely no evidence that they're from your connection. In other words, the link is entirely meaningless.

"Like I'm going to complain if it's slightly more."

It's not just slightly more, it's 1.4Gbps more.

"Anyone sensible these days referring to Internet connection speeds are talking about the speeds they can get on the Internet."

You don't seem to grasp this topic. The speed you're claiming to get on the internet, i.e. 203Mb/s is equivalent to a 1.6Gbps connection. The speed result you've given doesn't even remotely match your claimed connection speed, it's literally over an order of magnitude faster than the speed you're claiming you have, that is, the internet speed test results you're showing do not even remotely match the connection you claim to have, they far and away exceed it, well beyond the speed of any connection available to consumers in the UK. If you're trying to complain that this site is using the terms Mbps and Mb/s interchangeable then you're simply lying, because, thanks to the fact the site is public, it was trivial for me to run the test and confirm that it does in fact use the terms correctly, which you are not doing. You therefore need to either:

1) Admit you're lying, and that this isn't your speed test

or

2) Change your claim to pretend you have a 1.6Gbps connection

"But if you want to know more about the Virgin Media aspect, I believe it's using EuroDOCSIS v3.0 currently, there is a nice little table on Wikipedia showing the connection through-puts on https://en.wikipedia.org/wiki/..."

Ironically this entirely disproves your whole argument because even with the maximum channel configuration in that link, you still couldn't quite achieve the connection speed you claim to have from your link.

I suggest next time you try and fake your connection speed as being 200Mbps you look for a speed test of just short of 25Mb/s, then it'll actually make sense, but right now you're showing a speed test for a connection that no consumer in the UK can get and that isn't even available as any standard business package, it'd require a specially commissioned line from BT, which, given your apparent lack of understanding of the difference between Mbps and Mb/s it's pretty clear you'd have no idea what to ask for to acquire anyway, much less before you manage to find the necessary hundreds of thousands, likely millions of pounds to get the line installed and commissioned. Virgin don't even have the capacity to provide such a connection.

I was a bit surprised when I read the thread that someone would accuse you of lying, it seemed a bit odd and I wasn't aware you're a person that would be considered to be a serial liar. Given the fact you've now utterly destroyed your own case by providing "evidence" that isn't, I'm saddened to see that whoever it is that declared you a liar does in fact seem to be right. It seems an incredibly odd thing to lie about, but I assume you must have some insecurity issues or similar that cause you to do it, but whatever, I'm not your psychiatrist. I'd just suggest you either figure out what motivates you to lie, and stop it, or if you must persist in lying, to actually understand the topic well enough to actually pull it off in future.

Re: This is why you need redundancy and backups.

[Ash-Fox]

I don't know what you want me to do to prove it to you. I'm more than happy to cooperate in any test you'd like me to perform.

Here is a quick and dirty video I recorded of me doing the speed test.

Feel free to suggest another way for me to prove it if that's insufficient.

Re: This is why you need redundancy and backups.

[Ash-Fox]

it'd require a specially commissioned line from BT

Added note: Virgin Media's VIVID offering does not run on BT openreach. It runs on Virgin Media's cable network.

Re: This is why you need redundancy and backups.

[Ash-Fox]

This is a follow up video correcting a misunderstanding.

Re: This is why you need redundancy and backups.

[Ash-Fox]

I should really watch these things before I upload them. Saying "megabytes per second" instead of "megabits per second" was dumb.

Re: This is why you need redundancy and backups.

[Xest]

Yes, but Virgin Media does not have any 1.6Gbps+ offering, which is what you're claiming you have. For that you'd have to go to BT because Virgin Media does not commission lines of that speed to individuals.

Re: This is why you need redundancy and backups.

[Ash-Fox]

If you looked at my second video link, I actually show you the mislabelling on speedtest.net's part. Where they are showing Mbps, then when I choose share html link, it turns into Mb/s on the shared link.

Re: So

[Anonymous Coward]

Because the 10% part of the story is a lie. It was closer to 60% of total UK broadband users and no transatlantic traffic for 80%.

Re:So

[Ash-Fox]

A single service like this shouldn't have this kind of power. It illustrates the necessity of having alternate hookups that can *route around the damage*

As someone was affected, that's exactly what I have.

Who?

[Anonymous Coward]

What is Telecity/Equiniz or BT or Docklands? This ain't "News for British Inbreds".

Re: Who?

[Anonymous Coward]

Bt is the retail side of bt group, who supply phone and Internet services, pre privitization they had a monopoly on phones (but not cable), now, open reach (the other part of the company) run the POTS network in the UK and sell the service to bt and other suppliers.

Docklands Is the slang term for the ex industrial areas along the Thames in the east end of london

"All services have been restored"

[CaptainOfSpray]

The outage lasted a LOT longer than 75 minutes. I tried repeatedly to get into BT webmail all morning - it was at least 3 hours after the outage before I succeeded. And during all that time, the BT DNS service was not working, so I couldn't do any other work.

#RANT# The BT-supplied router, the fornicating clunky useless and slow Home Hub 5, does not allow you to put in your own DNS servers. So while it is proof against subscriber morons, it is totally vulnerable to central morons#/RANT#

Re:"All services have been restored"

[Ash-Fox]

The outage lasted a LOT longer than 75 minutes.

I expect downtime, which is precisely why I have two separate Internet connections at home that run on separate infrastructure (BT's Infinity, Virgin Media's VIVID) and if I have to, I could connect my Vodafone 4G dongle, but my router isn't setup for that currently. Honestly, switched over within the first 5 minutes and had no problems the rest of the day.

tried repeatedly to get into BT webmail all morning

I'm shocked you use BT webmail though, it's never had a good track record and doesn't that lock you in as far as ISPs go?

The BT-supplied router, the fornicating clunky useless and slow Home Hub 5, does not allow you to put in your own DNS servers.

If you can get the BT openreach modem, you can put any router that supports PPPoE on it.

Re: "All services have been restored"

[Anonymous Coward]

How many times are you going to post that you have 2 internet connections at home?

Also BT doesn't use PPPoE it uses PPP over ATM like most of the rest of the UK ADSL market.

Ash Fox, the guy whose female friend have him a place to live and he couldn't resist wearing her underwear. I see you still have the same superior attitude backed up by a complete lack of actual knowledge...

Re:"All services have been restored"

[tomxor]

...And during all that time, the BT DNS service was not working, so I couldn't do any other work. #RANT# The BT-supplied router, the fornicating clunky useless and slow Home Hub 5, does not allow you to put in your own DNS servers...

1. Never rely on routers supplied by ISPs (especially BT), they reliably suck giant fucking donkey balls.

2. Your OS doesn't have to (and arguably shouldn't) use the DNS server address given by the DHCP (router).

Arguably you should specify the DNS on your OS for security purposes anyway (e.g a compromised router sending all your DNS requests to a malicious server and sending you off to some amazon.com impersonation... that's what you risk every time your computer connects to a public AP).

Another reason not to use ISP's DNS is they all poison it these days, I'm not on BT anymore but they used to redirect all missing DNS entries to their search page with loads of advertising... I can't begin to go into whats wrong with that because i'd take too long.

Re: "All services have been restored"

[Ash-Fox]

How many times are you going to post that you have 2 internet connections at home?

I have allocated a quota of 13.

Also BT doesn't use PPPoE it uses PPP over ATM like most of the rest of the UK ADSL market.

I'm on VDSL, not ADSL and when you use the BT openreach modem, it provides a PPPoE tunnel for any router you connect to it.

Ash Fox, the guy whose female friend have him a place to live and he couldn't resist wearing her underwear.

I think you're confusing me for someone else, I live with my boyfriend and sometimes we share underwear!

I see you still have the same superior attitude backed up by a complete lack of actual knowledge...

If you were knowledgeable and read my other posts, you would have noted that my speeds are beyond ADSL capabilities and wouldn't have made such a silly response.

Re: "All services have been restored"

[jabuzz]

You are dead wrong there. If you have a VDSL2 circuit then you mostly use PPPoE, unless you are with TalkTalk in which case you need to set your router to do a VLAN insertion and there is no authentication whatsoever.

If you are going to pull someone up, make sure that you actually know what you are talking about in the first place.

I would also note that most people in the UK could simply put their mobile phone into hotspot mode and leach off their data allowance for a backup internet connection. As such the majority of people have two internet connections even if they don't know how to use the second.

"The" Docklands.

[Anonymous Coward]

Must be somewhere in The London.

Heh.

Re:So

[AmiMoJo]

Lucky you. For many a BT line is the only choice. At best I can switch to mobile tethering as a backup. In fact my router even has a USB port for that very purpose.

Re:So

[Guybrush_T]

Not in the UK, BT (Bouygues Telecom) is an ISP in France !

I don't think it is about British Telecom, the editor would have replaced BT by "British Telecom" so that everybody can understand what the article is about.

Re:So

[Anonymous Coward]

Did you pay for a back up circuit on a diverse path? If not then no one can bitch when equipment failure happens and your circuit does not work.

Re:Who cares what happens in irrelevant UK?

[Anonymous Coward]

Agreed, /. should only be about 'murica. The world be damned!

Re:So

[mrbester]

It's from El Reg, who, being British, would have specified Bouygues. That, plus Telehouse is in London and services British Telecom (the ISP part).

Here's a graph. Meanwhile, BT Care suggested....

[hazeii]

Graph of outage.

It was pretty funny; downdetector.co.uk showed the problem very clearly, affecting large swathes of the country for about 3 hours. And on the same page, there was BT Care suggesting that people reset their routers and reboot their PCs :)

When it went down, a quick traceroute showed the problem to be at BT@Telehouse. Luckily, we retained connectivity to our hosted server (even though most of the rest of the net was unreachable) so a combination of 'ssh -D 1080' and twiddling proxy settings worked around it (note: must look into 'tsocks').

It was a very big outage (despite all the PR flackery seeking to minimise it). And shame on BT, for having a single point failure like that cause such disruption.

"Brief Outage"

[maxrate]

I own an ISP. We consider a 'brief outage' something under 2 minutes, not 75+

Re:"Brief Outage"

[bruce_the_loon]

Now you know better. Next time you can finish that cup of coffee and the BLT before going to sort out the problem. It will still be well within the new BT-defined brief outage and nobody can complain at you.

Re:Hillary for prison

[Dishevel]

"They really want to drop the bomb and exterminate everyone outside the borders."

See. This is where people stop caring about your opinion. Where people suddenly realize that you just say shit that makes you feel good.
No truth or facts needed. Where exactly did you see this "Information" you are passing on?
No real need to answer. You think that having borders in a country, is equivalent to murdering everyone outside them?
or are you just lying?

Just be glad that you don't have Windstream

[GerryGilmore]

As someone who lives in a captive Windstream area, I can tell you that 75 minutes of outage would be GREAT! We regularly have outages that last for over a day!! Of course, here in Conservativia-land, any discussion of using the Gummint to force Windstream to allow competing ISPs to use the existing copper plant won't even get started, despite the suffering that local businesses go through over the outages.

Re:Just be glad that you don't have Windstream

[JackieBrown]

As someone who lives in a captive Windstream area, I can tell you that 75 minutes of outage would be GREAT! We regularly have outages that last for over a day!! Of course, here in Conservativia-land, any discussion of using the Gummint to force Windstream to allow competing ISPs to use the existing copper plant won't even get started, despite the suffering that local businesses go through over the outages.

I live in a blue city and have the same problem. You are naive in thinking that just because the democrats patronize you, that means they are in bed just as much - and a lot of times more - than the conservatives.

Re:Here's a graph. Meanwhile, BT Care suggested...

[Anonymous Coward]

I thought that the whole point of the Internet was that it could route around problems. It doesn't seem to have help here - does anyone know why not?

Faulty UPS?

[Gunfighter]

If it's a UPS that's not U, doesn't that just make it a PS? Perhaps an IPS, or even a PoS?

Faulty UPS?

[Not-a-Neg]

Reminds me of a company I know that built a brand new data center, put in an over-sized UPS system, over-sized electric generator, state of the art power monitoring/transfer system, and tested the generator bi-annually. Only there were three problems when the area finally suffered a blackout:

1.) They never tested fail-over to the UPS, they had only tested starting up the generator and then manually switched off mainline power once the generator was fully operational to see if it worked.

2.) The UPS installer never bothered to connect the batteries to the power control unit so when power did fail everything immediately lost power (these were racks of large batteries wired together, not the plug-and-play consumer stuff.) When the generator kicked in, everything tried to turn back on at the same time and tripped the breaker. LOL.

3.) The air conditioner was not connected to the fancy power transfer system and after a little over an hour, servers started throttling and eventually shut down from the heat. This all happened in early August on one of the hottest days of that year.

Don't know how to configure a data centre

[CanadianMacFan]

Back when I was a system administrator for a government department looking after the computers for the web sites losing a UPS in the data centre would have been no big deal. Each chassis holding our blade servers held four power supplies. Two power supplies were connected to one power distribution unit (PDU) and the other two power supplies were connected to a different PDU. The PDUs were cabinet models and each PDU was connected to a separate UPS. The whole data centre had a diesel generator for a backup too. So losing a UPS, or a PDU, would have had no impact on my servers since they still would have gotten the electricity through the other path. I also had redundant network and SAN switches. I made sure that every server had at least two instances running so that one could be brought down for service without impacting the users. The only issue that I ever had with those IBM blades was the SCSI hard drives that started to fail after about three years (not bad in a server environment) which were replaced with the SAN. All running Linux of course.

As an aside we got stuck with a rack of the first generation of the HP blades and they were horrible. They ran so hot special cooling had to be installed in the data centre for them and we kept on have RAM failures. I was thankful that I had as little to do with them as possible.

Re: Here's a graph. Meanwhile, BT Care suggested..

[Anonymous Coward]

tsocks? Something offered by Yorkshire themed PlusNet?

Re: Here's a graph. Meanwhile, BT Care suggested..

[hazeii]

man 8 tsocks

Re:So

[Anonymous Coward]

The company is BT so its correct, they lost the "british telecom" name a while back rebranding for a global market. Somewhat like France telecom became Orange...

Re:Hillary for prison

[Anonymous Coward]

Where people suddenly realize that you just say shit that makes you feel good.

No truth or facts needed.

Oh, what? Now you're accusing me of plagiarizing his MO? Sorry, his supporters are fucking nutcases (are you one of them?), and really kind of dangerous. Black people and Mexicans won't be safe with this kind of shit going on.

Where exactly did you see this "Information" you are passing on?

Listen to him talk. The man is off his rocker, except I know he's just appealing to the monkey within us all. He sure did bring out the cockroaches. If he wins, it will be a true disaster. Don't ever doubt it. For all our benefit, just let Clinton take the job until the damn republicans pick up what's left of their brains off the floor and reassemble Bob Dole. I actually liked him. We should've elected him back in '96. He's a good man, with much more honor than the buckets of scum that run the party now.

Re:So

[Anonymous Coward]

Was actually pretty much just their DNS servers that were down. I could still access pages that had DNS requests cached.

Re:So

[Ash-Fox]

For many a BT line is the only choice.

Maybe for the last mile from the cabinet. Even when I was living in the middle of nowhere in Somerset, I was able to get Local Loop Unbundled providers (Sky is one of the biggest LLUs, but their stuff is aging [ADSL only], Andrew and Arnold [ADSL+, line bondings etc] which is typically more expensive will go to the effort of installing equipment in your cabinet and exchange to provide you service outside of BT's backhaul).

Re:So

[AmiMoJo]

I have an LLU provider as my ISP, but the line BT provided is basically a bit of wet string. It just barely scrapes though the minimum requirement so they won't fix it.

Re:So

[Ash-Fox]

the line BT provided is basically a bit of wet string. It just barely scrapes though the minimum requirement so they won't fix it.

If you're certain that's the problem (because the only reason why I was able to tell issue was, was because of a neighbor's connectivity)...

I was able to get BT Openreach to replace a line for a friend of mine (even though they said nothing was wrong with it) up to the master socket for 150GBP, they had a call out fee of 130GBP on top though.

Now, if your LLU provider (my friend's wasn't) is any good, they should be able to arrange that for you and get the call out fee waived at the very least.

Re:So

[AmiMoJo]

Openreach charge £130 to replace the wiring to the pole, but to replace the wiring all the way to cabinet... I can't find a price, but it's likely to be similar to installing a new line, i.e. a few thousand minimum.

The line is partly aluminium, rather than being all copper.

Re:Hillary for prison

[Dishevel]

Again. If you have a quote that has him wanting kill everyone that is not white or American, let me know. There are a lot of issues with Trump. Not one of them the drivel that you are posting though.

Re:Hillary for prison

[Anonymous Coward]

And once again, you didn't hear me say that about Trump. His followers want to do it, desperately. They are a real threat and should be locked up. We should have kicked the South out of the Union, but it would like Haiti by now if we did, only with "white black people, not as ugly as the white man, but just as crazy".

Re:Hillary for prison

[Dishevel]

I have not met those people.
All the liberals hate America and want to see thousands of dead cops.

All Democrats are liars and sell American secrets.

All black people are gang members and kill indiscriminately.

Hur, Hur. I am making arguments on the internet.

Re:Hillary for prison

[Anonymous Coward]

You are right about the democrats being liars... Without Trump's assistance they can't win any other way.