EBone/KPNQwest Network Shutting Down

reginald.barclay writes "As KPNQwest has filed for bankruptcy some time ago, also EBone, which they aquired some months ago, goes down the drain. Together, these two companies carried betwenn 1/3 and 1/2 of European IP Traffic (and, in the case of KPNQwest, an unknown portion of voice). Employees at Ebone were laid off last week and told to abandon their NOC. But instead of getting drunk and over with it, they occupied their former workplace. Now even their time is running out, and one of Europes oldest backbone carriers will probably be shut down today, at 1700 CET. I wonder how many of their customers (mostly ISPs and VBCs themselves) have managed to run to the competition in time. Nevertheless, I expect the routing in large parts of Europe to be very interesting (in the chinese sense, of course) over the coming weekend and early next week." Update: 06/14 18:02 GMT by M : Apparently KPNQwest's creditors have agreed to pay to keep the place going until the end of June.

Comments

Hm.

[Anonymous Coward]

The next time someone is talking on slashdot about how mergers are natural in the telecom industry, and telecom companies that lock down entire markets until they are local monopolies in some cities are Just Trying to Make a Living, and the government has no right to dictate that a telecom company be "nice" to their competitors, and there's absolutely no harm in megamerger after megamerger followed by competitors being disallowed from leasing space on the local telecom equipment..

I'm going to link this article.

And then i'm going to scream something incoherent along the lines of "BAD AYNDROID!! BAD!! BAD!! SINGLE POINT OF FAILURE BAD!! MONOCULTURE BAD!!, and then curl up in a little ball and cry because no-one really cares.

25%-50% of traffic on Ebone?

[pieterh]

I'm sitting in an office in Brussels, it will be interesting to see how stopping this backbone affects the Internet in Europe. The BBD reported that Ebone carries 25%, their website reports 50%, of traffic in Europe.

Anyone tried this kind of nuclear blast on th Internet before?

Re:25%-50% of traffic on Ebone?

[RocketJeff]

Ebone carries about 25%, the rest of KPNQwest carries another 25% - the total company carries about 50%. Ebone was never really integrated into the rest of the KPNQwest network (hadn't had time before the company went splat).

6 pm GMT

[Ethelred Unraed]

7 pm CET is 6 pm GMT, or 1 pm US Eastern Time, or 10 am US Pacific Time.

HTH

Ethelred

Network being bought?

[mefistofeles]

At least one ISP (Bahnhof) in Sweden has put in a bid for some of the european network infrastructure according to Computer Sweden. So perhaps parts of it wont go down just yet.

Re:Network being bought?

[psych031337]

At least one ISP (Bahnhof) in Sweden has put in a bid for some of the european network infrastructure according to Computer Sweden. So perhaps parts of it wont go down just yet.

Add...
AT&T [US, according to WSJ]
Teles/Strato [.de, according to heise]
1&1 Webhosting (former Puretec) [.de]
...to the list of interested companies.

But I think it will not work out. Remember, suits are involved. Insolvency laws are complicated (esp. when it comes to a paneuropean company, there are at least three different "daughters" of KPNQwest going down around Europe). There have been quite some offers for the network (or at least parts of it) but so far obviouly nothing was good enough. And keep in mind that just 1 year or so ago most major european telco (and other potential buyers/investors) spent billions of euros they don't have to license a net that does not exist, in order to (maybe sometime) deliver services no one wants (yeah, talking UMTS here). I think most potential purchasers are kinda short on money.

According to varios news items on heise.de the debts (of KPNQwest) total around 2 billion euros by now, every day of further operation adds a mere million to that. Technically KPN was insolvent and done with on 5/31/02 at midnight.

Most parts of their nets still seem up and OK, and even if they flip the switch I suppose the "big internet outage of '02" won't take place. Most hosters and ISPs have already set up a fallback option. The insolvency is in the cooking pot for quite some time now, allowing preparation. Considering the fact that KPN did something 40%-50% of all european IP is scary, but when you realize that the other 50% have been properly delivered by other providers gives hope. Especially considering that most of their fibers ran on a very low capacity (one source says the utilisation of the KPNQ net is still a 1-digit percentage.

Already routing trouble in Germany

[mocm]

We already started to have routing trouble here in Germany. Mostly routes to German sites can't be established. That is from my regionla ISP here in Cologne (NetCologne), don't know how T-Online is doing. You could even see where a traceroute broke down when reaching KPN and was later established through other providers.

At least there is no problem to connect to /.

EBone

[sporkee]

s KPNQwest has filed for bankruptcy some time ago, also EBone, which they aquired some months ago, goes down the drain.

So they're EBoned then?

healthy business plans

[stock]

If a new company presents a business plan
to their potential investers, then the investors
decide if its a healthy business plan, and either
say yes or no. If a business plan is actually
a vapourware plan, like most .COM startups were,
then these same investors should have said that
its a nogo.

So today these same investers are loosing
a rather large amount of money. Now who
is to blame in then end?

Robert

Re:7 pm CET ???

[danamania]

live.save-ebone.com

that has a counter that indicates how long is left until all's closed...

a grrl & her server

Easynet Belgium had problems two weeks ago...

[sconest]

...due to KPNQwest's problems.
It took them 3 days to solve and now I don't see anymore problems.

Re:The UK

[LBU.Zorro]

Supposedly there aren't going to be many problems.

TheRegister has been running a few stories on this for a while, and I believe they at one point got comments from a number of ISPs, most said no worries, AOL said they use some of their net but that they have multiple other provides. BT claims there is stacks of capacity kicking around, so hopefully nothing but a minor glitch for the majority of users, although anyone who did NOT move from the KPNQwest network is screwed.

TheReg story on the latest is here: TheReg 4:45 BST Shutdown (Oddly enough the article states 4pm BST but the headline 4:45... Weird)
and the BT comment is here: TheReg - BT on Capacity

Z.

Shouldn't it be CE*S*T rather than CET?

[Moderation abuser]

I thought CEST was Central European *S*ummer Time, equivalent to BST and CET was equivalent to GMT. Except being an hour out of course.

Re:Shouldn't it be CE*S*T rather than CET?

[Moderation abuser]

But CET is for winter, CEST is for summer, they do change the times on the clock back/forward, so the time quoted may actually be an hour out if they didn't account for it in the article.

See, they've gone and confused everything now.

First sign?

[amorsen]

According to Internet Traffic Report the router defra229-tc.ebone.ne is not responding. Several other KPNQwest/Ebone routers are still up though.

Re:First sign?

[xmedh02]

Well, that router (with .net at the end) is not in the DNS (anymore). Other nodes in Frankturt are OK:

--- defra0202-tc-f3-3.ebone.net ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 46.6/50.8/54.3 ms

So far, from what can I see here, Ebone/KPNQwest works.

Lack of solid facts

[Zocalo]

It's on The Register too, but no one really seems to know for sure what's going on, or what the effects of a shutdown will be. I've seen reports of between 20% and 50% of current traffic, but no one is sure about what contingency plans (if any) are in place with KPNQwest's customers. As a result what are really only guesses as to the effect of shutdown vary from "none" to "disaster". It seems to me that the only thing that can be said for certain is that only time will tell. Anyway, are we always being told that the Internet was designed to withstand and route around this kind of thing? ;)

Re:Lack of solid facts

[Neon Spiral Injector]

Yeah, we'll just route right around Europe. I just hope you aren't trying to talk to anyone in the hole.

The point everyone makes about routing around damage is just that, the Internet is designed to be able to route around sections that go away. But if half the ISPs in Europe are behind the routers that go away, they'll be gone. But everyone else will still be able to talk to Asia, Africa, or what ever.

I do know that just about every time I've tracerouted to a site in Europe I've seen "ebone" in the trace. I think this could be bad for a lot of people.

Re:Lack of solid facts

[rosewood]

I beleive it was designed to withstand such an impact -- but if that was how it was actually implemented is yet to be seen. I know when the internet was all new and cool they used to say that the USA could be nuked and the rest of the world would be fine for connections. However, there have been stories over the years to the contrary.

Re:Lack of solid facts

[Zocalo]

I do know that just about every time I've tracerouted to a site in Europe I've seen "ebone" in the trace. I think this could be bad for a lot of people.

Yeah, and almost everytime I traceroute to East Asia (I'm in the UK) I see the US in there, but that doesn't mean I can't hop across Eurasia. KPNQwest isn't the only major network in Europe, so if the worst comes to the worst, we're going to lose those sites only connected via KPNQwest and experience an unknown speed reduction on the whole. The problem is no one can say for sure what will happen.

While I wish those caught on the hop by their management the best of luck, I have to admit to a certain desire to see what happens if the plug is pulled. There are bound to be some interesting lessons in there about network design and monopolies both I'm sure...

Re:Lack of solid facts

[Spruitje]

It's on The Register [theregister.co.uk] too


Well, according to mtr www.theregister.co.uk is now 28 hops away.
Traffic is now routed from chellonetworks to alter.net.
Instead of the normal 6 hops.
O yeah, it is routed from Amsterdam to London, then to New York, Washington and then back to London and to www.theregister.co.uk.
I can't imagen that this is the fastest route...
But to be fair i'm using an UPC/Chello connection.

Re:Redundancy

[Clay Mitchell]

even if they are deundant, it's going to put a hurtin on the other guys. if they are carrying 25-50% of the traffic, and it gets moved over to other backbones... well let's just say it's like trying to get 10 pounds of crap into a 5 pound bag.

ISPs not warning customers?

[Moderation abuser]

I haven't heard a thing about it from my ISP, or on the news.

Maybe everyone's confident it won't effect them.

Re:ISPs not warning customers?

[rot26]

Or maybe they're frantically emailing out their resumes before the T1 dies.

Re:ISPs not warning customers?

[AVee]

UPC/Chello gave out a warning here in the Netherlands that there users might experience a slowdown when the KPN-Qwest network goes down. They will not lose their connectivity, but the capacity will drop.

I think the situation will be the same for most ISP in europe, there are plenty of networks left when KPN-Qwest goes down, but the overall capacity will be less. Any decent ISP will have more then one uplink anyway, so i guess no one will be completely offline when the stop the network. Unless there are routing problems ofcourse.

Gmx.net

[rosewood]

Im not familiar with most Euro lines, etc. but I use GMX for my primary email ... and if that goes down with this Im going to cry. Any idea how I can check?

Re:Gmx.net

[psych031337]

must....resist...urge...to....post....flame...grng llxs....

Aw, what the heck, it's only karma.

Anyone using GMX.NET as a primary email deserves EVERYTHING bound to possibly happen... Do you have a record of how many outages/security leaks these guys had in the past. If they get the transition correct on first try, I'll happily revise my opinion about those guys... And you won't have to check, the problem will materialize on its own - if it does. Depending on where you are you might be lucky and sleep over the troublesome period because of the time shift.

And FYI: I can still call GMX in the browser, but traceroutes fail after a dozen hops... Keep your finger crossed.

And if you think you're unaffected...

[bryanp]

because you don't live in Europe, think again.

I live in Tennessee, but my email provider (Runbox.com) is based in Norway. Fortunately they managed to get their stuff together and should* be safe.

*should: a moral term that has nothing to do with computers. "It' should work." is a worthless statement.

Network won't be down long.

[peterdaly]

For better or worse, a couple large networking companies in my aread have gone out business recent, one of whom is our upstream provider. Unlinke what sounds like may happen in this case, we had zero downtime. Another company bought the network for pennies on the dollar. Only way we can tell anything changes is we havn't gotten a bill for two months because the new company doesn't have their act together in that regard yet.

Similar thing happened with the major compition to Verizon. They went out of business, and to the best of my knowledge their customers who have not left (many did), have not lost service, although that whole fiasco is not finished.

Anyway, chances are, another large networking company will buy the network for almost nothing, and pick up where the existing company left off...just with much lower capital investments, which may lead to lower prices.

I for one believe things like this are a mixed blessing, and in some cases needed to lower the cost structure for providing these services. Kinda like a built in cost correction.

I may be way off base with this, but that's what I think base on what I have seen locally.

-Pete

Re:Network won't be down long.

[tshoppa]

Another company bought the network for pennies on the dollar. Only way we can tell anything changes is we havn't gotten a bill for two months because the new company doesn't have their act together in that regard yet.

That would worry me if I was in the same situation. If the company that was succesfully issuing invoices went broke, then the replacement company which cannot issue invoices is going to soon be (if it isn't already) in even worse financial shape.

Re:Network won't be down long.

[RollingThunder]

Not if they aren't saddled with debt payments that the previous company had.

KPNQWest network will continue to run....

[Idaho]

...until clients find alternative solutions.

Mind you, KPN (which owns 40% of KPNQWest shares) has several Really Big Contracts with rather big companies, such as Schiphol Airport (which also has a very big hosting colo), guaranteeing that the network will ALWAYS run, or they'll have to pay the damages of breaking their contract.

So, as long as 'cost to keep the network running' < 'cost to piss off biggest customers REAL good', the network will keep running.

You can check this article (in Dutch), which says at least the Belgian network will keep running. Short translation of the article: employees where working for free to keep the network up and running, now they have a temporary contract for a few weeks, guaranteeing them they'll get paid if they keep the network running.

Re:KPNQWest network will continue to run....

[xmedh02]

Well, they are bankrupt, so they would only pay their SLA obligation from what's left after the bankrupt process.. Which will be a few per cent. And their biggest customers are pissed off REAL good, as you put it, already. Those Big Contracts have other connectivity now, I think..

Re:KPNQWest network will continue to run....

[Idaho]

Well, they are bankrupt, so they would only pay their SLA obligation from what's left after the bankrupt process..

Yup...but you should read carefully (or maybe I didn't explain well enough). It's like this:

• KPNQWest has gone bankrupt. They can't pay their employees. Still, some are working (for over a week now) for free, just because they feel they can't 'pull the plug' on their network.
• KPN has *not* gone bankrupt. It's KPN that has contracts with Very Big Customers and guaranteed them that their (KPNQWest provided) connections would always work (they can make promises like this because they own 40% of KPNQWest stock).

So, it's very much in KPN's interest that the network stays running. That is also why they are paying millions a day (by my understanding) to keep things running.

Because as soon as they don't, they will

1. Loose some VERY big customers
2. Get some REAL nasty lawsuits claiming the damage for breaking their contractual obligations. Those will be multi-million dollar law-suits, which KPN is very likely to loose.

And because KPN, as any large telecom provider (at least in Europe at the moment), is - to say the least - not exactly doing great by itself either, they will probably watch out REAL good not to let something like this happen!

Posting from Demon Internet

[caveman]

The following was recently posted to demon.announce by Malcolm Muir (although I've snipped it a bit; demon.announce is propogated outside of demon.net; so you can probably read the whole post at Yahoo by now, if you really want..

Whilst we have been aware of this possibility for some time now
our networks have been designed to take account of this type of
event. Naturally, given the issues with KPNQwest, we have recently
confirmed that our network can accommodate the loss of Ebone.

We estimate that about 5% of our traffic is currently routed by
Ebone and that is mostly to other European sites.

When or if Ebone is turned off we expect traffic to reroute via
other connections in London and Amsterdam.

We believe it is unlikely that there are any other networks
connected exclusively to Ebone, so we do not expect any
destinations to become unavailable in the event that Ebone closes.

In the event that Ebone close it will take time for traffic
patterns across Europe to settle down, however we expect the
effect will be limited to some traffic following less than
optimal routes and occasional hot spots of congestion.

I think that says it all, really.

The Internet is NOT dying

[peterdaly]

The internet is coming off its "high" from the late 90's, and is in the hangover stage. It will get over the hangover, swear off alc^M^M^M unrealistic spending on stupid ideas, and return to existance as a much more healthy mature Internet.

The bubble burst, and is coming back down to where it should be.

-Pete

Re:The UK

[hollow_man]

As Netcom UK we are now in control of large parts of the UK network and we fully intend to keep it running for as long as possible. However our main concern are the transmission links as they're all managed from Brussels.

As the Netcom entity we're pretty secure although we anticipate a mad scrambling to reconfigure part of our network, we will also try to keep the majority of the UK Ebone customers online, but all of that depends on the extend of the backbone shutdown, so Irish customers might be not as "lucky". But AS5571 should be largely okay.

More news:

[bons]

How did this happen?

quote: "According to information gathered by the group, the three executives awarded themselves ten-fold salary and bonus increases in May 2001, which were kept secret till the day after the sale of GTS's assets was announced. The three received a total of $21m (£14.7m) in 2001, a sum that amounted to 52 percent of the stock value of the company at the time, said Kaplan. By comparison, Enron's much criticized "loyalty bonuses" only amounted to a few percent of its value."

Worry not - ebone is staying up....

[Boss, Pointy Haired]

PHB: That's it, money gone, shut down the ebone.

NETOP: Right on cap'n, shutting down ebone now...

PHB: Thank you.

NETOP: Slight problem, the guy that knows the router config to stop it working left, he was made redundant last week.

PHB: Ah, can't we just switch them off?

NETOP: No can do cap'n, no remote power off for security reasons.

PHB: So we need somebody to go out to site and switch all the routers off?

NETOP: That's what i'm telling you captain. These CISCO's just work so well, and without the guy who knows how to configure them to stop working we can't shut the network down.

PHB: OK, how many sites is that?

NETOP: About 23,239

PHB: Ok, get onto field maintenance.

NETOP: Slight problem, they were all made redundant last week.

PHB: Anyone fancy a pint?

what's the sound of BGP flapping?

[mdouglas]

BGP statistics pertaining to KPNQwest AS286 also, keep your eye on NANOGfor any info related to the impact of the shutdown.

Execution of KPNQwest postponed by 5 weeks.

[pa3gvr]

According to this article (in Dutch only) Belgian unions (their members) are not going to shutdown the NOC in Hoeilaart(B).
They have come to an agreement with the curators. The curators have offered 40 employees a 5 week contract so the NOC can stay in operation with a skeleton crew. Employees of other NOCs ( 200 in total) around Europe were offered similar contracts.

It will probably be a 5 week long last breath.

Sjaak.

I heared a (slightly) different story.

[Fuzzums]

Yesterday (thursday) I heared someone say on the news that they found some investors and got enough money to live through this month.

That leaves them with roughly two weeks to find a real solution. The bottom line stays the same. They have to find a lot of money somewhere.

The strange thing is that announcement was made on th 6 o'clock news and the press release is from 5 o'clock.

KPNQwest Finland will continue to run

[vinsci]

According to this press release today (in Finnish) there have been several offers to buy the Finnish daughter company of KPNQwest.

[...] "In any way, our future looks secured and we are able to continue servicing our [Finnish] customers as in the past. The national network and the services of our customers works normally during the process. In addition, we have secured internation IP connections, in case there are disturbances in our Eurorings network." [...]

The rerouting has began...

[ManiaX Killerian]

I've been doing traceroutes , and about 30 mins ago i saw the following:

4 faste0-0-rtr13.Sofia.0rbitel.net (195.24.32.13) 6 ms
5 Orbitel-BTCNET.btc-net.bg (212.39.66.137) 8 ms
6 S5-1-0.PASAR2.Pastourelle.opentransit.net (193.251.248.85) 44 ms
7 P0-0.PASBB2.Pastourelle.opentransit.net (193.251.128.81) 1002 ms
8 P13-0.PASCR2.Pastourelle.opentransit.net (193.251.241.169) 2619 ms
9 P11-0.PASCR1.Pastourelle.opentransit.net (193.251.241.97) 2674 ms

and so on... looks like a lot of things moved in opentransit... Here's the trace from the other direction:

8 P3-0.NYKCR3.New-york.opentransit.net (193.251.248.110) 13 ms
9 P11-0.NYKCR2.New-york.opentransit.net (193.251.241.217) 16 ms
10 P4-0.PASCR1.Pastourelle.opentransit.net (193.251.241.133) 100 ms
11 P12-0.PASCR2.Pastourelle.opentransit.net (193.251.241.98) 100 ms
12 P7-0.PASBB2.Pastourelle.opentransit.net (193.251.241.170) 100 ms
13 P8-0-0.PASAR2.Pastourelle.opentransit.net (193.251.128.82) 2682 ms
14 Btc.GW.opentransit.net (193.251.248.86) 2701 ms

Let's hope they'll sort it out in the next 3-4 days.

What to listen to as we wait for it to go down ?

[little_fluffy_clouds]

Europe - "The Final Countdown"

Beligian Government should simply take it over.

[uncoveror]

The Belgian Government, either unilaterally or on the behalf of the European Union, should simply take this NOC over. Nationalize it. The backbone is too important to let anyone just pull the plug on it. Some things are just too important to be left to greedy businessmen. Kudos to the former employees who have kept the network up as volunteers because it was neccesary and the right thing to do.

Re:Nice of the NOC folks

[Paul Jakma]

it's extremely unlikely to be OC-anything, cause it's in Europe and they use the STM/SDH system. :)

--paulj

Re:Chinese Sense?

[Tim C]

He was referring to the saying "May you live in interesting times", which is apparently of Chinese origin.

The idea is that, whilst that *sounds* good, if you think about it, "interesting times" are generally either full of danger or hardship (eg war, recession, etc).

Don't worry, the poster wasn't being racist.

Cheers,

Tim

Ebone is still alive

[chrysalis]

The Ebone network hasn't been shut down. I'm a pround Ebone customer, and our network is properly wor

Bailouts are by far the WORST solution

[FreeUser]

I dunno, I think there should be a government bailout of something like this.

No. No. No.

If the infrastructure is really that critical (like a country's highway system is), then the government should nationalize the backbone and make it available to competing ISPs under identical terms (i.e. actually allow competition and prevent vertical monopolies leveraged from physical monopolies over last mile cable and critical backbone links from forming).

If it isn't that important, then they should simply stand aside and let these companies go belly up, with all the consiquences that entails.

In no way should an existing, unsuccessful commercial enterprise be propped up by government: either the free market works, or there is no free market (read: monopoly), in which case the underlying structural cause of the monopoly (if any, in this case perhaps the copper, esp. if last-mile copper is involved) should be nationalized, and the market opened up so it can operate freely, with competition.

Bailouts are the worst of both possible worlds: government intervention and expenditure of public funds AND private corporate control with no public accountability (beyond their stockholders, if they happen to be traded publicly).

When Northpoint went under with no warning it sucked (we were off the net for 2 days due to that fiasco, and NSI didn't fix our DNS for 10 days), but even there a government bailout would have been wrong.

Nationalizing Ameritech's last mile of copper, so that Ameritech wouldn't be able to maliciously leverage that monopoly to drive competing DSL providers like Northpoint out of business, on the other hand, would have been a reasonable response. Unfortunately the ayndroids of the far right have managed to convince a large percentage of people that free enterprise is a panacea in all contexts and the only good governance is no governance. Nonsense, of course, as anyone can see (just try applying that logic to public highways and try to imagine the economic impact of Road Monopolies), but it is a widespread and in many respects crippling meme that has infected much of America.

Re:The UK

[RussGarrett]

If you take a look at the LINX peering matrix (warning: insanely large HTML table), you'll see that JAnet (JNT) a.k.a. UKERNA is peered with many other providers, including UUNet and PSI, so no, I don't think they're going to disappear overnight.

BGP diff

[Cramer]

Before 11am (US/Eastern):

• BGP router identifier XXX, local AS number XXX
  BGP table version is 15846057, main routing table version 15846057
  114307 network entries and 335645 paths using 23170999 bytes of memory
  58200 BGP path attribute entries using 3260264 bytes of memory
  50933 BGP AS-PATH entries using 1314460 bytes of memory
  1 BGP community entries using 24 bytes of memory
  15 BGP route-map cache entries using 240 bytes of memory
  80699 BGP filter-list cache entries using 968388 bytes of memory
  Dampening enabled. 271 history paths, 458 dampened paths
  3 received paths for inbound soft reconfiguration
  BGP activity 396089/9002885 prefixes, 4768424/4432779 paths, scan interval 15 secs
  
  Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
  xxx.xxx.xxx.xxx 4 1239 6691013 160774 15845983 0 0 2d07h 113727
  xxx.xxx.xxx.xxx 4 701 5769835 160662 15846057 0 0 7w0d 110579
  xxx.xxx.xxx.xxx 4 2548 5018626 160732 15846051 0 0 04:34:35 110994
  

After 11am (US/Eastern):

• BGP router identifier XXX, local AS number XXX
  BGP table version is 15847879, main routing table version 15847879
  114302 network entries and 335626 paths using 23169830 bytes of memory
  58207 BGP path attribute entries using 3266312 bytes of memory
  51007 BGP AS-PATH entries using 1317248 bytes of memory
  1 BGP community entries using 24 bytes of memory
  15 BGP route-map cache entries using 240 bytes of memory
  80841 BGP filter-list cache entries using 970092 bytes of memory
  Dampening enabled. 759 history paths, 578 dampened paths
  3 received paths for inbound soft reconfiguration
  BGP activity 396098/9003023 prefixes, 4768452/4432826 paths, scan interval 15 secs
  
  Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
  xxx.xxx.xxx.xxx 4 1239 6691412 160782 15847825 0 0 2d07h 113560
  xxx.xxx.xxx.xxx 4 701 5770381 160670 15847879 0 0 7w0d 110410
  xxx.xxx.xxx.xxx 4 2548 5019004 160740 15847859 0 0 04:42:31 110823
  

That's about 170 routes lost. That doesn't look too bad. However, that could be 170 /16's or shorter lost.

What is going down, Ebone or KPNQwest?

[xmedh02]

What puzzles me is that they talk about Ebone all the time.. And that Ebone and KNPQwest networks never fully integrated. Which is strange, they may have not ingerated physically (i.e. switch to KPNQwest's Eurorings fibre network instead of leased capacities old Ebone was using), but Ebone's autonomous system 1755 has been merged with KNPNQwest's autonomous system 286. I have really no idea what consequences will have shutting the power off on the Ebone part.

My other comments:

50% of European Internet traffic carried via Ebone+KPNQwest is way of an overstatement. I don't believe it's that much.

Also, lot of the European daugther companies of KPNQwest, such as
Eastern European division has not filled bankruptcy protection yet, and have their connectivity backed up via other IP transit providers. (The Czech KNPQwest+GTS use BT and SprintLink via GTS Hungary.)

Right now, it's 17.13 and Ebone still seems to be alive, even in Beligum and Netherlands:

3 inway.k.telia.net (193.45.9.49) 7.994 ms 1.361 ms 1.428 ms
4 213.248.76.153 (213.248.76.153) 9.360 ms 1.657 ms 8.429 ms
5 ffm-new-b2-pos1-1.telia.net (213.248.76.141) 20.434 ms 25.261 ms 26.440 ms
6 hbg-bb1-pos3-0-0.telia.net (213.248.64.173) 46.174 ms 50.120 ms 45.800 ms
7 kbn-bb1-pos2-0-0.telia.net (213.248.64.29) 51.553 ms 57.683 ms 52.923 ms
8 adm-bb1-pos0-1-0.telia.net (213.248.64.18) 63.009 ms 69.511 ms 63.573 ms
9 adm-b1-pos1-0.telia.net (213.248.72.2) 56.644 ms 54.084 ms 54.100 ms
10 r4-PO3-1.Ledn-KQ1.NL.KPNQwest.net (134.222.249.77) 66.515 ms 67.437 ms 69.472 ms
11 r3-PO6-0.ledn-KQ1.NL.kpnqwest.net (134.222.229.122) 68.316 ms 67.024 ms 72.428 ms
12 r1-Se0-1-0.ledn-KQ1.NL.KPNQwest.net (134.222.230.5) 56.098 ms 58.623 ms 56.102 ms
13 nlams0605-tc-p6-0.kpnqwest.net (213.174.71.21) 57.973 ms 57.703 ms 58.233 ms
14 nlams0910-tc-r5-0.kpnqwest.net (213.174.69.179) 56.918 ms 59.852 ms 60.050 ms
15 bebru0421-tc-p3-0.kpnqwest.net (213.174.70.113) 60.133 ms 59.639 ms 60.028 ms
16 bebru408-nc-r1-0.be.kpnqwest.net (213.174.69.107) 75.493 ms 73.059 ms 73.376 ms
17 beXPL001-1-s0.cust.kpnqwest.net (213.181.136.101) 77.309 ms * 75.688 ms

2 rib-off.inway.cz (212.24.132.65) 8.421 ms 0.993 ms 1.131 ms
3 inway.k.telia.net (193.45.9.49) 8.760 ms 8.802 ms 1.217 ms
4 213.248.76.153 (213.248.76.153) 6.470 ms 1.763 ms 9.104 ms
5 ffm-new-b2-pos1-1.telia.net (213.248.76.141) 28.248 ms 24.308 ms 20.823 ms
6 213.248.68.86 (213.248.68.86) 20.762 ms 25.753 ms 26.480 ms
7 r5-PO1-2.Ffm-IXA1.DE.KPNQwest.net (134.222.249.89) 45.512 ms 40.712 ms 39.720 ms
8 defra0228-tc-r12-0.kpnqwest.net (213.174.68.17) 47.529 ms 38.075 ms 41.792 ms
9 nlams0921-tc-p4-0.kpnqwest.net (213.174.70.90) 51.055 ms 46.508 ms 52.263 ms
10 nlams0910-tc-r5-0.kpnqwest.net (213.174.69.179) 38.279 ms * 44.290 ms

Re:How are 2 dead companies better than one?

[ch-chuck]

I have to disagree - w/o some kind of competition, it's just not capitalism anymore, IMHO. Hmmm, I'd sure like to see Ayn Rand and Karl Marx on the Jerry Springer show ;))

Anyway, once a market gets monopolized it's all over - they have no real incentive to innovate, improve effeciency; all they have to do is get fat dumb and happy, collect extortion fees from captive customers and enjoy the good life on the golf course (cf. ATT - the phone system started out long ago as many independent small phone systems, eventually competing in long dist, etc, but by the late 60's and 70's had stagnated a very important industry, etc.) Gobbling up the competition in a free for all laissez faire capitalist system is a poor, temporary substitute for getting the real job done, redefining business plans, and ultimately doesn't benefit the consumer. Makes the current balance sheet look good tho, but is just postponing the inevitable toward an even bigger collapse later.

Re:NO FREE BANDWITH

[mborland]

Although your point is a little vague, it seems to mean 'tough sh*t, consumers couldn't cough up for bandwidth.'

You're playing into a common fallacy, which is that the only reason a company can go broke is if they didn't have enough demand. Well...actually businesses go broke for many reasons. Businesses make decisions outside of demand that affect their health. Particularly in a more monopolistic situation, it is hard to argue that there's a sufficient market to distribute the risk of bad decision-making.

For example, maybe they thought the bandwidth need was going to be 50% greater than it was, and though they could accommodate 100% of the traffic profitably, they scaled their business out of reach and it came down like a heap of bricks. In that situation, the demand could have been met profitably, but bad decisions caused them to fail completely--regardless of ability to meet demand, or for customers to pay.

Demon, large UK ISP, response to customers:

[Elphin]

Demon have just sent out the following reassuring email:

• "You may have seen reports in the press that Ebone (one of the European backbone networks) is being closed down at 16:00 BST today following the insolvency of KPNQwest.
• 
  
  Whilst we have been aware of this possibility for some time now our networks have been designed to take account of this type of event. Naturally, given the issues with KPNQwest, we have recently confirmed that our network can accommodate the loss of Ebone.
  
  We estimate that about 5% of our traffic is currently routed by Ebone and that is mostly to other European sites.
  
  When or if Ebone is turned off we expect traffic to reroute via other connections in London and Amsterdam.
  
  We believe it is unlikely that there are any other networks connected exclusively to Ebone, so we do not expect any destinations to become unavailable in the event that Ebone closes.
  
  In the event that Ebone close it will take time for traffic patterns across Europe to settle down, however we expect the effect will be limited to some traffic following less than optimal routes and occasional hot spots of congestion.
  
  We will of course be monitoring our network for adverse effects in the event that Ebone closes and we will carry out any necessary maintenance should we find such hot spots that do not resolve themselves."

UPDATE: Won't be shutdown today.

[egghat]

See the update at Heise (google translated). Or use the original in German.

Have a nice weekend.

Bye egghat.

OTOH, KQPNQwest *is* still still alive

[Surak]

John Walker's site, fourmilab.ch, is still up and running. fourmilab's ISP is KPNQWest, so I'd have to assume it's still running at this point. As the timestamp on this note says, its' well passed 1700 CET.

Shutdown?

[mnordstr]

traceroute to www.microsoft.akadns.net (207.46.197.113), 30 hops max, 38 byte packets
1 192.168.1.1 (192.168.1.1) 0.489 ms 0.282 ms 0.125 ms
2 ip212-226-*-*.adsl.kpnqwest.fi (212.226.*.*) 0.941 ms 0.924 ms 0.875 ms
3 co1-Fa0-0-KQ1.Hel.FI.KPNQwest.net (212.226.253.1) 7.655 ms 7.773 ms 8.015 ms
4 r2-Ge1-2-0-100-KQ1.Hel.FI.KPNQwest.net (212.226.253.14) 8.109 ms 9.383 ms 8.097 ms
5 r3-Se1-0-0-0.Sthm-KQ1.SE.KPNQwest.net (134.222.119.234) 15.505 ms 16.291 ms 16.217 ms
6 r1-Se1-1-0.0.hmbg-KQ1.DE.kpnqwest.net (134.222.230.149) 33.069 ms 33.293 ms 32.848 ms
7 r1-Se0-2-0.0.ffm-KQ1.DE.kpnqwest.net (134.222.230.109) 41.762 ms 41.708 ms 40.848 ms
8 r1-Se0-3-0.0.ledn-KQ1.NL.kpnqwest.net (134.222.230.13) 44.251 ms 43.529 ms 43.273 ms
9 r20-Gi0-0-0.Asd-KQ6.NL.KPNQwest.net (134.222.96.110) 43.892 ms 45.664 ms 43.990 ms
10 UNKNOWN.KPNQwest.net (134.222.249.118) 45.033 ms 45.174 ms 45.007 ms
11 zcr2-ge-2-0-0.Amsterdamamt.cw.net (208.173.220.130) 44.583 ms * 47.331 ms
12 bcr2-so-2-0-0.Amsterdam.cw.net (208.173.209.197) 46.189 ms 43.965 ms 46.527 ms
13 dcr1-loopback.Washington.cw.net (206.24.226.99) 149.072 ms 133.766 ms 133.792 ms
14 agr4-so-0-0-0.Washington.cw.net (206.24.238.62) 133.727 ms 133.032 ms *
15 acr1-loopback.Seattle.cw.net (208.172.82.61) 197.099 ms 217.579 ms 195.529 ms
16 bpr1.SeattleSwitchDesign.cw.net (208.172.82.7) 197.339 ms 197.426 ms 195.313 ms
etc...

Still going through the good old EuroRings network. I'm a little suspicious about the UNKNOWN.KPNQwest.net, but my connection is fully functional.

Not Being Shut Down!

[ebmedia]

from the ebone forum:

We are pleased to announce the hard-line strategy of the Union-led volunteers at Ebone in Belgium appears to have finally reaped rewards. Sufficient funds have been provided to maintain operations throughout Europe. This vital capital will be used to cover operational costs for 2 weeks. This includes the salaries for 200 people, of which 40 people will be from the Belgian Operations Centre. During this time, we are very confident of reaching a positive outcome to the negotiations which are continuing to find a buyer for the Ebone network.

It can therefore be confirmed that the now passed deadline of 17.00 C.E.T. for a network shutdown, will not be executed.

(emphasis mine)

ITs still hanging on?

[weycrest]

PRESS RELEASE

We are pleased to announce the hard-line strategy of the Union-led volunteers at Ebone in Belgium appears to have finally reaped rewards. Sufficient funds have been provided to maintain operations throughout Europe. This vital capital will be used to cover operational costs for 2 weeks. This includes the salaries for 200 people, of which 40 people will be from the Belgian Operations Centre. During this time, we are very confident of reaching a positive outcome to the negotiations which are continuing to find a buyer for the Ebone network.

It can therefore be confirmed that the now passed deadline of 17.00 C.E.T. for a network shutdown, will not be executed.

A further press release will follow in due course.

Employees of Ebone, with their Unions

Resp.editors. Henri Jean Ruttiens, secretary BBTK Setca

Re:The UK

[AndrewRUK]

Well, according to JaNet's "External Network access Provision" page, there's another 2.5Gbits from JaNet to the US that isn't KPNQ. And, incidentally, my connection to slashdot, from bris.ac.uk (on janet,) doesn't go via any of janet's transatlantic links, it goes though linx and then UUNet. No black holes here, thank you very much :-)

Re:How are 2 dead companies better than one?

[bentini]

cf. ATT - the phone system started out long ago as many independent small phone systems, eventually competing in long dist, etc, but by the late 60's and 70's had stagnated a very important industry, etc

Wow. It's amazing how wrong you are. ATT from the60's and 70's invented UNIX, C, AWK (whence Perl), etc. And that was just in one department. They made so many amazing improvements in CS, Information Theory, etc. Basically, anything that could in some way benefit the largest company in the world.

In fact, if you look at it, most of the world's great research labs (Xerox PARC, IBM, Bell Labs) have been the result of monopolies, not start-ups.

Stupid Question:

[ewhac]

How much does it cost to run a NOC of this size?

Near as I can tell, the recurring costs are power, data pipes, a building lease, and people. The Ciscos may have a maintenance agreement, but they've mostly already been paid for. Of these, the people are obviously the most expensive component, but how many people does it really take when so much is automated?

Like I said, it's a stupid question...

Schwab