KPNQWest Admins Keep Bankrupt Network Running

sebastianw writes "Some of the network administrators from KPNQWest, although they have been (apparently) ordered to shutdown the network, took over control of the KPNQWest NOC. They are trying to keep the network running and keep customers up, regardless of KPNQWest's insolvency. The company warned on Thursday that they would be forced to shut down KPNQwest's entire European data network on Monday unless its customers paid their bills in full immediately." There's a related story on the Register, talking about the possible effect on UK internet access. If anyone needs to hire some network engineers...

Comments

At least it's efficient

[ThesQuid]

It makes you wonder just how many people they really need to run such a network.

It appears to be less than the number they thought.

Re:At least it's efficient

[Sorthum]

For a day or so, you can keep most large networks running with only one person or so. I work in a NOC that handles access for over 2000 sites, and for nights and holidays (weekends too), we have exactly one person here.

Though when things go bad, we have other people on call-- it can get hectic.

So if all goes well, they could hold out for quite some time (until something important breaks).

Re:At least it's efficient

[Guido69]

Likely that's true to an extent, but it's not sustainable. Many large networks routinely run with reduced staff on evenings, weekends, holidays, Star Trek premiers, and off-hours. But there is more to running the network than just keeping it up (maintenance work). What about upgrades or project work? It's hard to move forward with technology if your entire staff is focused solely on maintaining your existing infrastructure.

I'm not saying they likely couldn't trim some fat, but a staff reduction of more than 2/3 cuts too deep.

Re:At least it's efficient

[Sorthum]

It does, but right now they're focused solely on "keeping it up" until the trustees come to a conclusion. The only "project" is to keep the network up so half of Europe doesn't drop offline. I'd imagine all expansions are on hold until they figure out whether they're selling the old equipment on Ebay once the switch gets flipped...

Re:At least it's efficient

[ch-chuck]

Reminds me of when Verizon's staff took industrial action (strike) a couple of years ago for the Communication Workers of America union - one industry wag wrote in Network Computing, "Did anybody notice when Verizons 800,000 employees went on strike? Neither did I". But seriously, no new lines were installed, and I'm sure all the maintenance work would have soon caught up with what little mgmt could do to run such a network.

Re:At least it's efficient

[Anonymous Coward]

Seriously. On the Web cam, most of them are wandering around drinking. Cannot tell what it is, but how long before we see the amazing bottomless paper bags? What are they going to do to them for drinking on the job, fire them again? :)

Re:At least it's efficient

[blair1q]

Which points up the problem.

MTBF is far too short for crummy, overpriced little hardware boxes, and diagnostics still aren't automatic.

With the trillions of dollars that have flowed through the industry, network equipment should be less trouble than a refrigerator by now, but certain (CSCO) companies (CSCO) have had a vested interest (CSCO) in keeping the system unstable (CSCO) and difficult to maintain (CSCO).

--Blair

Re:At least it's efficient

[PurpleFloyd]

Since when has Cisco intentionally made bad hardware? I have found that while their prices are high, it is hard to make the hardware on a Cisco box fail short of running mains voltage through an Ethernet line.

Re:At least it's efficient

[GrenDel Fuego]

>Since when has Cisco intentionally made bad hardware?

Since their hardware crashed if Code Red tried to infect it?

I see an "intentionally" in there. How do you intentionally plan to be vunerable to a virus that hadn't been written yet?

Re:At least it's efficient

[eMilkshake]

What? Our systems work great when the techs are gone. It's only while we're here that things really break.

(yes, usually that's upgrades, but it is a standing joke that things only break when we're here)

Re:At least it's efficient

[ostiguy]

You are silly. Refrigerators suck. They all come with ice makers now. Why? Well, what part always breaks? The ice maker? My mom is going to special order a refrigerator without one - very few of the big box stores sell *any* without - why? Well, the big box stores all have service departments, and are in the extended warranty business right?

Most products today suck. At least computing and network gear has an excuse - widget 2002 is 50 to 200% faster than widget 2001.

ostiguy

Re:At least it's efficient

[uncleFester]

For a day or so, you can keep most large networks running with only one person or so...

I'm sorry, but this kinda comment chafes my ass. Where I worked (I was laid off recently) we had a satellite fiber-optic cable production plant in Canada. My manager once stated in a meeting the place 'was running with only 17 staff...' completely neglecting his IS department in the room that remotely managed, upgraded, troubleshot, etc etc...

'You can keep most networks running with one person' does not jive with 'we have people on call..' the on-call person(s) cannot be discounted so easily. Otherwise, why have them in the first place?

Re:At least it's efficient

[balloonpup]

Well, the only thing I can say to that is that the intentionally doesn't mean much. Pontiac didn't intentionally make the Fiero such that when caught fire, it would emit toxic gas either. With the ammount of Cisco uBRs (the only thing *I* come in contact with of that nature) I see crashing randomly, irregardless of intent, the problem still remains.

Re:At least it's efficient

[blair1q]

Your mom has the right idea. Ice makers are made of cheap parts. But look under the fridge. There's a motor, gearing or a belt and pulley, electronics, and a compressor head, plus a couple dozen yards of tubing. It'll turn on and off twenty times a day for fifty years if you don't screw with them. And there are still ten companies that make money off of them. But CSCO doesn't care. They want high margins for cheap crap and a software paradigm invented by the same guy that gave us colonoscopy.

Re:At least it's efficient

[Cramer]

Obviously, you are unaware (as was I) of the recall field notice for all Cisco 7401ASR's with a model number less that '-10'. They screwed up the L3 cache logic which causes memory corruption (and thus random reboots and even complete hardware lockups.) So much for QA and customer support -- we wen't notified of the notice for over three months.

Oh, and I've seen an ethernet module in a cat5000 suddenly start turning every packet into a broadcast packet. Oh, that was a day in hell.

But, on the whole, the hardware is impressive and stable. The software provided is a very different story.

[See Also: http://www.cisco.com/warp/customer/770/fn18164.sht ml (CCO required)]

Just been on their webcam in the NOC

[Boss, Pointy Haired]

http://195.158.250.186/ebonelive1.asx

You can see the problem, none of them are wearing shirts and ties. If they were wearing shirts and ties none of this would have happened.

I'm being more serious than you might think.

Re:Just been on their webcam in the NOC

[Anonymous Coward]

Great thanks.

Now I'm gonna have to find a way explain "ebonelive" to my girlfriend when she finds it in my recent files list later tonight.

Re:Just been on their webcam in the NOC

[meis31337]

Interesting.... you wonder why they went out of business. An exec probably came down and said, "You guys would look a lot better with flat panel monitors."

Re:Just been on their webcam in the NOC

[maan]

...and Aeron chairs... (you can see it on the picture).

Honestly, I still don't see what the big deal is with that chair. I got to sit on one for a few days. It _is_ preety comfortable, but honestly, a $50 chair isn't all that bad either...

Maan

Re:Just been on their webcam in the NOC

[Manitcor]

That's probably why they decided to stay in the NOC they aren't getting paid and the aero chairs are more comfortable than the curb in front of their apartment building.

Plus they can use the Internet access to do job hunting and play really fast QIII. (What you actually thought that they were maintaining the network?)

Actually I think it's great to see such dedication.

Re:Just been on their webcam in the NOC

[Sorthum]

No kidding. The NOC I work at has dual monitors on most machines here, the way it's been working is one is a flat panel, one's a CRT.

But we're state sponsored, so it's the taxpayers' money. Not sure how economical it would be if we were a corporate entity.

Re:Just been on their webcam in the NOC

[SerialHistorian]

Heh, watch /. crash the whole infrastructure by hitting the webcam all day...

Re:Just been on their webcam in the NOC

[Cmdr Taco (luser)]

The only person who seems to be doing any work is the big guy with the beard.

Ah, hell. He's probably just reading about himself on /.

Re:Just been on their webcam in the NOC

[Art Tatum]

The only person who seems to be doing any work is the big guy with the beard.

That's because the 'big guy with the beard' is a true UNIX guru. The beard part gives it away.

Re:Just been on their webcam in the NOC

[ReTay]

You know I was wondering about that. CNN (I think) said that the NOC carries between 33 and 50% of the traffic in Europe. Want to teach the ones that don't get it why they are always hearing the word redundancy from the network people. (Shrug) Bet they would be back there as soon as the dingbat judge found out his day trading access was shot to hell. Last one out turn off the lights and AC errr just the lights.

fat chance...

[vitalidea]

"unless its customers paid their bills in full immediately."

Fat chance that will ever happen. People in the US can't even their bill payment service bills on time. =)

How?

[Sorthum]

It's great that they're doing this, but how long can they keep it up? I mean, it's great that they're volunteering like this, but you can only go so long without a source of income. After a few weeks, bills start to pile up, or a major router breaks and needs replacing. Running a network's not ridiculously hard, but it does take money...

Re:How?

[mnordstr]

They only need to keep it running for a short time, KPNQwest is trying to find a buyer for the network. Let's hope it finds one, as the european Internet will suffer a huge bandwidth shortage if the network goes down.

As a KPNQwest customer myself, I'd like to say that they've provided a great service over the years. This is absolutely the best ISP ever, no real outages, great customer service (as you might have noticed). It's a real shame it turned out like this.

Re:How?

[Indras]

As a KPNQwest customer myself, I'd like to say that they've provided a great service over the years.

Must be the flat panel monitors and Aeron chairs.

Re:How?

[^DA]

True, but the little KPNQwests are being bought by other companies. In Norway the Norwegian Branch of KPNQwes was bought on thursday and is running smoothly witn new owners. I guess the same is happening in other contries too.

Re:How?

[matt4077]

> as the european Internet will suffer a huge bandwidth shortage if the network goes down. Because of the increase in channels per line, the network usage is at an all-time low of about 5% right now, that's why they're out of money in the first place. Bandwidth is not a problem. Routing might be, but that should only last for a few days.

noc admins

[meis31337]

I think this is great. Really... Big business could care less about their customers in recent years, but we have these guys generously keeping customsers online...even after they aren't being payed.... This is how customer service SHOULD be done. It is ashame that this is the EXCEPTION and not the true state of an industry in peril.

Re:noc admins

[Indras]

Or... this could be another case of a person doing what's best for themselves is improving things for everyone. I mean, this their job, it's what they get paid to do. Unless, of course, they aren't receiving a paycheck for this... in which case...

QUADROUPLE KUDOS TO EVERYONE THERE!

Re:noc admins

[Manitcor]

They were fired on the 6th, yet they stayed becasue they were the only ones left to support the network.

Re:noc admins

[Jonny Ringo]

Right, but what award will they recieve for their efforts? Their company doesn't seem to care, so their company obvously won't reward them (they're going against orders). Customers, obvously like this but appears that a lot of these customers aren't even paying their bills.

If I was a admin for that company, I'd be like fuck it.

Re:noc admins

[hollow_man]

Let's not forget that whilst the guys are fighting to save the main NOC and keeping it running, other temporary NOCs have sprung up and the old KQ NOC in The Hague has also been resurrected.

We're all working very hard to make sure that disruption to our customers stays limited to the absolute minimum and all techies in this company have now shown to management that they can make their stand in the darkest hour of this company.

Needless to say I'm proud to work with these guys.

Re:noc admins

[screenbert]

I would like to think that I would be that noble. If my company were shutting down and had taken good care of me, I might continue to work until I found another job. There might also be the thought in the back of my head about being rewarded if things were to turn around... Regardless I think that it would only be beneficial to the Engineers. They keep working, so even if they are asked about that job, the fact they are still working after being laid off says a lot to another employer about their character. Meanwhile they have a high-speed internet connection for sending out those resumes.

Screenbert - Thou wenching pottle-deep whey-face!

Re:noc admins

[blue trane]

no matter that noone would ever do the shit jobs if everything was volunteer based

hippie coders will write the code to automate all the shit jobs.

Re:noc admins

[cpeterso]

Big business could care less about their customers in recent years

I think you mean: "Big business could NOT care less about their customers in recent years."

Re:noc admins

[uglyduckling]

It depends.... I reckon that there are some great companies out there that would jump at the chance to hire people with that sort of attitude.

Re:noc admins

[uglyduckling]

Guess you're not from the US! I'm British but have a few US friends. Certainly amongst my friends "could care less" is a standard saying and means exactly the same as "couldn't care less". I found it bizarre for a few weeks but I'm used to it now.

Maybe you are American and it's just slang from a certain city or state or something.... I dunno!

I think they just found out...

[ceejayoz]

It looks like they just found out they're on slashdot... they all started looking at the camera and something on the computer screens, lol.

Re:I think they just found out...

[garcia]

now only if we could have smell-a-vision...

Burning and melting hardware and then the distinctive odor of shit in their pants.

;-)

Re:I think they just found out...

[Strog]

A couple guys were waving at the camera when I checked it. They must realize someone is watching them. They might not know it is /.

in the end, it was Slashdot that ended them....

[bbh]

It would be funny if customers wondered why it was finally shutdown only to find that it was the Slashdot effect that took down the network. We can only wait and see...

bbh

Potential european outtage map

[sniggly]

Its good these guys keep on working even though they havent been paid for a long time (some in Holland since may). If KPNQwests network goes down well.. here is their map. Too bad most of the news around this is in english.

Re:Potential european outtage map

[sniggly]

I mean in dutch - rtlz is doing the story in dutch.

There is an emergency plan in place that would keep them running until the end of the month if customers pay their june fees in advance.

If you have a lot of money and want to have a state of the art network with REALLY dedicated people now is the time to buy!!

Re:Potential european outtage map

[pipsqueak]

This map is not a real reflection of what would happen should KPNQwest cease functioning.

I think what the KPN noc guys are doing is great and it says a lot about their service... but being more ruthless and thinking about it in terms of business, there is a huge glut of fibre at the moment (as all telcos bought into the extravagent growth predictions being touted at the time of the internet boom) therefore bandwidth is cheap and most ISP's & telcos are selling their services around cost or even at a loss at the moment just trying to compete with each other (i.e. price war). This means current business models aren't sustainable and we're seeing lots of telcos & ISP's go bust, including KPNQwest & remember...

... when they do, the fibre is still in the ground waiting to be used by someone else. KPNQwest going bust doesn't mean their network disappears forever. It only disappears for as long as it takes the customer's to shift to alternate providers and the alternate providers to increase bandwidth to cope (if they even have to), including buying up portions of KPNQwest's network.

Serious customers would already have alternate providers... people running their business off their internet connection who don't have an alternate provider... well, that's just plain dumb.

The Ultimate Playground

[Caradoc]

Hell, if it weren't for my need to eat, I might put in some free time in a NOC like that one, if for no other reason than to have access to the hardware.

Call it the ultimate training lab. Make any change you want - nobody can really bitch much about it, since they're not paying for it anyway. The only motives to keep things running are to, well, keep things running...

Re: The Ultimate Playground

[Omniscient Ferret]

I've been considering something similar, just for the experience. Specifically: On the baylisa mailing list, David Weekly noted that he'd "founded the California Community Colocation Project, a 501c3 non-profit whose mission it is to provide virtual and colocated Internet access for individuals and non-profits. ... If you'd like to help, we'd love your help! You can find out more about us at http://CommunityColo.net/"

Internet COULD be free

[mnmn]

Take some powerful countries, like France, Britain Japan, allocating funds for high speed backbones whose bandwidth is provided and distributed by volunteer organizations or other NGOs in the country, distributing mainly by the districts population. Now take laws that say a telcom company should allow DSL for free thru its network by its phoneline customers. Also take government policies that update the speed of the bones every three or so years to the several adjacent countries.

Naturally other nations would follow suit seeing the economic improvement such communications allow. Government run backbones would be kept up to speed and private competitors would be hard pressed to provide higher speeds and lower costs. Once the threshold number of countries have this, the countries that started it would be under pressure to continue providing the service, thus a positive feedback can be attained that sustains Free Internet. Apparently all we need is a few smart politicians in a few powerful countries.

Re:Internet COULD be free

[Anonymous Coward]

For an example of how efficient governmental run organizations are, see the US Post Office, or the ex-Soviet Union's agriculture ministry. Brilliant idea.

Re:Internet COULD be free

[unclelib]

How is this free? This would be a service paid for with tax revenues that come out of your pocket. Theres no such thing as a free lunch!

Re:Internet COULD be free

[NevarMore]

Whats wrong with the US post office?

for 34 cents i can send pages across the country in about 3 days. theyre usually cheaper than ups/fedex. 50 cents for a postal money order vs my banks $3 money orders.

Re:Internet COULD be free

[Brightest Light]

Riiiight....

And as an added bonus, the gov'ts will not have to fight with ISP's in order to get carnivore, echelon, and whatever else on the network.

Re:Internet COULD be free

[rmadmin]

This would suck.

Naturally other nations would follow suit seeing the economic improvement such communications allow.

Economic improvement? WTF?! DSL companies keep going bankrupt charging money for their service, if its 'free', does that mean the government is gonna go bankrupt?

Though if the telco's would back the fark off, that would help. Anyways, A freely connected internet would NOT be good. Your killing LOADS of income for alot of companies (small and big). Competition is bad enough the way it is. We dont need that crap out there.

Re:Internet COULD be free

[batkiwi]

"Governments" get money from TAXES.

My tax dollars going to "free" broadband does NOT make it free.

What you meant to say is "Internet COULD be socialized"...

Re:Internet COULD be free

[dada21]

The Post Office is HEAVILY subsidized by tax dollars, even though they say they are self sufficient.

There is a law that says that NO ONE can compete with the Post Office offering 1st class mail. There used to be a law that said no one can offer next day service, and when that law was overturned, look at how many companies popped up, and are employing people off the government dole.

Socialists bug me.

Re:Internet COULD be free

[Strog]

Give me your fax number. I want to fax you a birthday card with some money in it. I'm thinking a few hundred.

Re:Internet COULD be free

[blue trane]

Theres no such thing as a free lunch!

If even universal constants can change over time, perhaps this truism can too?

Re:Internet COULD be free

[Cramer]

And let's not forget how much money they make every year and yet still increase the price of stamps almost yearly. My personal favorite was the bullshit remark of increasing to 25 cents so people can "buy a stamp with a single coin."

It's a lot like the "touch tone" service fee charged by Bell South for almost 30 years. I remember when they started that to "pay for upgrades to the phone system" to support touch tone. They continued to charge that fee until a few years ago.

Re:Internet COULD be free

[raju1kabir]

The Post Office is HEAVILY subsidized by tax dollars, even though they say they are self sufficient.

Let's have a cite for that. Otherwise I say you're blowing smoke. The only thing that could even vaguely be termed a subsidy would be existing depreciated assets that were not funded from revenues at the time of purchase. That's not an ongoing subsidy and the only "cost" is the opportunity cost of not retroactively remunerating the treasury.

There is a law that says that NO ONE can compete with the Post Office offering 1st class mail. There used to be a law that said no one can offer next day service, and when that law was overturned, look at how many companies popped up, and are employing people off the government dole.

There's a specific reason for that. The USPS is able to provide flat-rate service nationwide because money-losing rural routes are subsidized by highly efficient metropolitan deliveries (i.e., all the mail within New York City that costs the same to send as a letter from the Aleutians to the Bayou).

This situation is maintained as a policy choice; the government has (wisely) decided that a baseline communications infrastructure is essential to running a country.

Open it up to the market, and companies will move in to the profitable areas, ignoring the rest. FedEx can't make money delivering 35-cent letters once a week to Zeke in his mountain cabin. Nobody can. But if we want to keep Zeke within the fabric of our society, we have to accommodate his choice to live there.

Socialists bug me.

Tinpot anarchists amuse me.

Re:Internet COULD be free

[raju1kabir]

And let's not forget how much money they make every year and yet still increase the price of stamps almost yearly.

Apparently the schools aren't as efficient as the Post Office. Pay attention much in math class? Ever hear of something called inflation?

The price of a stamp has remained pretty much exactly constant over the past 30 years.

In 1971, a first class stamp cost $0.08. Plug that into the Inflation Calculator and see what you get.

Webcam!

[unclelib]

The coolest part of the article was the link to the live webcam feed at the bottom! There is some pretty interesting stuff going on at that place! I don't know if anyone caught this, but I'd swear one of the techs was drinking beer from a longneck bottle! Another time there was a a guy making faces into the camera. These guys are way cooler than my coworkers!

Re:Webcam!

[Indras]

These guys are way cooler than my coworkers!

It's the lack of ties around their necks. They have oxygen going to their brains. Makes people friendly and creative!

Re:Webcam!

[xanthus]

>These guys are way cooler than my coworkers!
It's the lack of ties around their necks. They have oxygen going to their brains. Makes people friendly and creative!

Shhh! Don't tell my boss or he'll start wanting me to go business casual! Then he'll raise the creative and productive bar.

You just can't let secrets like this out into the general public, man! What were you thinking?!

:lol:

Re:Webcam!

[(void*)]

Those strips of cloth are called nooses!

Re:Webcam!

[Ozymandias_KoK]

Jeez....did nose just go the way of lose?

Re:Webcam!

[Ozymandias_KoK]

While you're at it, look up the definition of pun. :)

You fool!

[stefanlasiewski]

You can't wear a tie in a shop like that!

Just think of what could happen if your tie got caught in the bitstream!

*ZOINK* there goes access to Poland!

Wedding rings? *ZAAM* Oh damn, I think we just transported Roger's ring finger into the shared computer in Ms. Kingston's 3rd Grade class in Manchester, UK.

You gotta be careful when working around heavy data processing machinery, man! They aren't safe or simple, like a table saw or anything!

How bad will it be?

[Nighttime]

I thought the Internet routed around damage?

I'm sure I read somewhere that during the Gulf War the Allies had problems trying to disrupt Iraqi communications as the Iraqis were using a TCP/IP-based network similar in configuration to the Internet.

If the plug is eventually pulled on the KPNQWest network how drastic will the effect be?

Re:How bad will it be?

[ceejayoz]

Yes, it is designed to route around damage, but this isn't damage - it's the whole network going down at once.

Take a look at this map and you'll see the problem...

Re:How bad will it be?

[Manitcor]

Yes, the basis is that there will be routing around the damaged portion of the network. However because they are a major backbone provider the other links in Europe will not have the capacity to handle the extra traffic on top of their current traffic.

Just like in a big city, sure you can get around without the major highway but without it, it takes 2 - 3 times longer to get there (Unless your in LA).

You also have the possibility of overloading the other links and then almost no one in Europe could get online.

Re:How bad will it be?

[Jacer]

very bad, those pipes will have to take other, less efficent routes, less bandwidth handeling more traffic, and then they may have to pay x#of$ for the amount of the pipe used, there's a lot of administrative cost when not using your own autonomous network, and as for the tie scenario, i'm sitting at work now in a t-shirt and jeans, with a pair of vans on, same thing every day, well, no not like that, i have more than one tshirt and one pair of jeans, but that's the style of clothing i wear, no really, i mean it, yeah so what if it's what i wore yesterday, i did the laundry last night, i swear! ehhh, what smell? oh bloody hell...shower? yes, i showered this morning, or was that yesterday morning, i think it was yesterday, but it could have been monday....monday....the 28th, of january.....

Re:How bad will it be?

[AstynaxX]

To the newly disconnected customers, very VERY bad.

Re:How bad will it be?

[GlassUser]

This is why we do not want backbone monopolies.

Re:How bad will it be?

[Artifex]

I thought the Internet routed around damage?

Yes, but first you have to get to "the Internet" itself. Taking down KPNQWest's backbone will take down any customer who is not multi-homed; that is, those who do not have transit through other providers besides KPNQWest.

Oh, and for those customer who are multihomed, they're going to see surges in latency & packet loss as their other providers try to keep up with the demand from all their, uh, "peers." =)

Question

[Sorthum]

Assuming that they go down, how much of an impact will it have?

The story so far...

[hollow_man]

The actions of the guys in Brussels have been an inspiration for the rest of us in the company (those that are left).

I can pretty much assure that the UK network will keep running, definitely one of the main AS numbers of the UK (5571) has alternate transit so do most in-country networks. If anything it's the Eurorings that are at risks but even they are supported right now from Brussels AND The Hague (both IP NOCs are operational right now).

The actions of the last few days make me proud to be part of KQ, Ebone and Netcom and I really hope that we can all survive this crisis, and if we don't, it won't be from the lack of trying or courage.

just another Netcom/Ebone/KQ techie

Re:The story so far...

[killmenow]

I really hope that we can all survive this crisis, and if we don't, it won't be from the lack of trying or courage.

Of course not...it will be from the shite-heads in management making all the lame-ass decisions that got you so fucked in the first place.

As always.

Re:The story so far...

[Subcarrier]

it will be from the shite-heads in management making all the lame-ass decisions that got you so fucked in the first place.

Do you mean to say that a competent management would have recognized the decreasing demand and sacked half the work force long before it got this bad?

Re:The story so far...

[killmenow]

Not necessarily. I mean to say "competent management" implies an ability to manage ... in other words, to handle problems before they get out of hand. So, naturally, competent management would have made better decisions and kept things in line, avoiding the fscking of the employees and customers.

But, let me ask you this: Would it be better for half the employees to be out of work than all of them?

Re:The story so far...

[Subcarrier]

"competent management" implies an ability to manage ... in other words, to handle problems before they get out of hand.

That's the theory. All I can say is never go into management, unless you REALLY like the money. During an economic downturn it's the shittiest job I can imagine.

But, let me ask you this: Would it be better for half the employees to be out of work than all of them?

Depends on which half you ask. If you mean, which is better for the company, the answer is self-evident.

Re:The story so far...

[da_Den_man]

Management maybe the shittiest job you can imagine, but at least they still get PAID.

Re:The story so far...

[killmenow]

During an economic downturn it's the shittiest job I can imagine.

I am in management. Our company is unlike many other businesses at present.

We are actually growing right now and have been for the last few years...in spite of an economic downturn. This is for two reasons:

1. Competent management - at least for now <grin>
2. We are small enough (~$20M US annual revenue, 100-150 employees) so as to remain agile and mostly devoid of bloat.

As an example, we recently had an all-hands afternoon team-building exercise with free food and drink for all. One person dropped and broke a glass bottle. Our CEO cleaned it up. Get that? Not the janitor...not the guy who broke it...the CEO got out a broom and dustpan and cleaned up the mess.

We all work hard, and nobody is above the small stuff.

Re:The story so far...

[aallan]

...As an example, we recently had an all-hands afternoon team-building exercise with free food and drink for all. One person dropped and broke a glass bottle. Our CEO cleaned it up. Get that?

I'm more concerned that you used the phrase all-hands afternoon team-building exercise without cracking up.

Al.

Payments

[Anonymous Coward]

Hmmm,
I find it interesting that they seem to be having trouble with users paying them. The most serious problem I face is customers not paying for the services they receive.

Just shut down one site 2 months ago which was overdue by 6 months... Sortly thereafter got a call berating me for their website not being up. I pointed out that they were past due and stated the site would not go back up until they paid their bill... Couple of days later I had a check.

It supprised me since the site has always been a problem with paying late. I've finally decided to take a no nonsense approach to billing rather than trying to work with customers. I've had too many customers just walk away without paying their bills and thus I have been shedding customers during the last couple of months.

It's definately a lot easier now, without maintaining a bunch of sites that don't pay.

Re:Payments

[nelsonal]

You can't get blood from a stone, its tough to collect unsecured debts from bankrupt companies. I doubt that the collections problems are from companies that are still in business, its likely that they extended credit to any and all willing buyers during the good times, and now are trying to collect from these bankrupt companies which entails lots of court visits and negotiations with all the other creditors. They would have extended credit to show revenue growth to their owners and the public market analysts, which drove their increaseing stock price in 2000. Its sort of amazing the complete difference between then and now, the equity alone was worth several billion then, and now no one is willing to pay $200 million for their network assets.
Someone is going to become the next Standard Oil by scooping up all of these destressed telecomunications networks nursing them back to life over the next decade. Our kids will revile them as much as we do to Microsoft.

Re:Payments

[Codifex Maximus]

I agree; get rid of clients that don't pay their bills. I had that same idea quite a few years back when I was delivering papers (don't ask). Anyway, I cut the non-paying subscribers after many diligent attempts to collect. I was trashing alot of extra papers every day (I had to pay for these extra papers you know) and wanted to cut down on wastage and increase my profit margin.

I ended up getting fired by my boss for having too low a subscriber base. They didn't care that the people weren't paying... they just wanted the subscriber numbers to look good for advertising revenue.

Worst job I ever had! But I'd cut those non-payers again without a second thought if I was in the same position again.

Bottom Line: In my book, you don't pay... you don't play.

Finnish KPNQwest standing by to re-route

[Judge_Fire]

Interestingly, our local KPNQwest is leaving the mothership and this seems to be happening elsewhere, too.

On the Finnish company's page they state they're ready to re-route traffic and do what it takes. KPNQwest Norway got bought by Catch Communications and this might be the fate of the others, too. Seems Nokia is already on board as a customer.

http://www.kpnqwest.fi/yritys/uutiset/uutinen200 20 607.shtml (in Finnish)

Anyone got more on this?
J

Re:Finnish KPNQwest standing by to re-route

[hollow_man]

A lot of us local entities are preparing to leave the mothership. Netcom UK is about to go solo again after 2 years under the GTS/Ebone/KQ banner.

KQ portugal and italy are going standalone as well I believe.
We're now all waiting to see who will pick up the network so we can buy cheap transit from our former parent ;)

Seriously though, even though we're going our separate ways, right now we're still one company (apart from KQ NO that got sold already) and we're still running the network, even if we have to do it without the assurance of being paid.

Re:Finnish KPNQwest standing by to re-route

[Subcarrier]

we're still running the network, even if we have to do it without the assurance of being paid.

Two thumbs up! I would buy you a beer if I could.

Re:Finnish KPNQwest standing by to re-route

[t-man]

It seems the same applies to KPNQwest Estonia.

http://www.kpnqwest.ee/uudised.php?ID=65 (in Estonian).

It states that the customers will not be plugged out and they're setting up the necessary backup routes.

It is also stated that some HQ supported services (such as roaming) will probably discontinued or reorganized.

Re:Finnish KPNQwest standing by to re-route

[frankske]

i will next time he is in .be ;-) Hollow, give me a call when you are around ...

Re:Slashdotting.. Blah.. Blah..

[fruey]

I'm totally with you on that. Posts about "Star Wars speak" type jokes (this is not the NOC you're looking for), Beowulf clusters, and the "Slashdot effect" are systematic and just not funny, and yet currently this comment is the highest rated of all.

However, I find that being a bit more poetic than "Shut up you Troll" works better. ;-)

I hope some companies repay the admin with jobs

[f00zbll]

that's a lot of dedication and persistence. I hope all the guys that are volunteering get good jobs else where.

Shows what he knows...

[JaredOfEuropa]

one of the techs was drinking beer from a longneck bottle!

Same beer as the regular kind but less of it, at a higher price. Longnecks, pfft.

Hiring network admins

[AnalogBoy]

Sorry, but they probably will need to stand at the rear of the "I need a job" line, like the rest of us.

However, i do admire their dedication to their jobs. It's either customer loyalty or "Hell no. Not again. HELL NO." Mentality of losing a job.

:)

Re:Hiring network admins

[Artifex]

Sorry, but they probably will need to stand at the rear of the "I need a job" line, like the rest of us.

Is this where the line ends? I just got laid off on Friday =)

However, i do admire their dedication to their jobs. It's either customer loyalty or "Hell no. Not again. HELL NO." Mentality of losing a job.

Yah, well. The market sucks for admins in general right now, and at least while they volunteer to stay on the job, their skills are kept fresh. Plus, think of all the free soda they would miss! =)

No, seriously, system admins can build their own boxes at home (or buy cheap discarded Suns) during their downtime, and keep practicing and learning new stuff. It's a bit harder for network engineers - most of us can't buy multiple Ciscos big enough to run BGP and ISIS on, much less any Junipers, etc. Staying with the toys until they are made to go home is a decent plan.

Besides, we don't know the full labor situation there; there was mention of the government and labor unions in one of the articles, and so they may be retaining some legal status by staying there.

Re:Hiring network admins

[AnalogBoy]

Yeah, the line ends about three months back thataway. You can get "Skip ahead free" cards if your college educated with gobs of certs, 10+ years of experience, a great skill set and a willingness to work for $cheap. You can get a job right off the bat if you have a TS/SBI. SO MANY FSCKING PEOPLE want security clears right now. I want to smack half of them. It's more or less a military-only club out there.

Do the dying dot-com thing and take home the switches and rouers you need. Replace them with 3COMs & Linsys's (Linksii?) from Best Buy. :)

No company, but still running?

[hkmwbz]

So what would happen if KPNQWest customers continued to pay the people spending their time to maintain the network? Couldn't they in theory continue to run the network for the time being, even though the wages would be terrible and the work would be hard and stressful? They would at least get money...

The ones still there seem to be a dedicated bunch of people, and I wouldn't be surprised if they managed to build something from what they have now.

Although reality would probably rear its ugly head, and they would be forced to shut down anyway. Everything is nice in theory, I guess...

Re:No company, but still running?

[uweber]

Problem is, traffic costs money and if those ISPs who peer with KPNQuest decide to pull the plug the network might still be running flawlessly but of no use anyway. Tough luck that is.

Not Linux, Cisco and Fiberthings and HVAC

[billstewart]

The parts of KPNQwest that everybody's worried about these days aren't running either Linux or Windows - they're routers (probably Cisco) and fiber muxes of various sorts (which do have OSs for their configuration and user interfaces) and the air conditioners for the rooms the equipment runs in. Both non-HVAC categories are pretty reliable as long as there aren't technicians adding or changing things and as long as nobody backhoes any important circuits.

There are other parts of KPNQwest that do run on general-purpose computers, such as the administrative and billing databases, network management systems, etc. Probably some are running Windows or Linux, and others are probably running Solaris or HPUX.

Female, pretty

[the grace of R'hllor]

Just an observation.

What's a nice girl like her doing in a place like that? (cleaning up/out, obviously, which is perfectly fine in this gender equal world, not to say that women are identical to men, nor superior, nor inferior, but on a level that is as high, though not higher, and certainly not lower, than that of the men who are browsing the internet in the background.)

Anyway, I'd like some nice person there to knock that orange cap on the ground for me, about a meter further away from the camera.

Hmm.

What do you mean, 'customer service' does not equal 'fan service'?!

Anyway, kudos on the operation to everyone there.

Re:depends on the architecture, really

[nelsonal]

Actually it takes 0 to fly an airline jet. The computer can do it, if there are no problems. However because problems are not scheduled and computers are not to the level where they can adapt well, people are still needed for emergencies.

Re:oops, scratch that Nokia.

[Judge_Fire]

LOL, misread Norja (Norway) for Nokia. You really read best what you read most.

Then again, might have been accidentally correct, but whatever :P

J

Operated by a man and a dog

[billstewart]

The standard joke about telco operations is that the network is made to be run by a man and a dog:

• The man's job is to feed the dog.
• The dog's job is to keep the man from touching any of the equipment.

As you and others have said, if nothing goes bad, it doesn't take much maintenance to keep running, though it's adding new service that needs resources and causes mistakes that break things.

Re:Operated by a man and a dog

[Sorthum]

As you and others have said, if nothing goes bad, it doesn't take much maintenance to keep running, though it's adding new service that needs resources and causes mistakes that break things.

Yeah, but you should SEE how many people it takes to fix things when it goes wrong. We had a guy in a pickup truck driving a new mainframe processor assembly from Boston to northern Maine one day just to get things up again...

Here's to network stability in Europe for the next week.

watch the live stream on linux using xine

[gooofy]

xine mms://195.158.250.186/EboneLive
make sure you have the necessary win32codecs installed, though.

Re:watch the live stream on linux using xine

[demon]

I had to run xine -p 'mms://195.158.250.186/EboneLive?.asf' to make it work. However, I'm on a PPC Linux system (Debian on a blue/white PowerMac G3) so no Win32 codecs, but it streams just fine - it's apparently in DivX format anyway.

KPNQwest's bankruptcy is already causing problems

[OblongPlatypus]

I run a high-traffic web site based close to Oslo, Norway, and I've been getting reports of problems from my users since the day after the bankruptcy was announced. My users are from all over the world (about 50% from the US), and at any given time there seem to be several different groups of them who can't connect to the server at all. The problems are usually temporary, and the users affected seem to be spread across the world. So far the most consistently affected users seem to be those from the Netherlands and Singapore (.nl seems logical, but why .sg of all places?).

We were leasing our lines from KPNQwest Norway, so that might explain some of it, but I suspect anyone who's trying to connect to anything through KPNQwest's backbone will be experiencing intermittent problems if things aren't resolved soon.

Freeserve could be okay?

[Xcrap]

According to the register, Freeserve won't be affected, which is good because thats the ISP ive been using. They provide a really fast connection for a modem (which is good) so it seems that those AOL lusers will get punnished for using a rubbish provider.

Re:depends on the architecture, really

[blue trane]

"computers are not to the level where they can adapt well"

why not?

if all of us out-of-work programmers could write the ai...we could replace ourselves completely or free up more time for unabashed hedonism.

AT&T story

[ehiris]

I wonder what happened to this

Most of the people just left the NOC

[Animats]

I was watching the webcam of their NOC. There was a visible protest, with people holding up signs in front of the webcam. A few minutes ago, somebody in a suit came in and spent a few minutes talking to people. Then most of the people picked up their stuff and left. A few people remain, quietly sitting at computers.

I haven't been paid either

[swb]

I've haven't been paid since May, either, and I suspect that most people haven't been paid since May.

Oh. I bet you didn't mean bimonthly payroll...

Re:I haven't been paid either

[sniggly]

I read the article a few days ago and justtyped whatever was in my mind (last month!) stupid innit. Mustve been march because when i read it I did grasp a "3 months" thing :)

Network going down

[ebmedia]

How much damage could really come of nobody being in the buildings? I mean, once a network is set up with proper equipment and settings, how bad could it get? Plus, this is supposed to be a massive network... do massive networks really have random outages for no reason at all? I thought their equipment was top-notch..

Computers are just dumb machines, they only do what you tell them to, when you tell them to do it. Right?

(I ask because I don't know)

Re:Network going down

[Cramer]

• do massive networks really have random outages for no reason at all?

Yes, yes they do. I'm afraid say it, but things break and often you never figure out why. Sure, it's very likely someone did something or some sequence of things that ultimately causes a problem, but getting someone to stand up and say "yeah, I fscked it up. Sorry." is rare.

Oddly enough, I've spent most of today (along with a half dozen others) trying to restore connectivity on an SDMS ring. It's working now, but nobody knows why. Nothing at all changed on anything, anywhere to break it. And there were no errors visable on any device. Packets go in, but don't come out; yet other data flows across the loop unfettered the whole time!

(if I didn't know better, I'd say Microsoft must be involved.)

Re:Network going down

[pipsqueak]

Sure networks go down for no reason. You get bugs in router code, power outages, machinery digging up bits of ground they shouldn't, trains / trucks / whatever crashing into things.

A big network will suffer outages on a pretty regular basis, hopefully your provider is running it's network properly and has the capacity to cope.

Bankrupt due to high expenses ?

[Anonymous Coward]

If you look at the quality of the tables, chairs and LCD monitors, someone has been on quite a shopping spree...

One can only guess if they have been throwing similar amounts of money on everything else. In that case no wonder they could make ends meet.

Ever heard of a tightly run ship ?

Re:Bankrupt due to high expenses ?

[dr_db]

I wonder how much they save in air conditioning costs? There is costs involved besides the original purchase.

On running a homogenous network

[@madeus]

As a senior systems engineer from a similar organization (Carrier1 (FALCO!)) I can say there were no issue running a multi unix environment, and I've never had any issue with it at any of my previous companies (nor have any of the engineers I've worked with).

At Carrier1 had FreeBSD, Red Hat & Debian Linux, Solaris 9 & 9, HP-UX, even GNU/Hurd and Mac OS X (well, on *my* system :). I had Mac OS X, GNU/HURD, Debian and Solaris all on my desk at one point.

The only problem I've ever had is the fairly trivial (?!) one of getting the command flags right - stuff like the 'ps','route','ipchains, 'ipfw' and 'ifconfig' commands syntax being different, the different flags for package management tools, that sort of thing.

I quickly came to realise that it's not possible to remember all the flags for all programs and remember the best way to do something on a particular system if you are busy all the time, things just seem to seep out. This happens if you are spending lots of time programming or in meetings or working on large projects - in which case you might not touch one type of system for months (until there is a problem with it), at which point you find your self quickly reading man pages and referring to Google a lot. All you need to do is remeber what's improrant, especially things you'll need for troubleshooting, and not worry about the rest - it's enough to know about tool's like Solaris 'ndd' and Linux's 'mknod' and what they do, if you need to remeber exactly how to use them in a given instance you can refer to man pages, O'Reilly Books or Google (which I often find the fastest).

Staying current, reading Freshmeat everyday, installing and configuring new Unixes and new & un-familer packages regularly, being on mailing lists and reading Slashdot are good ways to stay up to date - the more you know the less likely you are to run into something completely unexpected. If your resourceful (which you should be as a Systems Engineer) the only real problems arise went you don't even know where to start, everything else is a piece of cake.

Basically, if you really know unix (and are not just a Red Hat Linux or Solaris flunky who has convinced themselves they are Gurus while they still run Windows 2000 day to day) then you won't have any problems.

Oh, and making lame excuses like 'well I need Windows for work stuff' and 'they won't let me run Unix on my desktop' DO NOT wash - they are just that - excuses for lameness.

I have been for job interviews and been introduced to guys who called themselves (literally!) 'Unix Gods', yet they had only ever used Solaris - if you have any of those you are in deep shit right now. [ Needless to say I ran a mile! ]

Most people fall somewhere in the middle of those two, you'll probably only have one or two decent guys, if your lucky, though if you need to ask you are very possibly in trouble already!

YMMV. :)

dedication and commitment

[danny]

I don't find it that surprising that these guys are so dedicated. Managing computing and network systems isn't so different to engineering or medicine or other occupations where the worker's commitment may be as much to the systems they are building, maintaining, or fixing as to a narrow conception of a job.

The professionalism of doctors and engineers is one of the things that help to leaven the more purely cost/profit approach of their managers.

Danny.

Not as heroic as it sounds?

[Sabalon]

One thing in the article said that a handful of the engineers volunteered to stay while buyers/solutions to their problem are sought.

So it doesn't sound like they locked the doors and kept the pipe going as much as they believe in their company and want to keep working there.

Hang in there guys

Two people left in the NOC now

[Animats]

And they look really tired.

Dawn is breaking there. Soon we'll see if a day shift shows up.

Re:On running a homogenous network - Mis-Post

[@madeus]

Oh bollocks. Your right - I must have got my Mozilla tabs mixed up and posted to the wrong story.

Cheers.

I'll get my coat....

Re:The next Standard Oil, very smart tactics here

[nelsonal]

It would have been a great idea, except that Qwest can't afford the network now. They are facing their own cash crunch. The only suitors I have heard mentioned are AT&T, BT, and Colt Telecomm if they dont get all of KPNQwest's customers first. Qwest might surprise everyone with an offer, but I doubt their stock and bond holders would be very happy about it. They might get mad enough to do something about it, like can current managment.