Slashdot Mirror
KPNQWest Admins Keep Bankrupt Network Running
sebastianw writes "Some of the network administrators from KPNQWest, although they have been (apparently) ordered to shutdown the network, took over control of the KPNQWest NOC. They are trying to keep the network running and keep customers up, regardless of KPNQWest's insolvency. The company warned on Thursday that they would be forced to shut down KPNQwest's entire European data network on Monday unless its customers paid their bills in full immediately." There's a related story on the Register, talking about the possible effect on UK internet access. If anyone needs to hire some network engineers...
It makes you wonder just how many people they really need to run such a network.
It appears to be less than the number they thought.
Hang in there! Do interesting stuff on the webcam! We even forgive you that it's an ASX stream.
Keep my bits flowing! XXX * hugs * kisses!!!
- A european bandwidth addict
http://195.158.250.186/ebonelive1.asx
You can see the problem, none of them are wearing shirts and ties. If they were wearing shirts and ties none of this would have happened.
I'm being more serious than you might think.
"unless its customers paid their bills in full immediately."
Fat chance that will ever happen. People in the US can't even their bill payment service bills on time. =)
Vital Idea
It's great that they're doing this, but how long can they keep it up? I mean, it's great that they're volunteering like this, but you can only go so long without a source of income. After a few weeks, bills start to pile up, or a major router breaks and needs replacing. Running a network's not ridiculously hard, but it does take money...
I think this is great. Really... Big business could care less about their customers in recent years, but we have these guys generously keeping customsers online...even after they aren't being payed.... This is how customer service SHOULD be done. It is ashame that this is the EXCEPTION and not the true state of an industry in peril.
It looks like they just found out they're on slashdot... they all started looking at the camera and something on the computer screens, lol.
It would be funny if customers wondered why it was finally shutdown only to find that it was the Slashdot effect that took down the network. We can only wait and see...
bbh
The webcam rules.. very smooth!
Huh?
Its good these guys keep on working even though they havent been paid for a long time (some in Holland since may). If KPNQwests network goes down well.. here is their map. Too bad most of the news around this is in english.
Of those to whom much is given, much is required.
Hell, if it weren't for my need to eat, I might put in some free time in a NOC like that one, if for no other reason than to have access to the hardware.
Call it the ultimate training lab. Make any change you want - nobody can really bitch much about it, since they're not paying for it anyway. The only motives to keep things running are to, well, keep things running...
Specialization is for insects. - R.A.H.
Take some powerful countries, like France, Britain Japan, allocating funds for high speed backbones whose bandwidth is provided and distributed by volunteer organizations or other NGOs in the country, distributing mainly by the districts population. Now take laws that say a telcom company should allow DSL for free thru its network by its phoneline customers. Also take government policies that update the speed of the bones every three or so years to the several adjacent countries.
Naturally other nations would follow suit seeing the economic improvement such communications allow. Government run backbones would be kept up to speed and private competitors would be hard pressed to provide higher speeds and lower costs. Once the threshold number of countries have this, the countries that started it would be under pressure to continue providing the service, thus a positive feedback can be attained that sustains Free Internet. Apparently all we need is a few smart politicians in a few powerful countries.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
The coolest part of the article was the link to the live webcam feed at the bottom! There is some pretty interesting stuff going on at that place! I don't know if anyone caught this, but I'd swear one of the techs was drinking beer from a longneck bottle! Another time there was a a guy making faces into the camera. These guys are way cooler than my coworkers!
You can't wear a tie in a shop like that!
Just think of what could happen if your tie got caught in the bitstream!
*ZOINK* there goes access to Poland!
Wedding rings? *ZAAM* Oh damn, I think we just transported Roger's ring finger into the shared computer in Ms. Kingston's 3rd Grade class in Manchester, UK.
You gotta be careful when working around heavy data processing machinery, man! They aren't safe or simple, like a table saw or anything!
"Can of worms? The can is open... the worms are everywhere."
I thought the Internet routed around damage?
I'm sure I read somewhere that during the Gulf War the Allies had problems trying to disrupt Iraqi communications as the Iraqis were using a TCP/IP-based network similar in configuration to the Internet.
If the plug is eventually pulled on the KPNQWest network how drastic will the effect be?
I've got a fever and the only prescription is more COBOL.
Assuming that they go down, how much of an impact will it have?
Speak english in the NOC?
Lameness filter
More Filter
Doesn't it?
The actions of the guys in Brussels have been an inspiration for the rest of us in the company (those that are left).
I can pretty much assure that the UK network will keep running, definitely one of the main AS numbers of the UK (5571) has alternate transit so do most in-country networks. If anything it's the Eurorings that are at risks but even they are supported right now from Brussels AND The Hague (both IP NOCs are operational right now).
The actions of the last few days make me proud to be part of KQ, Ebone and Netcom and I really hope that we can all survive this crisis, and if we don't, it won't be from the lack of trying or courage.
just another Netcom/Ebone/KQ techie
Full Time Idiot and Miserable Sod
Nothing is real but the pain
Hmmm,
I find it interesting that they seem to be having trouble with users paying them. The most serious problem I face is customers not paying for the services they receive.
Just shut down one site 2 months ago which was overdue by 6 months... Sortly thereafter got a call berating me for their website not being up. I pointed out that they were past due and stated the site would not go back up until they paid their bill... Couple of days later I had a check.
It supprised me since the site has always been a problem with paying late. I've finally decided to take a no nonsense approach to billing rather than trying to work with customers. I've had too many customers just walk away without paying their bills and thus I have been shedding customers during the last couple of months.
It's definately a lot easier now, without maintaining a bunch of sites that don't pay.
Somehow all of this left me thinking about Atlas Shrugged, but in reverse. In AS the "socialist" forces are trying to ruin the world, but here it's the business world. And people try to keep things running because it's their own interest to do so.
Interestingly, our local KPNQwest is leaving the mothership and this seems to be happening elsewhere, too.
0 20 607.shtml (in Finnish)
On the Finnish company's page they state they're ready to re-route traffic and do what it takes. KPNQwest Norway got bought by Catch Communications and this might be the fate of the others, too. Seems Nokia is already on board as a customer.
http://www.kpnqwest.fi/yritys/uutiset/uutinen20
Anyone got more on this?
J
Well, how many it takes to run an operation like that depends a lot on how it's all set up. I mean, how many people does it take to fly a jet? Well, maybe one, if it is a little jet, but like four for a big jetliner. Actually, if you think about it, the jet wouldn't be flying at all if the passengers didn't buy tickets, so maybe it takes a few hundred.
An ISP's no different: it's a support network. Now, if you want to run it all as a Windows shop, then yeah, you're going to need a small army to keep things going. But I'm guessing they did a little switcheroo over to Linux, allowing them to get by with a much smaller staff. Basically, the auto-correction features in the 2.5 kernel allow a lot more leeway with configuration, and the low-latency threaded process spooling has increased our productivity by a factor of 12. That's nothing to sneeze at, IMHO.
Linux has done for the IT industry what Boeing did for the airline industry, only more gently. This is just more of the fruit from that particular tree of knowledge.
Karma: Good (despite my invention of the Karma: sig)
I applaud the initiative taken by these sysadmins. It is rather similar to the altruistic nature of the vast groups of Open Source developers that are so dedicated in producing quality software and systems. It is only when the common man realizes that Open Source leads the way that we can shake off the corporate mantra of only using MS products.
Wearing pants should always be optional.
However, I find that being a bit more poetic than "Shut up you Troll" works better. ;-)
Conversion Rate Optimisation French / English consultant
that's a lot of dedication and persistence. I hope all the guys that are volunteering get good jobs else where.
one of the techs was drinking beer from a longneck bottle!
Same beer as the regular kind but less of it, at a higher price. Longnecks, pfft.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
Sorry, but they probably will need to stand at the rear of the "I need a job" line, like the rest of us.
However, i do admire their dedication to their jobs. It's either customer loyalty or "Hell no. Not again. HELL NO." Mentality of losing a job.
:)
Every story that affects the US has some moron crying that the story is "US-centric."
I could do the same and say "How is this news -- the Internet in Europe -- it doesn't affect me!"
Not all news stories involes the whole world, so stop whining.
[troll]Besides, nothing good ever comes out of the UK, (Except for The Stone Roses and The Verve).[/troll]
-----
Problems regarding accounts or comment posting should be sent to CowboyNeal.
- they're hoping that by staying they'll be given a job by the new buyer/management
- there's nothing else for them to do (have you seen the job market for out of work network techs?)
- if the company does close, they may be able to sneak out a couple flat panels
- they're trying to show potential buyers that they "care" about the customers, making it more attractive to the public (anyone who has worked on support hates the customer)
I think most people would stay for a couple of days, just to see what pans out, but beyond that?
They're efforts would be better directed in the Open Source developer community.
Wearing pants should always be optional.
The ones still there seem to be a dedicated bunch of people, and I wouldn't be surprised if they managed to build something from what they have now.
Although reality would probably rear its ugly head, and they would be forced to shut down anyway. Everything is nice in theory, I guess...
Clever signature text goes here.
There are other parts of KPNQwest that do run on general-purpose computers, such as the administrative and billing databases, network management systems, etc. Probably some are running Windows or Linux, and others are probably running Solaris or HPUX.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Just an observation.
What's a nice girl like her doing in a place like that? (cleaning up/out, obviously, which is perfectly fine in this gender equal world, not to say that women are identical to men, nor superior, nor inferior, but on a level that is as high, though not higher, and certainly not lower, than that of the men who are browsing the internet in the background.)
Anyway, I'd like some nice person there to knock that orange cap on the ground for me, about a meter further away from the camera.
Hmm.
What do you mean, 'customer service' does not equal 'fan service'?!
Anyway, kudos on the operation to everyone there.
LOL, misread Norja (Norway) for Nokia. You really read best what you read most.
:P
Then again, might have been accidentally correct, but whatever
J
As you and others have said, if nothing goes bad, it doesn't take much maintenance to keep running, though it's adding new service that needs resources and causes mistakes that break things.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
xine mms://195.158.250.186/EboneLive
make sure you have the necessary win32codecs installed, though.
time is a funny concept
I run a high-traffic web site based close to Oslo, Norway, and I've been getting reports of problems from my users since the day after the bankruptcy was announced. My users are from all over the world (about 50% from the US), and at any given time there seem to be several different groups of them who can't connect to the server at all. The problems are usually temporary, and the users affected seem to be spread across the world. So far the most consistently affected users seem to be those from the Netherlands and Singapore (.nl seems logical, but why .sg of all places?).
We were leasing our lines from KPNQwest Norway, so that might explain some of it, but I suspect anyone who's trying to connect to anything through KPNQwest's backbone will be experiencing intermittent problems if things aren't resolved soon.
-- If no truths are spoken then no lies can hide --
According to the register, Freeserve won't be affected, which is good because thats the ISP ive been using. They provide a really fast connection for a modem (which is good) so it seems that those AOL lusers will get punnished for using a rubbish provider.
$ yes >
yhbt
das wel bijzonder, meestal hebben jullie een grote lul die de bloedstroom naar de hersenen vermindert, gefeliciteert dus dan maar
I wonder what happened to this
I was watching the webcam of their NOC. There was a visible protest, with people holding up signs in front of the webcam. A few minutes ago, somebody in a suit came in and spent a few minutes talking to people. Then most of the people picked up their stuff and left. A few people remain, quietly sitting at computers.
That's the whole point. Qwest couldn't create the network to Europe on their own, so they teamed up with KPN. Now they played out KPN by getting KPNQwest bankrupt, so they can get the backbone for dirt cheap.
Very chesslike move!I've haven't been paid since May, either, and I suspect that most people haven't been paid since May.
Oh. I bet you didn't mean bimonthly payroll...
I think people are over estimating the effect of this. The reason that they are going out of business is that there is over supply in the market and as a result the prices have been falling for a long time. Their competitors are scrambling to pick up the customers that will lose service (I know - I work for one of them). THe effect of this may lead to a few pockets of service interuption, but most smart customers have a backup plan and those that do not are not short of suppliers keen to step into the gap.
How much damage could really come of nobody being in the buildings? I mean, once a network is set up with proper equipment and settings, how bad could it get? Plus, this is supposed to be a massive network... do massive networks really have random outages for no reason at all? I thought their equipment was top-notch..
Computers are just dumb machines, they only do what you tell them to, when you tell them to do it. Right?
(I ask because I don't know)
...But here they don't have bulldozers to fight the cops that want to shut them down.
Where is Roberto D'Orazzio ??
... turns out US's Qwest branch paid to move all of our equipment to a British Telecom colo. i think it was in our contract or something. they changed our internet IPs though, so we still have to reconfigure all of our networking gear and machines.. kind of a pain in the ass, but the reacharound is better than nothing.
If you look at the quality of the tables, chairs and LCD monitors, someone has been on quite a shopping spree...
One can only guess if they have been throwing similar amounts of money on everything else. In that case no wonder they could make ends meet.
Ever heard of a tightly run ship ?
As a senior systems engineer from a similar organization (Carrier1 (FALCO!)) I can say there were no issue running a multi unix environment, and I've never had any issue with it at any of my previous companies (nor have any of the engineers I've worked with).
At Carrier1 had FreeBSD, Red Hat & Debian Linux, Solaris 9 & 9, HP-UX, even GNU/Hurd and Mac OS X (well, on *my* system
The only problem I've ever had is the fairly trivial (?!) one of getting the command flags right - stuff like the 'ps','route','ipchains, 'ipfw' and 'ifconfig' commands syntax being different, the different flags for package management tools, that sort of thing.
I quickly came to realise that it's not possible to remember all the flags for all programs and remember the best way to do something on a particular system if you are busy all the time, things just seem to seep out. This happens if you are spending lots of time programming or in meetings or working on large projects - in which case you might not touch one type of system for months (until there is a problem with it), at which point you find your self quickly reading man pages and referring to Google a lot. All you need to do is remeber what's improrant, especially things you'll need for troubleshooting, and not worry about the rest - it's enough to know about tool's like Solaris 'ndd' and Linux's 'mknod' and what they do, if you need to remeber exactly how to use them in a given instance you can refer to man pages, O'Reilly Books or Google (which I often find the fastest).
Staying current, reading Freshmeat everyday, installing and configuring new Unixes and new & un-familer packages regularly, being on mailing lists and reading Slashdot are good ways to stay up to date - the more you know the less likely you are to run into something completely unexpected. If your resourceful (which you should be as a Systems Engineer) the only real problems arise went you don't even know where to start, everything else is a piece of cake.
Basically, if you really know unix (and are not just a Red Hat Linux or Solaris flunky who has convinced themselves they are Gurus while they still run Windows 2000 day to day) then you won't have any problems.
Oh, and making lame excuses like 'well I need Windows for work stuff' and 'they won't let me run Unix on my desktop' DO NOT wash - they are just that - excuses for lameness.
I have been for job interviews and been introduced to guys who called themselves (literally!) 'Unix Gods', yet they had only ever used Solaris - if you have any of those you are in deep shit right now. [ Needless to say I ran a mile! ]
Most people fall somewhere in the middle of those two, you'll probably only have one or two decent guys, if your lucky, though if you need to ask you are very possibly in trouble already!
YMMV.
they are the employees. (story at mikemth.de)
Yeah, I am a recent graduate, MSEE and looking for a job in the Telecom industry (or related).
:-)
It is great that everytime I consider a company as possible employer, they issue a warning announcing record losses, they say they're going to make 30% of their employees 'redundant' or they go straight bankrupt.
Really, I hate all the managers and investors that made this bubble so big, taking all the money and now running before it gets even worse. And all we have left is a dire industry where it is damned difficult to find a job.
And I post AC to protect my identity and because I am at karma 50 !!
I am looking at you on the live webcam.
Do you mind writting "Hello Slashdot!" or something like that in a paper and placing it near the webcam for all of us to see?
It is an experiment I'd like to do, to see if they really read Slashdot, could you please mod this up?? On top it is easier to be seen...
http://195.158.250.186/ebonelive1.asx for the webcam stream.
The professionalism of doctors and engineers is one of the things that help to leaven the more purely cost/profit approach of their managers.
Danny.
I have written over 900 book reviews
...my domains are hostet on the Dt.Telekom backbone.
I feel bad for those people who are not so lucky though.
--Ulrich
On no accounts allow a Vogon to read poetry at you
One thing in the article said that a handful of the engineers volunteered to stay while buyers/solutions to their problem are sought.
So it doesn't sound like they locked the doors and kept the pipe going as much as they believe in their company and want to keep working there.
Hang in there guys
Dawn is breaking there. Soon we'll see if a day shift shows up.
Oh bollocks. Your right - I must have got my Mozilla tabs mixed up and posted to the wrong story.
Cheers.
I'll get my coat....