Slashdot Mirror

← Back to Stories (view on slashdot.org)

KPNQWest Admins Keep Bankrupt Network Running

Posted by ryuzaki0 on from the beyond-the-call-of-duty dept.

sebastianw writes "Some of the network administrators from KPNQWest, although they have been (apparently) ordered to shutdown the network, took over control of the KPNQWest NOC. They are trying to keep the network running and keep customers up, regardless of KPNQWest's insolvency. The company warned on Thursday that they would be forced to shut down KPNQwest's entire European data network on Monday unless its customers paid their bills in full immediately." There's a related story on the Register, talking about the possible effect on UK internet access. If anyone needs to hire some network engineers...

25 of 202 comments (clear)

  1. At least it's efficient by ThesQuid · · Score: 4, Insightful

    It makes you wonder just how many people they really need to run such a network.

    It appears to be less than the number they thought.

    1. Re:At least it's efficient by Sorthum · · Score: 5, Informative

      For a day or so, you can keep most large networks running with only one person or so. I work in a NOC that handles access for over 2000 sites, and for nights and holidays (weekends too), we have exactly one person here.

      Though when things go bad, we have other people on call-- it can get hectic.

      So if all goes well, they could hold out for quite some time (until something important breaks).

    2. Re:At least it's efficient by Guido69 · · Score: 3, Interesting

      Likely that's true to an extent, but it's not sustainable. Many large networks routinely run with reduced staff on evenings, weekends, holidays, Star Trek premiers, and off-hours. But there is more to running the network than just keeping it up (maintenance work). What about upgrades or project work? It's hard to move forward with technology if your entire staff is focused solely on maintaining your existing infrastructure.

      I'm not saying they likely couldn't trim some fat, but a staff reduction of more than 2/3 cuts too deep.

      --
      - If we aren't supposed to eat animals, then why are they made out of meat? - Steven Wright
    3. Re:At least it's efficient by ch-chuck · · Score: 3, Insightful

      Reminds me of when Verizon's staff took industrial action (strike) a couple of years ago for the Communication Workers of America union - one industry wag wrote in Network Computing, "Did anybody notice when Verizons 800,000 employees went on strike? Neither did I". But seriously, no new lines were installed, and I'm sure all the maintenance work would have soon caught up with what little mgmt could do to run such a network.

      --
      try { do() || do_not(); } catch (JediException err) { yoda(err); }
    4. Re:At least it's efficient by PurpleFloyd · · Score: 5, Interesting

      Since when has Cisco intentionally made bad hardware? I have found that while their prices are high, it is hard to make the hardware on a Cisco box fail short of running mains voltage through an Ethernet line.

      --

      That's it. I'm no longer part of Team Sanity.
  2. How? by Sorthum · · Score: 3, Interesting

    It's great that they're doing this, but how long can they keep it up? I mean, it's great that they're volunteering like this, but you can only go so long without a source of income. After a few weeks, bills start to pile up, or a major router breaks and needs replacing. Running a network's not ridiculously hard, but it does take money...

    1. Re:How? by mnordstr · · Score: 5, Informative

      They only need to keep it running for a short time, KPNQwest is trying to find a buyer for the network. Let's hope it finds one, as the european Internet will suffer a huge bandwidth shortage if the network goes down.

      As a KPNQwest customer myself, I'd like to say that they've provided a great service over the years. This is absolutely the best ISP ever, no real outages, great customer service (as you might have noticed). It's a real shame it turned out like this.

  3. noc admins by meis31337 · · Score: 5, Interesting

    I think this is great. Really... Big business could care less about their customers in recent years, but we have these guys generously keeping customsers online...even after they aren't being payed.... This is how customer service SHOULD be done. It is ashame that this is the EXCEPTION and not the true state of an industry in peril.

    1. Re:noc admins by Indras · · Score: 3, Interesting

      Or... this could be another case of a person doing what's best for themselves is improving things for everyone. I mean, this their job, it's what they get paid to do. Unless, of course, they aren't receiving a paycheck for this... in which case...

      QUADROUPLE KUDOS TO EVERYONE THERE!

      --
      The speed of time is one second per second.
    2. Re:noc admins by Manitcor · · Score: 3, Informative

      They were fired on the 6th, yet they stayed becasue they were the only ones left to support the network.

      --
      "Don't mess with him, he taunts the happy fun ball."
    3. Re:noc admins by hollow_man · · Score: 5, Insightful

      Let's not forget that whilst the guys are fighting to save the main NOC and keeping it running, other temporary NOCs have sprung up and the old KQ NOC in The Hague has also been resurrected.

      We're all working very hard to make sure that disruption to our customers stays limited to the absolute minimum and all techies in this company have now shown to management that they can make their stand in the darkest hour of this company.

      Needless to say I'm proud to work with these guys.

      --
      Full Time Idiot and Miserable Sod
      Nothing is real but the pain
  4. I think they just found out... by ceejayoz · · Score: 4, Funny

    It looks like they just found out they're on slashdot... they all started looking at the camera and something on the computer screens, lol.

  5. in the end, it was Slashdot that ended them.... by bbh · · Score: 5, Funny

    It would be funny if customers wondered why it was finally shutdown only to find that it was the Slashdot effect that took down the network. We can only wait and see...

    bbh

  6. Potential european outtage map by sniggly · · Score: 3, Informative


    Its good these guys keep on working even though they havent been paid for a long time (some in Holland since may). If KPNQwests network goes down well.. here is their map. Too bad most of the news around this is in english.

    --
    Of those to whom much is given, much is required.
  7. The Ultimate Playground by Caradoc · · Score: 4, Interesting

    Hell, if it weren't for my need to eat, I might put in some free time in a NOC like that one, if for no other reason than to have access to the hardware.

    Call it the ultimate training lab. Make any change you want - nobody can really bitch much about it, since they're not paying for it anyway. The only motives to keep things running are to, well, keep things running...

    --
    Specialization is for insects. - R.A.H.
  8. Re:Just been on their webcam in the NOC by Manitcor · · Score: 5, Funny

    That's probably why they decided to stay in the NOC they aren't getting paid and the aero chairs are more comfortable than the curb in front of their apartment building.

    Plus they can use the Internet access to do job hunting and play really fast QIII. (What you actually thought that they were maintaining the network?)

    Actually I think it's great to see such dedication.

    --
    "Don't mess with him, he taunts the happy fun ball."
  9. You fool! by stefanlasiewski · · Score: 5, Funny

    You can't wear a tie in a shop like that!

    Just think of what could happen if your tie got caught in the bitstream!

    *ZOINK* there goes access to Poland!

    Wedding rings? *ZAAM* Oh damn, I think we just transported Roger's ring finger into the shared computer in Ms. Kingston's 3rd Grade class in Manchester, UK.

    You gotta be careful when working around heavy data processing machinery, man! They aren't safe or simple, like a table saw or anything!

    --
    "Can of worms? The can is open... the worms are everywhere."
  10. Re:Webcam! by Indras · · Score: 5, Funny

    These guys are way cooler than my coworkers!

    It's the lack of ties around their necks. They have oxygen going to their brains. Makes people friendly and creative!

    --
    The speed of time is one second per second.
  11. The story so far... by hollow_man · · Score: 5, Interesting

    The actions of the guys in Brussels have been an inspiration for the rest of us in the company (those that are left).

    I can pretty much assure that the UK network will keep running, definitely one of the main AS numbers of the UK (5571) has alternate transit so do most in-country networks. If anything it's the Eurorings that are at risks but even they are supported right now from Brussels AND The Hague (both IP NOCs are operational right now).

    The actions of the last few days make me proud to be part of KQ, Ebone and Netcom and I really hope that we can all survive this crisis, and if we don't, it won't be from the lack of trying or courage.

    just another Netcom/Ebone/KQ techie

    --
    Full Time Idiot and Miserable Sod
    Nothing is real but the pain
  12. Payments by Anonymous Coward · · Score: 3, Interesting

    Hmmm,
    I find it interesting that they seem to be having trouble with users paying them. The most serious problem I face is customers not paying for the services they receive.

    Just shut down one site 2 months ago which was overdue by 6 months... Sortly thereafter got a call berating me for their website not being up. I pointed out that they were past due and stated the site would not go back up until they paid their bill... Couple of days later I had a check.

    It supprised me since the site has always been a problem with paying late. I've finally decided to take a no nonsense approach to billing rather than trying to work with customers. I've had too many customers just walk away without paying their bills and thus I have been shedding customers during the last couple of months.

    It's definately a lot easier now, without maintaining a bunch of sites that don't pay.

  13. Re:How bad will it be? by ceejayoz · · Score: 5, Informative

    Yes, it is designed to route around damage, but this isn't damage - it's the whole network going down at once.

    Take a look at this map and you'll see the problem...

  14. Finnish KPNQwest standing by to re-route by Judge_Fire · · Score: 5, Informative

    Interestingly, our local KPNQwest is leaving the mothership and this seems to be happening elsewhere, too.

    On the Finnish company's page they state they're ready to re-route traffic and do what it takes. KPNQwest Norway got bought by Catch Communications and this might be the fate of the others, too. Seems Nokia is already on board as a customer.

    http://www.kpnqwest.fi/yritys/uutiset/uutinen200 20 607.shtml (in Finnish)

    Anyone got more on this?
    J

  15. Re:Internet COULD be free by Brightest+Light · · Score: 3, Insightful
    Riiiight....


    And as an added bonus, the gov'ts will not have to fight with ISP's in order to get carnivore, echelon, and whatever else on the network.

  16. Not Linux, Cisco and Fiberthings and HVAC by billstewart · · Score: 5, Insightful
    The parts of KPNQwest that everybody's worried about these days aren't running either Linux or Windows - they're routers (probably Cisco) and fiber muxes of various sorts (which do have OSs for their configuration and user interfaces) and the air conditioners for the rooms the equipment runs in. Both non-HVAC categories are pretty reliable as long as there aren't technicians adding or changing things and as long as nobody backhoes any important circuits.

    There are other parts of KPNQwest that do run on general-purpose computers, such as the administrative and billing databases, network management systems, etc. Probably some are running Windows or Linux, and others are probably running Solaris or HPUX.

    --

    Bill Stewart
    New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
  17. On running a homogenous network by @madeus · · Score: 3, Interesting


    As a senior systems engineer from a similar organization (Carrier1 (FALCO!)) I can say there were no issue running a multi unix environment, and I've never had any issue with it at any of my previous companies (nor have any of the engineers I've worked with).

    At Carrier1 had FreeBSD, Red Hat & Debian Linux, Solaris 9 & 9, HP-UX, even GNU/Hurd and Mac OS X (well, on *my* system :). I had Mac OS X, GNU/HURD, Debian and Solaris all on my desk at one point.

    The only problem I've ever had is the fairly trivial (?!) one of getting the command flags right - stuff like the 'ps','route','ipchains, 'ipfw' and 'ifconfig' commands syntax being different, the different flags for package management tools, that sort of thing.

    I quickly came to realise that it's not possible to remember all the flags for all programs and remember the best way to do something on a particular system if you are busy all the time, things just seem to seep out. This happens if you are spending lots of time programming or in meetings or working on large projects - in which case you might not touch one type of system for months (until there is a problem with it), at which point you find your self quickly reading man pages and referring to Google a lot. All you need to do is remeber what's improrant, especially things you'll need for troubleshooting, and not worry about the rest - it's enough to know about tool's like Solaris 'ndd' and Linux's 'mknod' and what they do, if you need to remeber exactly how to use them in a given instance you can refer to man pages, O'Reilly Books or Google (which I often find the fastest).

    Staying current, reading Freshmeat everyday, installing and configuring new Unixes and new & un-familer packages regularly, being on mailing lists and reading Slashdot are good ways to stay up to date - the more you know the less likely you are to run into something completely unexpected. If your resourceful (which you should be as a Systems Engineer) the only real problems arise went you don't even know where to start, everything else is a piece of cake.

    Basically, if you really know unix (and are not just a Red Hat Linux or Solaris flunky who has convinced themselves they are Gurus while they still run Windows 2000 day to day) then you won't have any problems.

    Oh, and making lame excuses like 'well I need Windows for work stuff' and 'they won't let me run Unix on my desktop' DO NOT wash - they are just that - excuses for lameness.

    I have been for job interviews and been introduced to guys who called themselves (literally!) 'Unix Gods', yet they had only ever used Solaris - if you have any of those you are in deep shit right now. [ Needless to say I ran a mile! ]

    Most people fall somewhere in the middle of those two, you'll probably only have one or two decent guys, if your lucky, though if you need to ask you are very possibly in trouble already!

    YMMV. :)