Slashdot Mirror

← Back to Stories (view on slashdot.org)

Fire At AT&T Facility Causes Outage For Over a Million U-Verse Fiber Customers In Texas (wfaa.com)

Posted by BeauHD on from the too-hot-to-handle dept.

New submitter JustChapman writes: Local Dallas/Fort Worth WFAA is reporting a major outage of AT&T U-Verse fiber internet, due to a lightening strike at a switching facility in Richardson, TX. Apparently the strike took out primary and secondary power systems, setting fire to the building. One commenter states a representative allegedly said that 1.5 million customers are currently without service.

54 comments

  1. End of Time by Anonymous Coward · · Score: 0

    U-Verse met its H-Death.

    1. Re:End of Time by Anonymous Coward · · Score: 0

      The 'end of time' is when ATT will finally credit all the U-verse customers for this downtime. Maybe.

  2. Yup, it's been a stormy afternoon here... by bobbied · · Score: 3, Funny

    Thank the maker I don't have AT&T as my ISP, but it remains to be seen if I have internet at home.

    However, given AT&T's past transgressions, Somehow it makes sense that the Maker is mad with ATT and struck them with lighting.

    --
    "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    1. Re:Yup, it's been a stormy afternoon here... by GoTeam · · Score: 2

      Sadly I do have U-verse TV and internet. Seemed like a great deal for 1Gb internet service. Service at the house has been down since 10:30 this morning. They won't even attempt to provide an ETA on repairs or service restoration. What kind of tech company doesn't have a redundancy plan? Ohhhh, AT&T isn't a tech company. My bad.

    2. Re:Yup, it's been a stormy afternoon here... by Darinbob · · Score: 1

      But why did the lightning miss Comcast?

    3. Re:Yup, it's been a stormy afternoon here... by Joe_Dragon · · Score: 1

      Well with directv as long as you have power and clear view of the sky you have TV.

    4. Re:Yup, it's been a stormy afternoon here... by Joe_Dragon · · Score: 1

      With comcast there is a few super heads end that if they go down a BIG part of there service will go out.

    5. Re: Yup, it's been a stormy afternoon here... by backtick · · Score: 2

      Youâ(TM)re not paying for redundancy at the POP level. You said yourself it seemed like a great deal, indicating price was one of your major drivers. 99.99% of end users donâ(TM)t want to pay what redundancy costs for the small amount of outages, especially now that everyone has a cell device which can handle 10s of Mb/sec as a âfall backâ(TM). If you really need wired redundancy, have a cable company put in a line, and manage that with a router with prioritization. You donâ(TM)t want to pay for that? And thereâ(TM)s why itâ(TM)s down right now.

      Sorry, but thereâ(TM)s a point at which ATT, Comcast, etc have to limit their redundancy options to manage costs for the majority of their customers. If youâ(TM)re an outlier in your redundancy needs, pony up to get it, using ATT as one leg of a multi-leg solution.

      Heck, itâ(TM)s not hard to set up a cell plan as a hot spot for most devices, so bridge your WiFi to your cell (or your spare cell, since you need redundancy, right? And that cell isnâ(TM)t normally being used, so itâ(TM)s charged and ready when your dies or fails? Seems perfect to use it as a hotspot as a backup option for the few hours or days the carrier is down).

      Iâ(TM)ve spent the better part of 3 decades designing redundant systems, and at the end of the day, on a large or small scale, they cost something; end users have to decide if itâ(TM)s worth paying more, and the suppliers have to decide if itâ(TM)s worth losing business as a result of raising their prices to fund the increases. In this case, UVerse is likely âavailable enoughâ(TM) for enough customers that ATT has a business justification to not make things more redundant, and if you have a justification to need them more redundant, you likely have some options to do so, at a cost.

    6. Re:Yup, it's been a stormy afternoon here... by Anonymous Coward · · Score: 0

      The beauty of all of this? The internet is resilient, designed to withstand a nuclear attack, and will route around all of this nonsense. (Fiber run to my local loop, dark fiber to my datacenter, DC provides blended connectivity.)

    7. Re: Yup, it's been a stormy afternoon here... by GoTeam · · Score: 2

      Great story. I do have a mobile hot spot with unlimited data that I use when I travel or in case of an outage. I can watch TV on any of my mobile devices, or connect them to the TV and watch whatever I want. It's not about me and what I have. The problem is a large "tech" company thinks it can shove all of their services for a large population into one building. They can do it if they want, but it will cost them customers. If they decided as a company that losing a bunch of customers is fine, then hey, we all win. They stay in business with a smaller customer base and I get a more dependable service provider.

    8. Re:Yup, it's been a stormy afternoon here... by bobbied · · Score: 1

      Well with directv as long as you have power and clear view of the sky you have TV.

      Yea, but why? Even with a thousand channels, there is never anything good on any more.

      Personally, I just have a DVR full of interesting stuff and a drawer of DVD/Blu-ray disks for when the internet happens to be down. Not to mention the book shelves full of good lit books from our "home schooling the kids" phase that just ended. I literally have a life time or two of entertainment...

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    9. Re:Yup, it's been a stormy afternoon here... by bobbied · · Score: 2

      Sadly I do have U-verse TV and internet. Seemed like a great deal for 1Gb internet service. Service at the house has been down since 10:30 this morning. They won't even attempt to provide an ETA on repairs or service restoration. What kind of tech company doesn't have a redundancy plan? Ohhhh, AT&T isn't a tech company. My bad.

      I just heard a local report on this.. Apparently the building burned pretty badly and the Roof collapsed. Unless they have a totally redundant system in some other location, which I find improbable for a host of reasons, it's going to be a LONG time before this gets fixed. IF customer wiring goes though this building (which I find HIGHLLY likely) I'm going to guess it is going to take a lot of time to rewire everything to some new location.

      In short.. I'd be asking AT&T to let you out of any contracts based on their inability to provide services and find another provider in your area BEFORE the rush starts. (I.E. Call Spectrum, NOW, before they run out of equipment.)

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    10. Re: Yup, it's been a stormy afternoon here... by Anonymous Coward · · Score: 0

      Or try for directv no contact + 4g hot spot uncapped

    11. Re: Yup, it's been a stormy afternoon here... by kenh · · Score: 1

      The issue was a lightning strike that took out the redundant power feed for the facility when a fire started in the power room.

      Sure, they should have had two power rooms, but at some point there is diminishing return on the redundancy.

      --
      Ken
    12. Re:Yup, it's been a stormy afternoon here... by sjames · · Score: 2

      Not to worry, with their extensive redundancy in the critical departments, the bills will go out on time and the legal staff is at full power.

    13. Re: Yup, it's been a stormy afternoon here... by petermgreen · · Score: 1

      Youâ(TM)re not paying for redundancy at the POP level.

      Right but 1.5 million customers affected seems a bit higher up the network than "POP level"

      --
      note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
    14. Re:Yup, it's been a stormy afternoon here... by Hallux-F-Sinister · · Score: 1

      Your deity has shitty aim. If this is revenge for AT&T having shitty service, why attack the victims OF that shitty service by taking them from having shitty service, to having no service at all? This in NO way punishes the DECISION-MAKERS at AT&T responsible for their dickish behavior. Just saying.

      A REAL karmic revenge would be if SCOTUS handed down a ruling that blocked further ISP consolidation, undid mergers, and mandated that 200% of all profits from any ISP that had even a single household it served where there were no other comparable services available would be seized and returned to the subscribers, or that the city, municipality or county where this place was would go into competition with them and provide internet itself. OR BOTH. As someone who has lived in multiple places where ISPs had corruptly obtained illegal monopolies, I say FUCK AAALLLL of that shit.

      In fact, I keep getting mass mailings from my shitty old ISP that because where I moved has these things called OPTIONS, (which is so nice,) I am free not to have to use. I deposit each in the shredder, unopened. Gshrchrchrchrxhsshshshshshshshwaaahhh... Such a lovely sound.

      --
      Our reign has gone on long enough. Indeed. Summon the meteors.
    15. Re: Yup, it's been a stormy afternoon here... by kenh · · Score: 2

      I like on DFW, and I've had inverse for a year, and so far this outage has gone 12 hours, or half a day.

      Network reliability is measured in "nines", as in "5 nines", which means the network is available 99.999% of the time.

      2 nines is 99%, or about 90 hours of downtime per year.

      3 nines is 99.9%, or about 9 hours of down time per year.

      4 nines is 99.99%, or about 1 hour of downtime per year.

      5 nines, the holy grail of availability is about 5 minutes of downtime per year.

      Right now, U-Verse is at about "3 nines", perfectly acceptable for consumer internet service. Few consumers are willing to pay what would be required to deliver 4 nines availability.

      --
      Ken
    16. Re: Yup, it's been a stormy afternoon here... by kenh · · Score: 1

      The problem is a large "tech" company thinks it can shove all of their services for a large population into one building.

      This is how telcos achieve 'economy of scale', they have done this for about a century - same for municipal water and electricity

      --
      Ken
    17. Re: Yup, it's been a stormy afternoon here... by Anonymous Coward · · Score: 0

      You sound like an at&t shill. You provide a service to 1.5 million customers and don't host a redundant location off-site from your main datacenter?

    18. Re:Yup, it's been a stormy afternoon here... by Anonymous Coward · · Score: 0

      I just heard a local report on this.. Apparently the building burned pretty badly and the Roof collapsed. Unless they have a totally redundant system in some other location, which I find improbable for a host of reasons, it's going to be a LONG time before this gets fixed.

      Whatever happened, as of 11 pm I'm up and on the net again in FW. Dunno 'bout Dallas.

    19. Re:Yup, it's been a stormy afternoon here... by bobbied · · Score: 1

      I think you missed the joke...

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    20. Re:Yup, it's been a stormy afternoon here... by GoTeam · · Score: 1

      Yep, I'm in Richardson not far from the building that had the issue. We were up and running again around midnight.

    21. Re: Yup, it's been a stormy afternoon here... by kenh · · Score: 1

      It was resolved (for me, anyway) by midnight Monday - about 15 hours downtime.

      --
      Ken
    22. Re: Yup, it's been a stormy afternoon here... by kenh · · Score: 1

      Nonsense.

      This isnâ(TM)t a classical data center like google or amazon, this is for local access - by definition it has to be near itâ(TM)s customers/users. I live in the impacted area, and my internet service was restored by midnight, perfectly acceptable for consumer internet service IMHO.

      For those running businesses on a consumer ISP like this, this is why you should invest in business class service.

      --
      Ken
    23. Re: Yup, it's been a stormy afternoon here... by kenh · · Score: 1

      U-Verse had redundant power, the problem was the place where those two diverse power sources converged is where the fire was. Absent investing in a fully-redundant facility outside the disaster zone of the first facility this type of problem is unavoidable.

      This wasnâ(TM)t a classical âdatacenterâ(TM), where itâ(TM)s actual location is irrelevant, this was a facility where multiple access points converged. This facility had connections to community-level distribution (head) points, which in turn fed neighborhood pedestals. To create a redundant facility would require a second building with another set of fiber connections to each community head office, and each community head office would need additional hardware to select between either central office.

      --
      Ken
  3. Lightening by rrohbeck · · Score: 2

    Not what you think it is: https://www.dictionary.com/bro...

    --
    thegodmovie.com - watch it
    1. Re:Lightening by dgatwood · · Score: 1

      They were struck by someone painting all the grey walls with white paint. It reflected more light, so the building got too cold. This cause the air handlers to freeze up, at which point the equipment all melted down. It makes perfect sense to me.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    2. Re:Lightening by Anonymous Coward · · Score: 0

      A darkening strike might have been disastrous!

  4. lightening vs lightning by Anonymous Coward · · Score: 2, Interesting

    Lightening is where you lighten something up that's dark such as a photo in digital photo editing software.

    Lightning is the high-energy burst that almost rivals a nuclear bomb, causes audio compression (i.e. a thing called "thunder"), and can start building fires even if the building is properly grounded.

    Hopefully, this has been enlightening for JustChapman. Get a dictionary!

    1. Re:lightening vs lightning by Anonymous Coward · · Score: 0

      Is a lightening strike something to do with protesting workers that are decreasing their efforts to strike?

    2. Re: lightening vs lightning by Anonymous Coward · · Score: 0

      And it wasn't even lightning--that theory was debunked hours ago....

    3. Re: lightening vs lightning by JustChapman · · Score: 1

      Har. Of course I know the diff. My fingers, not so much. Lol. And youâ(TM)ve never mistyped anything while living there in your glass house, I can assume?

    4. Re: lightening vs lightning by Anonymous Coward · · Score: 0

      The energy in a lightning strike is in no way comparable to a nuclear bomb.

  5. That's odd... by rickb928 · · Score: 1

    Lightening strikes usually result in liftoff and sometimes an abrupt crash, equally devastating, and fire is a common feature...

    --
    deleting the extra space after periods so i can stay relevant, yeah.
  6. Like Hinsdale all over again by SIGBUS · · Score: 1

    This sounds eerily like the 1988 fire at the switching center in Hinsdale, IL. Hopefully they didn't ignore alarms as happened then.

    --
    Oh, no! You have walked into the slavering fangs of a lurking grue!
  7. Primary and Secondary Power Systems together!!? by mspohr · · Score: 1

    You really have to hand it to these people. They put the primary and secondary power systems together in the same room.
    This is cluelessness on a grand scale.

    --
    I don't read your sig. Why are you reading mine?
    1. Re:Primary and Secondary Power Systems together!!? by Anonymous Coward · · Score: 0

      You really have to hand it to these people. They put the primary and secondary power systems together in the same room.
      This is cluelessness on a grand scale.

      I think the cluelessness is in people's failure to realize this was no accident. Clearly an IntelAgency-OP.

  8. Just Dallas/Fort Worth area, not all of Texas by itsownreward · · Score: 1

    I've got AT&T fiber down in the Houston area and it works just fine currently. Ditto with someone I know in the Austin area.

  9. Redundancy dammit. by Anonymous Coward · · Score: 0

    This is why you need two sources of internet (wireline and wireless), and why when AT&T says LTE is 10mbit 3G good enough for the rural bumpkins, smack them.

    1. Re: Redundancy dammit. by kenh · · Score: 1

      I'm in DFW, my U-Verse Internet is down, yet my AT&T cellphone works fine - it acts as a backup WiFi via hotspot service.

      --
      Ken
  10. Impossible. by msauve · · Score: 1

    IP routes around damage. It wasn't a fire at a facility which caused an outage, it was incorrect network design.

    --
    "National Security is the chief cause of national insecurity." - Celine's First Law
    1. Re: Impossible. by Anonymous Coward · · Score: 0

      It should be the same thing on cable like your local headend that feeds our house is going down not the overall internet

  11. Re: Primary and Secondary Power Systems together!! by Anonymous Coward · · Score: 0

    Sometimes fire equals kill all power also are you going to tell the firemen we can't push the big red button

  12. Re: Primary and Secondary Power Systems together!! by nehumanuscrede · · Score: 1

    Primary power = Commercial AC
    Secondary = Generators

    Battery strings and rectifiers in that mix somewhere. Trivia: Many telco systems are DC. Usually -48v

    Lightning strike probably destroyed batteries and / or rectifiers meaning even IF the generator came out unscathed, it would not matter.

    If it is a building that housed a pass-through fiber node, it will not take long to reroute it.

    If it is a fiber drop / handoff point, may take a bit longer but AT&T will roll their mobile crisis units out ( essentially 18-wheelers loaded with routing, switching and fiber hardware that can be sent just about anywhere with short notice ). Think of them as a mobile Central Office.

    I know we all like to hate on AT&T but I guarantee there is an army of folks on site doing everything they can to restore service.

  13. Re: Primary and Secondary Power Systems together!! by kenh · · Score: 1

    Right, they should have had completely redundant wiring for the two power sources, not just two power sources for the facility.

    At some point your redundant systems have to converge.

    If your company has two ISP connections, diversity of service, they likely have both routers in the same rack in the same closet.

    But no, you're right, they need to put two power rooms on alternate sides of the building, with both running to a third room, where there is a cutover in case one fails... oh wait, what if that room catches on fire? I guess they need to run parallel wiring to every outlet....

    Your fantasy of how to set up a redundant system has no place in the IT industry.

    --
    Ken
  14. Re: Primary and Secondary Power Systems together! by Anonymous Coward · · Score: 0

    Since you seem to know what you're talking about (or falling it pretty good), let me ask.. and please excuse my ignorance if I don't know what I'm talking about: Aren't lightning rods designed to attract lightning away from "sensitive areas"? I know there are lightning rods mounted on most (all?) major communications towers to avoid damage from lightning. If that is the case, wouldn't that have also prevented this from happening to the AT&T facilities?

  15. Re: Primary and Secondary Power Systems together! by Anonymous Coward · · Score: 0

    There are large copper unshielded grounding cables in telecom rooms. A lightning strike wouldn't cause a fire, rather it's the reverse, there likely was a failure to to ground something, and the lightning came in on whatever that was. Probably a wireless antenna.

    experience level: I've been in the telecoms room at my office building making a report of all the stupidity in it.

  16. Re: Primary and Secondary Power Systems together! by kenh · · Score: 1

    there likely was a failure to to ground something, and the lightning came in on whatever that was. Probably a wireless antenna.

    Seriously? Why would there be a wireless antenna there?

    --
    Ken
  17. I'm one of them by Anonymous Coward · · Score: 0

    no internet access right now.

  18. Re: Primary and Secondary Power Systems together!! by Anonymous Coward · · Score: 0

    Actually, yes. Our specs called for Power from "A" to run through the "A" generator panel and the left side of the rack. "B" ran on the right. We'd have to have a power event take out the automatic transfer switch for BOTH A and B (opposite sides of the building) to knock the entire facility offline.

    Now a fire in the rack.. that'll take down that rack. But this design is in building redundnacy 101.

  19. Re: Primary and Secondary Power Systems together! by SuiteSisterMary · · Score: 1

    Hmm. Out-of-band management connection, GPS antenna for providing 1PPS timing services, microwave backup link...

    --
    Vintage computer games and RPG books available. Email me if you're interested.
  20. Re:TRUMP THE LYING MORON PLAYED AGAIN! by Anonymous Coward · · Score: 0

    You forgot to sign in with your NPC account.