Red Hat Suffers Massive Data Center Network Outage

← Back to Stories (view on slashdot.org)

Red Hat Suffers Massive Data Center Network Outage

Posted by msmash on Friday April 21, 2017 @06:10AM from the outage-report dept.

An anonymous reader writes: According to multiple reports on Twitter, the Fedora Infrastructure Status page, and the #fedora-admin Freenode IRC channel, Red Hat is suffering a massive network outage at their primary data center. Details are sketchy at this point, but it looks to be impacting the Red Hat Customer Portal; as well as all their repositories (including Fedora, EPEL, Copr); their public build system, Koji; and a whole host of other popular services. There is no ETA for restoration of services at this point.

46 of 85 comments (clear)

Min score:

Reason:

Sort:

DeadHat !! by TechyImmigrant · 2017-04-21 04:27 · Score: 2

'Nuff said.

--
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
1. Re:DeadHat !! by Anonymous Coward · 2017-04-21 04:34 · Score: 5, Funny
  
  That's what you get from running systemd in production.
2. Re:DeadHat !! by Anonymous Coward · 2017-04-21 04:43 · Score: 1
  
  German agent Lennart Poettering
3. Re:DeadHat !! by JWW · 2017-04-21 04:44 · Score: 1
  
  Damn, I wish I had some mod points!!
  Mod this up!!!!
4. Re:DeadHat !! by courteaudotbiz · 2017-04-21 04:47 · Score: 2, Insightful
  
  I don't understand all this systemd bashing. I've been working with it for a few months on CentOS 7, and all I can say is that it is easy to work with and until now, it has proven to be very stable. Never had a crash related to systemd.
5. Re:DeadHat !! by MSG · 2017-04-21 04:59 · Score: 1, Funny
  
  Who rated this insightful? Humorous, maybe, but insightful? No. Come on, moderators. Unless AC is a Red Hat employee and knows what caused the outage, that's not what "insightful" means.
6. Re:DeadHat !! by StormReaver · 2017-04-21 05:12 · Score: 2
  
  I don't understand all this systemd bashing.
  A lot of people don't like change.
  There is one gripe, though, that I can sympathize with, and that's how systemd is expanding to encompass much more than is readily understandable. There may be perfectly good reasons for the expansion, but they're not readily apparent.
  That said, I've been using systemd ever since Kubuntu switched to it, and I haven't had any problems with it. But then, I haven't tried a recursive rm recently.
7. Re:DeadHat !! by Hognoxious · 2017-04-21 05:18 · Score: 1
  
  A few whole months, eh? Mind if we call you grandpa?
  
  --
  Confucius say, "Find worm in apple - bad. Find half a worm - worse."
8. Re:DeadHat !! by CanadianMacFan · 2017-04-21 05:26 · Score: 3, Insightful
  
  The problem is that systemd keeps on expanding and that goes against the philosophy of UNIX/Linux where each thing is kept small in scale and does it well. systemd keeps up integrating applications that have worked perfectly for a long time for the philosophy of one person who isn't really well respected in some areas of Linux development.
9. Re:DeadHat !! by Anonymous Coward · 2017-04-21 05:31 · Score: 1, Insightful
  
  Systemd is great, right up until something goes wrong or you find something that doesn't fit the World According to Poettering. You'll never be able to figure out why things aren't working, you'll never be able to integrate whatever it is you want to do into the new systemd world. Ever wonder why they keep adding more and more junk to systemd? Because it doesn't play nice with anything that isn't itself, so they can't use things that already exist and already work. Instead everything has to be thrown into systemd.
10. Re:DeadHat !! by F.Ultra · 2017-04-21 05:32 · Score: 4, Insightful
  
  Not exactly true. The systemd that you talk about (encompassing applications) are the systemd the project and not systemd the pid 1 (init) application. Each new "application integration" is done via a separate application so the UNIX philosophy still stands. And these are not done in order to match the philosophy of one person (the whole systemd project have lots of developers this day) but are done in order to present a common plumbing layer mostly aimed at container developers at this moment, i.e to present a common set of tools that work and look the same.
11. Re:DeadHat !! by courteaudotbiz · 2017-04-21 05:38 · Score: 1
  
  Yeah well, a few like nearly a year, and running a 24/7 infrastructure on it, it has been reliable as a swiss clock.
  
  Maybe I call you grandpa for refusing to move on from init.d that dates back to the 70s.
12. Re:DeadHat !! by tepples · 2017-04-21 05:43 · Score: 1
  
  But do these "separate application[s]" break if pid 1 is something other than systemd?
13. Re:DeadHat !! by Hognoxious · 2017-04-21 06:17 · Score: 1
  
  It's DeadRat. You must be one of those multiply-pierced twats who thinks booting off an umbongo live CD makes you some kind of guru.
  
  --
  Confucius say, "Find worm in apple - bad. Find half a worm - worse."
14. Re:DeadHat !! by ZenShadow · 2017-04-21 07:17 · Score: 1
  
  It isn't just about stability. I run systemd in production on hundreds of servers, and have no stability or reliability problems with it.
  That doesn't mean I actually like it.
  The command naming is terrible; "systemctl" and "journalctl" do not roll off the keyboard for a touch typist. The logging is a mess. I could go on... ...and it just happened one day. There was little visible discussion, and little benefit outside the desktop.
  
  --
  -- sigs cause cancer.
15. Re:DeadHat !! by F.Ultra · 2017-04-21 07:48 · Score: 1
  
  Of course not. There might however be some that requires a process on the other side of a D-BUS interface to speak the same events and namespace but that is the hardest dependency any of these applications might have.
16. Re:DeadHat !! by xanclic · 2017-04-21 08:42 · Score: 1
  
  Yeah, right, the Linux philosophy. For which Linux itself is a prime example, of course, since it is kept so small in scale and fulfills only its core task of being a kernel (which is elementary resource management, in case you were wondering).
17. Re:DeadHat !! by sjames · 2017-04-21 09:42 · Score: 1
  
  This is a YMMV situation. Systemd seems to work OK these days as long as you aren't doing anything unusual. If you are, it can be damned near impossible to get it to do the right thing.
18. Re:DeadHat !! by jbengt · 2017-04-21 10:10 · Score: 1
  
  So you have a philosophical and personal problem with systemd, and not a technical one.
  Actually, whether you agree or not with the argument, CanadianMacFan did state a technical objection, not a personal one.
19. Re: DeadHat !! by zaphirplane · 2017-04-21 11:43 · Score: 1
  
  I don't know about that, what I see is a lot of hand waving.
  How about a for instance, provide examples
20. Re: DeadHat !! by sjames · 2017-04-21 12:07 · Score: 1
  
  It absolutely positively will not mount btrfs in degraded mode. It drops to the emergency shell.
  Same deal for RAID.
21. Re: DeadHat !! by Zero__Kelvin · 2017-04-21 12:29 · Score: 1
  
  Keep in mind that Slashdot is no longer a place where the majority are competent. Over the past decade or more it has increasingly become a place for those who think they are experts because they have fixed all their friends computers ( each time reinstalling Windows without having a clue as to what the real problem was or how to fix it.) For example there was a guy recently who was convinced that since the systemd logs said "initiating shutdown" that it was systemd that was causing his system to sit down at a time he didn't want. I couldn't get him to realize that his WM Power Management settings were the precipitant. systemd said it was shutting down the system so systemd was the problem. These are the people that blame all their woes on systemd.
  
  --
  Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
22. Re:DeadHat !! by Tough+Love · 2017-04-21 15:58 · Score: 1
  
  Openstack, more probably.
  
  --
  When all you have is a hammer, every problem starts to look like a thumb.
23. Re:DeadHat !! by See+Attached · 2017-04-21 17:35 · Score: 1
  
  I have a few Linux Laptops that .. by nature go on and off all the time. Systemd is good here, in that it automates things, and perhaps is better at being plastic and adaptable. My serves stay up for a long time, so many features are a second/third order benefit. I've always seen server roles as static.. and if the app quits, you may want to have a DBA or App person review before restart. Has Systemd subsumed Pacemaker/Cman/Corosync? In short, I see Systemd more useful in desktop/laptop, and boring old init for servers? what do you all think? My customers have no idea if a web page is from a system that is systemd managed.. whats the diff?
  
  --
  Time for a new Political party in the US (or two!) One is off the rails Other cant pony up a leader.
24. Re:DeadHat !! by TechyImmigrant · 2017-04-22 03:13 · Score: 1
  
  It's DeadRat. You must be one of those multiply-pierced twats who thinks booting off an umbongo live CD makes you some kind of guru.
  It's my domain name you insensitive clod!
  
  --
  I should use this sig to advertise my book ISBN-13 : 978-1501515132.
25. Re: DeadHat !! by TechyImmigrant · 2017-04-22 04:42 · Score: 1
  
  It absolutely positively will not mount btrfs in degraded mode. It drops to the emergency shell.
  That's because btrfs has some self respect.
  
  --
  I should use this sig to advertise my book ISBN-13 : 978-1501515132.
Let those Linux trolls begin... by bogaboga · 2017-04-21 04:31 · Score: 1

But seriously, we need to find out what happened. I hope it's a hardware issue and not software.
1. Re: Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 04:47 · Score: 1
  
  One of the read heads on the primary tape feeder wasn't properly bathed in mineral oils this morn.
2. Re:Let those Linux trolls begin... by courteaudotbiz · 2017-04-21 04:49 · Score: 1
  
  Just like the recent Amazon crash, I would bet the root cause is human.
3. Re:Let those Linux trolls begin... by __aaclcg7560 · 2017-04-21 05:13 · Score: 1
  
  That would imply its not a system-related outage, but rather an internet connection or switch issue.
  
  I used to get dial-up Internet through a one-man IPS for ten years. The owner had the unfortunate luck of losing his two connections from different vendors to his servers in an out-of-state data center at the same time. When calling into the local phone bank, the local server picked up but could go no further. It took the vendors a week to repair the lines. By the time the ISP came back up with an added third line to the data center, I had already migrated to a different ISP.
4. Re:Let those Linux trolls begin... by fche · 2017-04-21 06:10 · Score: 1
  
  cygwin's updater does indeed talk to a server in the troubled data center
  hey, it's up again
Re: Confirmed - Customer Portal impacted by Anonymous Coward · 2017-04-21 04:52 · Score: 1

Sso and SAML are brain dead concepts. People who use them deserve to be punished
M$!!11 by Merk42 · 2017-04-21 04:56 · Score: 1

$BadThing happened about $company||$person I like, therefore $conspiracy!!
Oh, wow... by __aaclcg7560 · 2017-04-21 05:00 · Score: 1

They should have used Ubuntu.
Can't they just run ... by fahrbot-bot · 2017-04-21 05:24 · Score: 4, Funny

... systemctl restart datacenter
(Okay, maybe only if systemd ran as PID 2 ...)

--
It must have been something you assimilated. . . .
if they were running Linux, this wouldn't... oh, by swschrad · 2017-04-21 05:28 · Score: 1

if they were running distributed... oh, wait.
guess it was a Beowulf cluster of Clusters.

--
if this is supposed to be a new economy, how come they still want my old fashioned money?
Systemd #WINNING by Drunkulus · 2017-04-21 05:37 · Score: 1

Uh oh... sounds like the datacenter team accidentally deployed the systemd package we cooked up for the North Korean missile program.
DOG BITES MAN!!! by bwanagary · 2017-04-21 05:42 · Score: 1

Yes, I agree, it is BIG NEWS when a Linux distro has any kind of problem because it almost never happens. MS outages, not so much.
San Francisco has a massive network outage - I must've missed your post about that.
Come back after you grow up @msmash.
1. Re:DOG BITES MAN!!! by bwanagary · 2017-04-21 05:43 · Score: 1
  
  Yes, I agree, it is BIG NEWS when a Linux distro has any kind of problem because it almost never happens. MS outages, not so much.
  San Francisco has a massive network outage - I must've missed your post about that.
  Come back after you grow up @msmash.
  https://www.google.com/amp/s/h...
2. Re:DOG BITES MAN!!! by Tough+Love · 2017-04-21 16:05 · Score: 1
  
  Well, Red Hat is buying into a lot of crap that is only marginally Linux, and is far from what you could call obviously reliable. Like Openstack, Ceph, Gluster, that kind of thing. When you build your house high enough on a foundation of shit, it eventually sinks into it. Like, Openstack actually depends on MySQL for distributed consistency, how far do you think that frisbee will fly?
  
  --
  When all you have is a hammer, every problem starts to look like a thumb.
wrong distro by iggymanz · 2017-04-21 06:17 · Score: 1

they should have been running Oracle Unbreakable LInux and none of this would have happened!
*snicker*
Apps complexity by DrYak · 2017-04-21 07:34 · Score: 2

But do these "separate application[s]" break if pid 1 is something other than systemd?
Depends on the applications.
Boot-loader ? NTP clients ? these aren't deeply interdependent.
You could very much use these and then run openrc if you want.
These are just handled by systemd in the sens that these are program who are now developed by people who are on the systemd *project* team.
At most systemd might leverage boot-loader in the sense that it can more easily send parameters to it for the next boot.
Though other daemon might be much more interlinked with systemd's (the daemon running at PID 1) job of starting/stopping things.
(I'm thinking about daemon managing seats, sessions, and starting/stopping hardware).
These are important as over time, the Linux kernel is starting to go way much more advanced than standard POSIX behaviour.
Linux kernel, for example, offers cgroup isolation, namespaces, etc. (the facilities which are leveraged by containers such as LXC, Docker, Andbox...) which are definitely useful (separating session in different name-spaces, jailing some daemons in containers, automatically configuring the content of a container state-lessly)
The older bash-script-based mess of code that used to predate systemd has absolutely zero ability to leverage them.
SystemD is one rather successful way to deal with those things that wasn't provided by SysVinit.

--
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
1. Re:Apps complexity by drinkypoo · 2017-04-21 10:55 · Score: 2
  
  Linux kernel, for example, offers cgroup isolation, namespaces, etc. [...] The older bash-script-based mess of code that used to predate systemd has absolutely zero ability to leverage them.
  This is ignorant at best. Both cgroups and containers can be created and manipulated from the command line. Nobody bothered to do this as a PoC before going off on a tear and creating systemd.
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Re: Confirmed - Customer Portal impacted by wezelboy · 2017-04-21 09:39 · Score: 1

Oh we are on a daily basis.
Re:Fedora Infrastructure up when I looked at 9 pm by See+Attached · 2017-04-21 17:43 · Score: 1

Looking forward to that reveal. Was it a hack/os abuse or Infra? 14 Hours now out? WHat is going on? Press is quiet too...

--
Time for a new Political party in the US (or two!) One is off the rails Other cant pony up a leader.
command-line by DrYak · 2017-04-22 20:52 · Score: 1

Both cgroups and containers can be created and manipulated from the command line. Nobody bothered to do this as a PoC before going off on a tear and creating systemd.
There were command-line demos of cgroup/namespace (video of devs launching "make -j 255" while the desktop remains responsive).
What nobody bothered, is trying to rewrite the mass amount of bash script to support it.
(Specially since nearly every distro seems to write their own script madness to handle starting/stopping jobs *).
You would either need each single distro to rewrite mass amount of in-house developed shell code in order to leverage such newer kernel functionnality.
Or, you need that a few standard component that leverage the facility for you an simplify using it.
For obvious resource-saving reasons, most people went for the 2nd option.
Systemd was one of such developed facilities, and is the one who ended up the most popular. (By redhat who developed it. But also by other distributions who picked it up very early like suse).
And as usual, Canonical went for their own "NIH-syndrom" solution (upstart), before eventually joining everyone else.
---
* : which is another advantage of systemd.
Most distribution wrote their own rc?.d scripts, usually with tons of boiler-plate code (for starting, checking status, etc.)
systemd relies on much smaller simpler static configuration files - easier to edit, also easier to share.
It's easier to maintain the the content of rc?d.
On the other hand, unlike SysVInit, where much of their processing is done by calling bash (i.e.: outside of PID 1), systemd move a little bit more functionality in there (into PID 1) in order to be able to interpret the conf files.

--
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]