Red Hat Suffers Massive Data Center Network Outage

← Back to Stories (view on slashdot.org)

Red Hat Suffers Massive Data Center Network Outage

Posted by msmash on Friday April 21, 2017 @06:10AM from the outage-report dept.

An anonymous reader writes: According to multiple reports on Twitter, the Fedora Infrastructure Status page, and the #fedora-admin Freenode IRC channel, Red Hat is suffering a massive network outage at their primary data center. Details are sketchy at this point, but it looks to be impacting the Red Hat Customer Portal; as well as all their repositories (including Fedora, EPEL, Copr); their public build system, Koji; and a whole host of other popular services. There is no ETA for restoration of services at this point.

85 comments

Min score:

Reason:

Sort:

DeadHat !! by TechyImmigrant · 2017-04-21 04:27 · Score: 2

'Nuff said.

--
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
1. Re:DeadHat !! by Anonymous Coward · 2017-04-21 04:34 · Score: 5, Funny
  
  That's what you get from running systemd in production.
2. Re:DeadHat !! by Anonymous Coward · 2017-04-21 04:43 · Score: 1
  
  German agent Lennart Poettering
3. Re:DeadHat !! by JWW · 2017-04-21 04:44 · Score: 1
  
  Damn, I wish I had some mod points!!
  Mod this up!!!!
4. Re:DeadHat !! by courteaudotbiz · 2017-04-21 04:47 · Score: 2, Insightful
  
  I don't understand all this systemd bashing. I've been working with it for a few months on CentOS 7, and all I can say is that it is easy to work with and until now, it has proven to be very stable. Never had a crash related to systemd.
5. Re:DeadHat !! by Anonymous Coward · 2017-04-21 04:54 · Score: 0
  
  What a dumb comment
6. Re:DeadHat !! by MSG · 2017-04-21 04:59 · Score: 1, Funny
  
  Who rated this insightful? Humorous, maybe, but insightful? No. Come on, moderators. Unless AC is a Red Hat employee and knows what caused the outage, that's not what "insightful" means.
7. Re:DeadHat !! by StormReaver · 2017-04-21 05:12 · Score: 2
  
  I don't understand all this systemd bashing.
  A lot of people don't like change.
  There is one gripe, though, that I can sympathize with, and that's how systemd is expanding to encompass much more than is readily understandable. There may be perfectly good reasons for the expansion, but they're not readily apparent.
  That said, I've been using systemd ever since Kubuntu switched to it, and I haven't had any problems with it. But then, I haven't tried a recursive rm recently.
8. Re:DeadHat !! by Hognoxious · 2017-04-21 05:18 · Score: 1
  
  A few whole months, eh? Mind if we call you grandpa?
  
  --
  Confucius say, "Find worm in apple - bad. Find half a worm - worse."
9. Re: DeadHat !! by Anonymous Coward · 2017-04-21 05:25 · Score: 0
  
  It goes against time proven engineering practices, just because it enables easier time for redhat developing the cloud stuff.
  Any bugs/backdoors/etc, have the potential of messing up the system.
  It's like using a sword to open a beancan, yes it works, however people are using can openers for a reason.
  Systemd just wasn't engineered well in its core concepts. Everything else is noise.
10. Re:DeadHat !! by CanadianMacFan · 2017-04-21 05:26 · Score: 3, Insightful
  
  The problem is that systemd keeps on expanding and that goes against the philosophy of UNIX/Linux where each thing is kept small in scale and does it well. systemd keeps up integrating applications that have worked perfectly for a long time for the philosophy of one person who isn't really well respected in some areas of Linux development.
11. Re:DeadHat !! by Anonymous Coward · 2017-04-21 05:31 · Score: 1, Insightful
  
  Systemd is great, right up until something goes wrong or you find something that doesn't fit the World According to Poettering. You'll never be able to figure out why things aren't working, you'll never be able to integrate whatever it is you want to do into the new systemd world. Ever wonder why they keep adding more and more junk to systemd? Because it doesn't play nice with anything that isn't itself, so they can't use things that already exist and already work. Instead everything has to be thrown into systemd.
12. Re:DeadHat !! by F.Ultra · 2017-04-21 05:32 · Score: 4, Insightful
  
  Not exactly true. The systemd that you talk about (encompassing applications) are the systemd the project and not systemd the pid 1 (init) application. Each new "application integration" is done via a separate application so the UNIX philosophy still stands. And these are not done in order to match the philosophy of one person (the whole systemd project have lots of developers this day) but are done in order to present a common plumbing layer mostly aimed at container developers at this moment, i.e to present a common set of tools that work and look the same.
13. Re:DeadHat !! by courteaudotbiz · 2017-04-21 05:38 · Score: 1
  
  Yeah well, a few like nearly a year, and running a 24/7 infrastructure on it, it has been reliable as a swiss clock.
  
  Maybe I call you grandpa for refusing to move on from init.d that dates back to the 70s.
14. Re:DeadHat !! by tepples · 2017-04-21 05:43 · Score: 1
  
  But do these "separate application[s]" break if pid 1 is something other than systemd?
15. Re:DeadHat !! by Anonymous Coward · 2017-04-21 06:05 · Score: 0
  
  Basically it makes one whole class of problems easier to diagnose, catch, and automatically recover from.
  However, it creates a whole new class of problems to deal with. When it *thinks* it knows better than the administrator, when it tries to do something automatic that wasn't expected, the optimization for boot causing non-determinism in the boot process.
  The problem being that in script based init system, the behavior is pretty consistent (whether it is consistently good or bad) and it was easy to dig in to understand the flow of things for admins because the entire behavior was defined by script. With things being different boot to boot and with most of the smarts in compiled C code, an admin is a bit less likely to be able to self-support should something go wrong outside of the scenarios expected to fail.
16. Re:DeadHat !! by Anonymous Coward · 2017-04-21 06:14 · Score: 0
  
  Yeah well, a few like nearly a year, and running a 24/7 infrastructure on it, it has been reliable as a swiss clock.
  Well that's probably a bit of an overstatement, but yeah, we have a few hundred 7.x systems in production environments, and we haven't had an unmanageable or unpredictable number of problems with them.
17. Re:DeadHat !! by Hognoxious · 2017-04-21 06:17 · Score: 1
  
  It's DeadRat. You must be one of those multiply-pierced twats who thinks booting off an umbongo live CD makes you some kind of guru.
  
  --
  Confucius say, "Find worm in apple - bad. Find half a worm - worse."
18. Re:DeadHat !! by Anonymous Coward · 2017-04-21 06:18 · Score: 0
  
  What's there not to understand?
  SystemD basically turns Linux into Windows.
  When the day comes that SystemD needs to be completely re-done or replaced by something better, as happens with all software which degrades over time;
  it will be too ingrained into the whole ecosystem and people will only comprehend SystemD, where replacement will be almost impossible.
  And no, the argument that Linux is in a way the same problem doesn't excuse SystemD from becoming just another one. One negative doesn't excuse another.
19. Re:DeadHat !! by ZenShadow · 2017-04-21 07:17 · Score: 1
  
  It isn't just about stability. I run systemd in production on hundreds of servers, and have no stability or reliability problems with it.
  That doesn't mean I actually like it.
  The command naming is terrible; "systemctl" and "journalctl" do not roll off the keyboard for a touch typist. The logging is a mess. I could go on... ...and it just happened one day. There was little visible discussion, and little benefit outside the desktop.
  
  --
  -- sigs cause cancer.
20. Re: DeadHat !! by Anonymous Coward · 2017-04-21 07:19 · Score: 0
  
  Well said.
21. Re:DeadHat !! by F.Ultra · 2017-04-21 07:48 · Score: 1
  
  Of course not. There might however be some that requires a process on the other side of a D-BUS interface to speak the same events and namespace but that is the hardest dependency any of these applications might have.
22. Re:DeadHat !! by Anonymous Coward · 2017-04-21 07:57 · Score: 0
  
  So you have a philosophical and personal problem with systemd, and not a technical one. Thanks for your honesty.
23. Re:DeadHat !! by xanclic · 2017-04-21 08:42 · Score: 1
  
  Yeah, right, the Linux philosophy. For which Linux itself is a prime example, of course, since it is kept so small in scale and fulfills only its core task of being a kernel (which is elementary resource management, in case you were wondering).
24. Re:DeadHat !! by Anonymous Coward · 2017-04-21 09:00 · Score: 0
  
  Systemd is great, right up until something goes wrong or you find something that doesn't fit the World According to Poettering. You'll never be able to figure out why things aren't working, you'll never be able to integrate whatever it is you want to do into the new systemd world. Ever wonder why they keep adding more and more junk to systemd? Because it doesn't play nice with anything that isn't itself, so they can't use things that already exist and already work. Instead everything has to be thrown into systemd.
  Speak for yourself. I can figure out the problem when something goes wrong.
25. Re:DeadHat !! by Anonymous Coward · 2017-04-21 09:04 · Score: 0
  
  Basically it makes one whole class of problems easier to diagnose, catch, and automatically recover from.
  However, it creates a whole new class of problems to deal with. When it *thinks* it knows better than the administrator, when it tries to do something automatic that wasn't expected, the optimization for boot causing non-determinism in the boot process.
  The problem being that in script based init system, the behavior is pretty consistent (whether it is consistently good or bad) and it was easy to dig in to understand the flow of things for admins because the entire behavior was defined by script. With things being different boot to boot and with most of the smarts in compiled C code, an admin is a bit less likely to be able to self-support should something go wrong outside of the scenarios expected to fail.
  Speak for yourself. Debugging C code is much easier than debugging a mess of shell scripts, thank you very much.
26. Re:DeadHat !! by Anonymous Coward · 2017-04-21 09:07 · Score: 0
  
  You'll never be able to figure out why things aren't working
  Only if you are a completely clueless muppet, then yeah, you'll never be able to figure shit out. Anybody who actually knows what the hell they are doing can figure out shit just fine with systemd. systemd sucks, but so does everything else. sysvinit scripts are terrible, especially ones from third party vendors. systemd is just a slightly different headache than sysvinit.
  In short, I've got other shit to do than go on a crusade over an init daemon, either make your shit work the systemd way, or just ignore systemd and start your processes from a @reboot entry in your crontab.
  Meh.
27. Re:DeadHat !! by Anonymous Coward · 2017-04-21 09:09 · Score: 0
  
  What's there not to understand?
  SystemD basically turns Linux into Windows.
  When the day comes that SystemD needs to be completely re-done or replaced by something better, as happens with all software which degrades over time;
  it will be too ingrained into the whole ecosystem and people will only comprehend SystemD, where replacement will be almost impossible.
  And no, the argument that Linux is in a way the same problem doesn't excuse SystemD from becoming just another one. One negative doesn't excuse another.
  Correction: SystemD turns GNU/Linux into MacOS X.
  I always hear complaints that people switch to MacOS X because "its Unix that just works", yet when people try to make Linux more like it with tech such is SystemD, whole hell breaks loose.
  Nobody is forcing you to use SystemD. Feel free to use whatever the hell you want to use. You can even write your own system manager!
28. Re:DeadHat !! by sjames · 2017-04-21 09:42 · Score: 1
  
  This is a YMMV situation. Systemd seems to work OK these days as long as you aren't doing anything unusual. If you are, it can be damned near impossible to get it to do the right thing.
29. Re:DeadHat !! by jbengt · 2017-04-21 10:10 · Score: 1
  
  So you have a philosophical and personal problem with systemd, and not a technical one.
  Actually, whether you agree or not with the argument, CanadianMacFan did state a technical objection, not a personal one.
30. Re:DeadHat !! by Anonymous Coward · 2017-04-21 10:51 · Score: 0
  
  Its a great strategy to tar any and all disagreement with as 'opposition to change'. It's always worked historically to push unpopular decisions down on people. It's worked for Systemd.
  There are also a couple more essential tools needed in the PR handbook. Characterize disagreement as 'hostility', brand opposition as 'grey beard luddidites'. These are also know to work effectively to shut down discussion and opposition. You can escape accountability using any of these strategies.
31. Re: DeadHat !! by zaphirplane · 2017-04-21 11:43 · Score: 1
  
  I don't know about that, what I see is a lot of hand waving.
  How about a for instance, provide examples
32. Re:DeadHat !! by Anonymous Coward · 2017-04-21 11:59 · Score: 0
  
  Yeah well, a few like nearly a year, and running a 24/7 infrastructure on it, it has been reliable as a swiss clock.
  Maybe I call you grandpa for refusing to move on from init.d that dates back to the 70s.
  Except, of course, that there are systems still up and running on init since the 70s...
  Now get off my lawn.
33. Re:DeadHat !! by Anonymous Coward · 2017-04-21 11:59 · Score: 0
  
  Hello and welcome to the Linux School! Lesson #1: Aliases!
  alias whatfuckingever="systemctl"
  alias gimmethoselogsbruh="journalctl"
  echo "Win!"
  Lesson complete.
34. Re: DeadHat !! by sjames · 2017-04-21 12:07 · Score: 1
  
  It absolutely positively will not mount btrfs in degraded mode. It drops to the emergency shell.
  Same deal for RAID.
35. Re: DeadHat !! by Anonymous Coward · 2017-04-21 12:21 · Score: 0
  
  Technically, it won't even status OK to let you know your request was successful to start/stop/etc. It's more complicated and takes me back to windows cmd days by having to type rather just make my script that does the work.
  The simple and powerful of *nix it has ruined. Sure you have to know what your doing, so let's let windoze admins become *nix admins now. Just gotta learn different command structures. Yeah, well, windoze admins is harsh but I hope the point sticks.
36. Re: DeadHat !! by Zero__Kelvin · 2017-04-21 12:29 · Score: 1
  
  Keep in mind that Slashdot is no longer a place where the majority are competent. Over the past decade or more it has increasingly become a place for those who think they are experts because they have fixed all their friends computers ( each time reinstalling Windows without having a clue as to what the real problem was or how to fix it.) For example there was a guy recently who was convinced that since the systemd logs said "initiating shutdown" that it was systemd that was causing his system to sit down at a time he didn't want. I couldn't get him to realize that his WM Power Management settings were the precipitant. systemd said it was shutting down the system so systemd was the problem. These are the people that blame all their woes on systemd.
  
  --
  Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
37. Re:DeadHat !! by Tough+Love · 2017-04-21 15:58 · Score: 1
  
  Openstack, more probably.
  
  --
  When all you have is a hammer, every problem starts to look like a thumb.
38. Re:DeadHat !! by See+Attached · 2017-04-21 17:35 · Score: 1
  
  I have a few Linux Laptops that .. by nature go on and off all the time. Systemd is good here, in that it automates things, and perhaps is better at being plastic and adaptable. My serves stay up for a long time, so many features are a second/third order benefit. I've always seen server roles as static.. and if the app quits, you may want to have a DBA or App person review before restart. Has Systemd subsumed Pacemaker/Cman/Corosync? In short, I see Systemd more useful in desktop/laptop, and boring old init for servers? what do you all think? My customers have no idea if a web page is from a system that is systemd managed.. whats the diff?
  
  --
  Time for a new Political party in the US (or two!) One is off the rails Other cant pony up a leader.
39. Re:DeadHat !! by TechyImmigrant · 2017-04-22 03:13 · Score: 1
  
  It's DeadRat. You must be one of those multiply-pierced twats who thinks booting off an umbongo live CD makes you some kind of guru.
  It's my domain name you insensitive clod!
  
  --
  I should use this sig to advertise my book ISBN-13 : 978-1501515132.
40. Re: DeadHat !! by TechyImmigrant · 2017-04-22 04:42 · Score: 1
  
  It absolutely positively will not mount btrfs in degraded mode. It drops to the emergency shell.
  That's because btrfs has some self respect.
  
  --
  I should use this sig to advertise my book ISBN-13 : 978-1501515132.
41. Re:DeadHat !! by Anonymous Coward · 2017-04-22 07:09 · Score: 0
  
  Please reread his post again. That is not a technical objection. It is a philosophical one, and therefore a personal one. Going against the UNIX philosophy is NOT a technical objection.
42. Re:DeadHat !! by Anonymous Coward · 2017-04-24 18:09 · Score: 0
  
  Perhaps Redhat just published the beta of the next systemd version which has IRC and web servers in it?
Let those Linux trolls begin... by bogaboga · 2017-04-21 04:31 · Score: 1

But seriously, we need to find out what happened. I hope it's a hardware issue and not software.
1. Re:Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 04:46 · Score: 0
  
  Let's just say if they had been running Windows 8 server, they'd be up right now. Take from that what you will.
  I'll take from it that you're a dumbass.
2. Re: Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 04:47 · Score: 1
  
  One of the read heads on the primary tape feeder wasn't properly bathed in mineral oils this morn.
3. Re:Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 04:48 · Score: 0
  
  The title says network outage. That would imply its not a system-related outage, but rather an internet connection or switch issue. Level3 has a few outages going on right now.
4. Re:Let those Linux trolls begin... by courteaudotbiz · 2017-04-21 04:49 · Score: 1
  
  Just like the recent Amazon crash, I would bet the root cause is human.
5. Re:Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 04:59 · Score: 0
  
  J... I would bet the root cause is human.
  Anne-Droid says: "You are the weakest link. Goodbye."
6. Re:Let those Linux trolls begin... by __aaclcg7560 · 2017-04-21 05:13 · Score: 1
  
  That would imply its not a system-related outage, but rather an internet connection or switch issue.
  
  I used to get dial-up Internet through a one-man IPS for ten years. The owner had the unfortunate luck of losing his two connections from different vendors to his servers in an out-of-state data center at the same time. When calling into the local phone bank, the local server picked up but could go no further. It took the vendors a week to repair the lines. By the time the ISP came back up with an added third line to the data center, I had already migrated to a different ISP.
7. Re:Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 05:15 · Score: 0
  
  According to spot on #fedora-admin, "velociraptors got in again". In all seriousness, it appears to be an issue with the data center provider in Phoenix, not RedHat's infrastructure. "it's network, edge routers, we don't know anything more." There are staff from Red Hat onsite, though I'm not sure they can do much at this point. (I'm the article submitter)
8. Re:Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 05:25 · Score: 0
  
  Is that why cygwin is so slow right now?
9. Re:Let those Linux trolls begin... by fche · 2017-04-21 06:10 · Score: 1
  
  cygwin's updater does indeed talk to a server in the troubled data center
  hey, it's up again
10. Re:Let those Linux trolls begin... by Anonymous Coward · 2017-04-21 13:38 · Score: 0
  
  But seriously, we need to find out what happened. I hope it's a hardware issue and not software.
  It's most likely a politics issue.
Gust of wind by Anonymous Coward · 2017-04-21 04:38 · Score: 0

From Bill Gates arsehole
Confirmed - Customer Portal impacted by Anonymous Coward · 2017-04-21 04:46 · Score: 0

"sso.redhat.com has encountered an error, please try your request again."
1. Re: Confirmed - Customer Portal impacted by Anonymous Coward · 2017-04-21 04:52 · Score: 1
  
  Sso and SAML are brain dead concepts. People who use them deserve to be punished
2. Re: Confirmed - Customer Portal impacted by wezelboy · 2017-04-21 09:39 · Score: 1
  
  Oh we are on a daily basis.
M$!!11 by Merk42 · 2017-04-21 04:56 · Score: 1

$BadThing happened about $company||$person I like, therefore $conspiracy!!
Oh, wow... by __aaclcg7560 · 2017-04-21 05:00 · Score: 1

They should have used Ubuntu.
Can't they just run ... by fahrbot-bot · 2017-04-21 05:24 · Score: 4, Funny

... systemctl restart datacenter
(Okay, maybe only if systemd ran as PID 2 ...)

--
It must have been something you assimilated. . . .
1. Re:Can't they just run ... by Anonymous Coward · 2017-04-21 09:18 · Score: 0
  
  ... systemctl restart datacenter
  (Okay, maybe only if systemd ran as PID 2 ...)
  I heard that the problem is that the edge router of the third party data center in Phoenix is not running systemd so admins are trying to debug bunch of shell init scripts and they can't figure it out.
Should Have Used FreeBSD by Anonymous Coward · 2017-04-21 05:24 · Score: 0

No SystemD
systemd related? by Anonymous Coward · 2017-04-21 05:28 · Score: 0

Seriously, when you don't care about quality, you have more problems. When you have more problems and you swallow log messages, you make it take a lot longer to fix those problems.
if they were running Linux, this wouldn't... oh, by swschrad · 2017-04-21 05:28 · Score: 1

if they were running distributed... oh, wait.
guess it was a Beowulf cluster of Clusters.

--
if this is supposed to be a new economy, how come they still want my old fashioned money?
Systemd #WINNING by Drunkulus · 2017-04-21 05:37 · Score: 1

Uh oh... sounds like the datacenter team accidentally deployed the systemd package we cooked up for the North Korean missile program.
1. Re:Systemd #WINNING by Anonymous Coward · 2017-04-21 05:49 · Score: 0
  
  systemd just became self-aware... time to get to the fallout shelters..
DOG BITES MAN!!! by bwanagary · 2017-04-21 05:42 · Score: 1

Yes, I agree, it is BIG NEWS when a Linux distro has any kind of problem because it almost never happens. MS outages, not so much.
San Francisco has a massive network outage - I must've missed your post about that.
Come back after you grow up @msmash.
1. Re:DOG BITES MAN!!! by bwanagary · 2017-04-21 05:43 · Score: 1
  
  Yes, I agree, it is BIG NEWS when a Linux distro has any kind of problem because it almost never happens. MS outages, not so much.
  San Francisco has a massive network outage - I must've missed your post about that.
  Come back after you grow up @msmash.
  https://www.google.com/amp/s/h...
2. Re:DOG BITES MAN!!! by Tough+Love · 2017-04-21 16:05 · Score: 1
  
  Well, Red Hat is buying into a lot of crap that is only marginally Linux, and is far from what you could call obviously reliable. Like Openstack, Ceph, Gluster, that kind of thing. When you build your house high enough on a foundation of shit, it eventually sinks into it. Like, Openstack actually depends on MySQL for distributed consistency, how far do you think that frisbee will fly?
  
  --
  When all you have is a hammer, every problem starts to look like a thumb.
3. Re:DOG BITES MAN!!! by Anonymous Coward · 2017-04-22 05:29 · Score: 0
  
  Power outages in San Francisco and New York possibly due to huge solar flares.
Systemd by Anonymous Coward · 2017-04-21 05:54 · Score: 0

It's a common interface so there's a way to apply cgroups.
wrong distro by iggymanz · 2017-04-21 06:17 · Score: 1

they should have been running Oracle Unbreakable LInux and none of this would have happened!
*snicker*
1. Re:wrong distro by Anonymous Coward · 2017-04-21 09:47 · Score: 0
  
  You magnificent bastard.
  I haven't wanted mod points this bad in years.
Apps complexity by DrYak · 2017-04-21 07:34 · Score: 2

But do these "separate application[s]" break if pid 1 is something other than systemd?
Depends on the applications.
Boot-loader ? NTP clients ? these aren't deeply interdependent.
You could very much use these and then run openrc if you want.
These are just handled by systemd in the sens that these are program who are now developed by people who are on the systemd *project* team.
At most systemd might leverage boot-loader in the sense that it can more easily send parameters to it for the next boot.
Though other daemon might be much more interlinked with systemd's (the daemon running at PID 1) job of starting/stopping things.
(I'm thinking about daemon managing seats, sessions, and starting/stopping hardware).
These are important as over time, the Linux kernel is starting to go way much more advanced than standard POSIX behaviour.
Linux kernel, for example, offers cgroup isolation, namespaces, etc. (the facilities which are leveraged by containers such as LXC, Docker, Andbox...) which are definitely useful (separating session in different name-spaces, jailing some daemons in containers, automatically configuring the content of a container state-lessly)
The older bash-script-based mess of code that used to predate systemd has absolutely zero ability to leverage them.
SystemD is one rather successful way to deal with those things that wasn't provided by SysVinit.

--
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
1. Re:Apps complexity by drinkypoo · 2017-04-21 10:55 · Score: 2
  
  Linux kernel, for example, offers cgroup isolation, namespaces, etc. [...] The older bash-script-based mess of code that used to predate systemd has absolutely zero ability to leverage them.
  This is ignorant at best. Both cgroups and containers can be created and manipulated from the command line. Nobody bothered to do this as a PoC before going off on a tear and creating systemd.
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Re:if they were running Linux, this wouldn't... oh by Anonymous Coward · 2017-04-21 08:19 · Score: 0

Wrong RAID (Redundant Array of Inexpensive Datacenters) level, perhaps?
More like power outages in the USA by Anonymous Coward · 2017-04-21 08:24 · Score: 0

Seems that some big cities are dark and it took some datacenters too... https://www.inverse.com/article/30631-lax-sf-ny-power-outages
Ex-Ubuntu users by Anonymous Coward · 2017-04-21 08:55 · Score: 0

Its all the Ubuntu users switching over to Fedora after the latest Canonical debacle.
Fedora Infrastructure up when I looked at 9 pm PST by Anonymous Coward · 2017-04-21 17:05 · Score: 0

At about 9:30 PM California time, the Fedora Infrastructure web page came up with every server showing Everything seems to be working.
About as strange as the news item itself is the absence from Slashdot postings from a system manager who can make a plausible guess as to what specifically would have set off a "massive outage"?
About 18 years ago, I failed the Red Hat Certified Network Engineer one week class and exam. I felt the exam was heavily loaded with "stupid dorm tricks" which are stupid things you can do to a Red Hat or Linux system to make it do strange things at boot time.
When this Red Hat event is finally reported, I will be looking to see what is the state of the art in tricks to screw up an enterprise cluster.
Re:Fedora Infrastructure up when I looked at 9 pm by See+Attached · 2017-04-21 17:43 · Score: 1

Looking forward to that reveal. Was it a hack/os abuse or Infra? 14 Hours now out? WHat is going on? Press is quiet too...

--
Time for a new Political party in the US (or two!) One is off the rails Other cant pony up a leader.
Re:Fedora Infrastructure up when I looked at 9 pm by Anonymous Coward · 2017-04-21 18:52 · Score: 0

From status.redhat.com: Update - Network issues are stabilized and we are working bring applications back online. Apr 21, 23:55 EDT Update - Connectivity between the Internet and the main production data center has been restored, however the network is not fully stable yet. Network team is fully engaged with the vendor and working hard to restore services. Apr 21, 19:25 EDT Identified - We have identified the issue and are working to resolve the problem. Apr 21, 13:52 EDT Investigating - Widespread system issues are causing parts of the Customer Portal to become non-responsive. We are working to resolve and will update this incident with relevant details. Apr 21, 11:13 EDT
command-line by DrYak · 2017-04-22 20:52 · Score: 1

Both cgroups and containers can be created and manipulated from the command line. Nobody bothered to do this as a PoC before going off on a tear and creating systemd.
There were command-line demos of cgroup/namespace (video of devs launching "make -j 255" while the desktop remains responsive).
What nobody bothered, is trying to rewrite the mass amount of bash script to support it.
(Specially since nearly every distro seems to write their own script madness to handle starting/stopping jobs *).
You would either need each single distro to rewrite mass amount of in-house developed shell code in order to leverage such newer kernel functionnality.
Or, you need that a few standard component that leverage the facility for you an simplify using it.
For obvious resource-saving reasons, most people went for the 2nd option.
Systemd was one of such developed facilities, and is the one who ended up the most popular. (By redhat who developed it. But also by other distributions who picked it up very early like suse).
And as usual, Canonical went for their own "NIH-syndrom" solution (upstart), before eventually joining everyone else.
---
* : which is another advantage of systemd.
Most distribution wrote their own rc?.d scripts, usually with tons of boiler-plate code (for starting, checking status, etc.)
systemd relies on much smaller simpler static configuration files - easier to edit, also easier to share.
It's easier to maintain the the content of rc?d.
On the other hand, unlike SysVInit, where much of their processing is done by calling bash (i.e.: outside of PID 1), systemd move a little bit more functionality in there (into PID 1) in order to be able to interpret the conf files.

--
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]