Slashdot Mirror
Red Hat's Linux Changes Raise New Questions
itwbennett writes "Last month two Red Hat developers proposed to replace the 30-year-old syslog system with a new Journal daemon. Initial reaction was mostly negative and 'focused on the Journal's use of a binary key-value form of data to log system events,' says blogger Brian Proffit. But now, says Proffitt, it seems that the proposal to replace syslog has less to do with the fixing syslog's problems than with Red Hat's desire to go its own way with Linux infrastructure."
That's one of the advantages of Linux: RedHat can go their own way without needing the rest of us to buy in, and without really messing things up for us. If they provide a reasonable API, it'll either be compatible with syslog with a simple library substitution or we'll quickly see a wrapper library that allows programs to use either syslog or Journal without needing code changes.
I think going to binary's a bad idea, myself. The fewer tools you need working to find out what the error is, the easier it is to debug and fix the problem. But let RedHat try this and see how it works, and then we can decide once we've got some real-world data to compare.
WTF!? First post and the linked article is already slashdotted?
When everything else is failing ... you still need to be able to dig into the the syslogs reliably no matter what! One little hiccup and you can easily lose everything in most binary type implementations, while at worst you see a little garbage in the syslogs!
Keep on fragmenting each distro ... at a certain point, people will just get tired of distro-hopping and dump the whole mess.
And people ask when the Year f the Linux Desktop will be. It's things likie this, and the constant breakage because of change for the sake of change or to "be different", rather than focusing on stability, that drive people to non-free vendors.
Not that it bothers me, but in forums people are quick to point out that they think Fedora's choice of kernel numbering is stupid. I mention I'm on 2.6.41.1-1.fc15.x86_64, and the first response is, "that kernel doesn't exist." (And yes, Fedora will move to the standard numbering scheme with 17 if I'm not mistaken)
I've found most of RH's decisions to do something their way is to prevent problems down the road. Same for kernel numbering, it was supposedly to prevent repo errors. I don't know for certain, but I'd expect this to also be the case here.
Absolute power corrupts absolutely. indymedia
Isn't this the point of Linux?
This is just whining by some guy who wrote a log analyzer that will no longer be necessary.
QNX has had a simple structured log daemon for years. Reading their log never tails off into junk; you always get a clean, current last record. Their solution even works on diskless systems. In many real-time applications, logs are transmitted to some remote location, rather than being kept on each local machine.
But what they're doing is going into Fedora, which is a much larger community than and more than just Red Hat.
And just because it's in Fedora does not guarantee that it'll end up in RHEL.
I'm not convinced that there's anything wrong with syslog any more than I was convinced there was anything wrong with the SysV init, but the thing about F/OSS is that to get your chops you have to own something that's important. At one end of the spectrum you've got people who own a little perl module or two, and at the other end you've got the big cheese himself, and everyone in between.
I'm not sure that what these developers aren't doing is trying to establish some high ground for themselves. Squeaky wheels get the grease and all that.
Is he not aware how terrible syslog is? syslog is ancient and has several series flaws from security to just stupid limitations. It should have been replaced ages ago.
> people would rather pay a couple hundred dollars for windows or the premium involved with buying a mac than use linux for free. what does that tell you?
What does it tell me? You are a liar.
Very few people in fact pay for the premium for a Mac.
On the other hand, most people use Windows because "most people use Windows". It has been that way since Macs were an MC68k based platform. This leads to little things like software not being available for Macs or AIO printer devices not working on Macs.
How any of them handle any particular technical detail is largely irrelevant.
Besides, we're talking about an enterprise server vendor we are talking about here. RHEL is not some MacOS wannabe.
A Pirate and a Puritan look the same on a balance sheet.
I can't read the article with all of the annoying scrolling on the right panel. No, I won't resize my browser just for that.
Didn't Ubuntu already change the original implementation of syslog as specified in the RFC? Can anyone name me a current popular and wide spread distribution which uses the original syslog? All red hat is doing is upgrade a dead standard to something modern.
> people would rather pay a couple hundred dollars for windows or the premium involved with buying a mac than use linux for free. what does that tell you?
What does it tell me? You are a liar.
Very few people in fact pay for the premium for a Mac.
OMG, you finally admit that the Mac 'premium' is illusory. No, that can't be it. It would make too much sense.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
I think the world would be better off if RedHat went off and annoyed some other planet. First dbus, and now this. Why in the name of all that's holy are they making simple things complicated?
You will also be stuck with all the good choices they make.
Reading what they are proposing it seems that is actually a very good idea. When you get out of hobbyist and small environments and into environments with more demanding requirements about security auditing the traditional syslog has not cut it for years anymore. The first step in many environments is usually to rip it mostly off and replace with some more or less proprietary environment.
The new ideas such as improving the reliability of log shipping, reducing possibilities towards tampering, and improving chances for more advanced log analysis are really awesome things - especially for people who are serious about their logging. Syslog and its text format are legacy poison and it will be good to see them die and vanish. Hopefully that happens fast.
Also, keep in mind that that RedHat is still open sourcing that stuff. They will provide tools and APIs - as they require those also themselves.
It seems like every time a distro tries to innovate they get a lot of screaming from the linux community.
There is this change, the screaming about Ubuntu going with Unity, screaming with every change GNOME makes.
Is the FOSS really about innovation or just mouthing the words?
Dear Proffitt,
My name is Inncommee. You should remove that extra f and t letters in your name.
It's a good move. Parsing syslog sucks. And I don't care how awesome you think you are as a developer--you need to use the system logging facilities to make it easier on those of us who adminster systems.
At the very least a unified format similar to Microsoft's format would be nice.
ID / DATE-Time / Severity / BLOB OF TEXT
I'm not sure, but I get the feeling that different groups in the opensource community are struggling to get control of their platform. Gnome peeps are doing their own thing, Ubuntu heads off in another direction. Red Hat does their own things.
The last 8 years were somewhat mixed in this regard. There was cooperation, like on freedesktop.org, but olso fragmentation and diversification. Now it all seems to fall apart somewhat. I don't see the different groups come together.
I'm really not fond of some things that are happening, like Systemd and all the other incompatible SysVinit systems. Also the mess that are the main desktops now. Then this new syslog proposal. I doubt other distro's will take this, I expect they will stick with syslog or syslog-ng.
For myself I think I'm going with Debian (testing that is) soon. Once old-school just meant old stuff, but nowadays it almost sounds like the best thing there is. All the new software with less bugs, but not the crummy new inventions which you'd rather let pass by.
Well, don't worry about that. We can get you back before you leave. (Dr. Who)
Just so we get your story straight, Mr. Blogger - when media darling Ubuntu trashes 30 years of platform compatibility and portability by moving away from X11, technology pundits like yourself praise them for being forward-looking and innovative. When Red Hat proposes a better mechanism for system logging that is less susceptible to spoofing log entries, for example, you crucify them on your blog for demonstrating the same qualities?
Hypocrites. At least be consistent, if not objective.
OMG, you finally admit that the Mac 'premium' is illusory. No, that can't be it. It would make too much sense.
Show me an Apple product where I can trivially change the video card, CPU and add a second drive. There is only one, and that's the massively expensive mac pro. There is no mac pro for $500, there isn't one for $1000, nor is there one for $1500, not event for $2000. 99% of people buying PCs aren't spending two grand, let alone $2500 for the lowest mac pro. iMacs are not upgradable, even swapping the single harddrive means a three hour job of dismantling the entire machine to get the screen off before you can access the drive. Never had a drive go bad, or need to install a bigger one?
Yup, Mac premium is very real because Apple make a ton of cash from fashion items and dweebs thinking they're l337 buying the gear with the most obsolescent built in.
OMG, you finally admit that the Mac 'premium' is illusory. No, that can't be it. It would make too much sense.
You sir, are an idiot and a shill.
another reason people don't use linux because when they need help: they have to deal with assholes like you.
It tells you that people want a Mac, it says nothing about the technical merits of the system; youre simply assuming that their reasons for choosing Mac OSX are technical in nature, or that they have heard of Linux, or understand the difference between a GUI and an OS.
Redhat's solution does not prevent tampering of the log files at all.
The server got to have a certificate and a key to log entries in the log according to the Redhat solution. Yes, you can only read the log using a key that is NOT stored on the server, however, the hacker DO have access to
1) modify system time at will
2) edit the syslog configuration of how to log entries
3) Can delete the current logs
4) Can generate new logs using the key and the certificate located on the server to make it appear as if everything is fine...
What an encrypted log DOES help with is preventing the hacker from gaining additional knowledge from the already existing logs.
Ever had the experience of typing too quick and ending up typing your password in the Login field? Since usernames are being logged, any user who can read the log can now read your password... Unless the logs are encrypted and you need a key not on the server that is...
blockquote fail
It is a minor source of amusement to me to periodically price out a MacBook, and then find an equivalent model on NewEgg, to find out just how many spare, identical laptops I could purchase for the same cost of the Macbook.
Last time I checked, I believe it was 3-- that is, every year for 3 years, I could toss out my smudged, scratched PC laptop and start up on a new one, and it would still make more financial sense than getting the Mac.
Just in case you dont believe me,
http://imgur.com/kvUkV
Whats that, the comparison isnt fair? The PC has better specs than the mac, and is half the price? Yea, well, thats what you get when you get a Mac, noone said the comparison was fair.
linux by just doing what we've always done. if we did, Linus would still be grading papers, Stallman would probably use use BSD, and the phone in my pocket would probably never exist.
part of what makes me love linux is the undying urge to try something new. granted, thats not everyones opinion. its a bunch of nerds and hackers and really cool people coming together and having the courage to say, "i just made this new thing."
To Red Hat: thanks for trying something new. i really hope it works out and im eager to try it too. just remember, haters are always going to hate. and because its the community that makes up linux, theres a linux for them too.
Good people go to bed earlier.
Please bear in mind that not everything RH developpers do is directed
or funded by RH.
Besides, any bigger feature targeted for RHEL would likely go to
Fedora first. This is where you can have your say.
Read Rainer's blog first and if you think journald is
not the way to go, do something about it.
Did Red Hat rape and murder a girl in 1990?
I've been using "ip" for at least 8 years now...it actually allows you to assign multiple IP addresses to a single network device. I don't know how anyone lives with ifconfig anymore.
If you have write access to the log file you can delete/change the entry you care about and then re-hash all the entries after that.
I think the move to binary storage for syslog files could be great for efficiency all the way around. A very simple CLI tool that dumps the ASCII syslog equivalent would make for a very nice transition piece.
You could continue using your existing syslog-based tools to monitor / alert / debug / whatever without having to change much at all. As an added bonus, the tool could accept optional search & filter parameters that are applied to the binary form before dumping ASCII output. That would save the CPU a bit of time to grep through thousands of lines of unrelated logs just to report on the one or two system services that you want to monitor.
I went to eat some animal crackers and the box said, "Do not eat if seal is broken." I opened the box and sure enough..
because its all tied together with a brilliant OS that people want to use. mash all the awesome hardware you want into a linux box and it still sucks herpes cock to use, if you can even use it for what you need to do with a computer.
The difference is that the Macbook will last 5+ years and the HP will be replaced in one.
So you can choose which mail server or java vm you use but not which syslog daemon you want to use?
Init has been broken for years and years on modern systems where hardware (such as the network) comes and goes.
There were unsolvable races in device setup, etc. which necessitated unbelievably ugly hacks such as "Waiting for devices to settle" pauses of a few seconds during startup.
Runtime configuration of wireless networking with networked services running on laptops is another example.
- False Dichotomy, this is metamatic.
- metamatic, this is False Dichotomy.
HAND.
Baloney, which part are you implying is inferior? The intel manufactured processor, the seagate manufactured hard drive, the kingston RAM, or the FoxConn motherboard?
As for 1 year, in all my time helping family and friends with computers, the least ive seen a laptop last is 3 years. So 3 years down the road the PC guy gets a brand new laptop with brand new processor and battery technology, while the Mac is still chugging on outdated hardware, and both have spent the same-- how is that a good deal for the HP dude?
Just to be clear, my home-built PC cost about $700 5.5 years ago and is still chugging along. An equivalent Mac (running the then-new Core2 series) would have probably cost around $1400. You telling me that mac would have another 5 years in it and that my desktop is set to die in a year or so? Care to make any bets on that?
how is that a good deal for the HP dude?
Should be "how is that NOT a good deal"
128-bit UUIDs are an idea that came out of the Distributed Computing Environment (DCE) project that Microsoft seized and ran with as the mechanism for generating unique identifiers for their COM/DCOM objects. Have a look at the Windows registry (using regedit.exe) to see what the result has been. Huge swaths of the registry are now completely unintelligible because of the reliance on cross-referenced UUIDs, which are impossible for humans to remember. It reminds me of Steve Jobs' famous comment about Microsoft: "They have no taste."
Not saying that the new daemon is a bad idea, but the inclusion of UUIDs in the proposal makes me think that the whole thing needs community review.
Who the fuck is this "Noone" guy? Sounds like that dude drinks way too much coffee.
there's no reason you can't have a substitute that logs both to their format and to a classic syslog... or a daemon that creates named pipes to let you view the logs from the database
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
There's a shocker. A publicly held company that wants to create vendor lock-in... who would have thought?
I know Linux isn't RH. I stopped participating in the local LUG when one of the members got a job w RH and started defending all the underhanded shit they do in the name of free software.
It's a good move. Parsing syslog sucks. And I don't care how awesome you think you are as a developer--you need to use the system logging facilities to make it easier on those of us who adminster systems.
At the very least a unified format similar to Microsoft's format would be nice.
ID / DATE-Time / Severity / BLOB OF TEXT
I beg to differ. Syslog is easy to parse, and the problems you're mentioning have nothing to do with either the protocol, nor the actual syslogd applicaiton. Microsoft's logging is horrendous, bloated, and extremely difficult to parse without external tools. Syslog has none of those issues.
Keep in mind, redhat is nothing like M$ in issues. They are a great Linux organization and FOSS friendly in most things. Expect what is done will be the new standard but also expect that you could pop in your own rpm for syslogd and run as you have in the past till you are ready to move forward.
> because its all tied together with a brilliant OS that people want to use
Constantly kidding yourself will not make that any more true.
People buy Macs because they are clueless idiots with no taste.
Although even those people are a very small minority.
A Pirate and a Puritan look the same on a balance sheet.
Nope.
Parts are parts and both machines have the same spare parts.
If anything, the HP will last longer due to better heating and cooling design and having those features more accessable to the end user. The desire to "be pretty" or "avoid geekiness" is not an advantage.
Although I could see were certain types of people could be confused.
A Pirate and a Puritan look the same on a balance sheet.
Don't ask any question you aren't willing to hear an honest response to.
If you do get an honest response but one that you don't want, then you really have no one to complain to but yourself.
If the truth is too painful, then perhaps you should try to avoid it rather than courting it.
A Pirate and a Puritan look the same on a balance sheet.
Of all of my "Mac Like" devices, my Apple ones have been the least reliable. I have had one Mini completely fail on me in contrast to several similar ION machines that are still chugging along nicely. Out of 2 other Minis of the older design, one of them has an internal component failure.
So out of 3 Macs, only 1 is still completely in one piece after just a few years.
I have a old slow Compaq that is still useful because I can easily upgrade it including the GPU. My older Minis are doorstops by comparison.
You need to stop kidding yourself. PCs aren't nearly that unreliable.
A Pirate and a Puritan look the same on a balance sheet.
The issue is based on what you need in different scenarios and to meet that I can't see anything wrong in doing both writing to syslog and a database.
Why do both? In larger systems the amount of data is difficult to cross reference and analyse as files due to the amount of sources, size of data, tools to visualise it all, etc. Writing syslog data to centralised syslog services that do use database backends to centralise logs and query/report against them are a key tool in these scenarios. Its one of a number of interfaces you have to analyse what is taking place on your systems.
However, I'd rather use the simplest method of getting log information out of a system if I'm going to use it for debugging an odd situation. There are situations where the overhead of writing to a database or a write remote data might fail and cause no debug information to be written. I'd rather a simple logging system locally.
Syslog isn't the be-all end-all, but it does provide system logging to review events that occurred on the system. Having a flat ascii file means you can use vi or emacs or any other file viewer you like to see what happened. Binary means you need someones proprietary reader to read system log files (meaning you need their thing on your system, you can't just remotely get into the system and read files, you have to have their thing running on the system in order to read files, or extract the files to a working system and read them with their reader somewhere else. Thats what makes this bad. As for RedHat, they have done a lot of things 'their own way', which is why I stopped using their stuff when I couldn't build stock kernels with their stuff. "Oh, you need *OUR* version of the kernel, with some stuff that diverges wildly from the norm, oh, and we broke some functionality...". And so I don't use their stuff anymore. Apple is basically a proprietary version of Unix, and Android is a proprietary version of Linux, likewise RedHat. They are a long way from Debian. Ubuntu is a different set of (still GPL) choices for Debian. At some point RedHat will be binary-incompatible with all other LInux. Its almost that way now.
when its not supported by the tools, lets have a /var/log/syslog and a /var/log/syslog.fine-timestamps. New tools use the new file.
How do you log errors with the new binary syslog?
"For I desired mercy, and not sacrifice" -- God
Though, being fair - I could see a way that it could be implemented while causing the least amount of distress. Also, you may be able to just turn it off, or have it pipe everything out into a nice text based log with FUCKING USEFUL ERROR MESSAGES. I don't run Windows because I can't fix it when it breaks (constantly) mostly due to the fact that you do not get useful error messages like you do in a Unix-like OS.
I could see the proposed system being useful in a large server setup where you care about logging I/O.
This has been a concern of mine. I have this happen frequently. I take solace in the fact that on the systems I use /var/log can only be read by root.
"For I desired mercy, and not sacrifice" -- God
Somebody still uses RedHat?
What's to stop you from removing the system logger you don't like and installing whatever you want?? See: http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=9
if it ain't broke, don't fix it
I didn't see anyone else mention this, but on windows and AIX, one of the reasons for using a binary log format is internationalization. Log messages are little more than application/facility id, log id, and parameters. The when the user displays the message the ids are looked up in a localization table and formatted according to the attached parameters.
A lot of people are complaining about changes that will destabilize their system - such as log analysis programs that would need to be changed to use the new system. A lot of people are complaining about how they would have to do something different - can't just /sbin/grep. A lot of people are making assumptions about deficiencies in a hypothetical system. There's a lot that is unknown about doing things in this different way, and its outside the experience and skills of many of the posters. And then, a lot of the messages are just bitching about how they hate one Linux distro or another (for inflicting a new GNOME on them, for example). Some good questions, also a lot of ignorant FUD.
Seems to me that those are all separate issues, which need to be addressed by more concrete, information. Above all this is the big question of how experimental the distro ought to be, for the customers it is intended to serve.
UNIX and Linux are already technologically about 40 years out of date, compared to previous (but not well known/popular) systems. Progress on operating systems is so slow it makes me want to kill myself when I think about.
Go for it, if it is well done it will be great, if it is badly done, then it wont work out.
Or we could all go back to Xfree86 and ext2.
Some of the old unix/linux fogies just need to be ignored. They hate change.
I've had an Averatec last 5 years. Ok the battery was shot at the end, but the rest was OK. Besides, what are the specs on a 5 year old Macbook?
The current init system is "simple and elegant"? The one with lots of shell scripts in /etc/init.d, and symbolic links in /etc/rc[0-7S].d to those links? Symbolic links with names like "K09dm" or "S51cups", where the first 3 characters are highly significant?
I disagree.
(Moreover, both upstart and systemd are significantly faster than the current system.)
Well, we have enough examples of wonderful pieces of programming art and creativity of Mr. Poettering. I wander if all of RedHat's top-programmers are genius?
Lennart's systemd is broken by design and pain in the ass, his PulseAudio is pain in the ass for each man who works with sound on Linux, his lidaemon is constantly unfinished and semi-working masterpiece...
This guy is just a soul of destruction. He breaks everything he touches.
Hey, RadHat management guys!!! Do you ever read what people think about your programmers?
PS. Yes, I remember that BSD is irrelevant, according to Mr. Poeterring. That's why I'm going to throw away all RedHat's stuff from my computers at home and at work until it's too late and replace it by FreeBSD and OpenBSD. Hope, BSD community is sane enough to keep guys like this one as far as possible.
This isn't about having EVERYTING in ASCII text.
This is about having logs in binary.
Nobody is saying we have to have everything in ASCII.
Except you.
Given it's mostly text anyway, how much space would be saved???
Date/time would be about the only thing reliably smaller in binary than ASCII (though using 32bit numbers means year 2038 becomes even scarier), but most numbers are in the one-two digit range and everything else is text.
Unless you're going to "normalise" the data or move some of it out or away in the log (e.g. have a lookup of applications in the log and assign a number to each log), but that means that you have to parse the entire file (correctly) to get anything out.
So, really, how much efficiency would be gained by making the log file binary?
If you have to ask "if", then you shouldn't burn your bridges.
What if, as seems likely, it's worse?
Charlie Fuggered.
That's what.
logs are NOT HUGE. If your log needs to be megabytes in size, UR DOIN IT WRONG.
Now, RH could have syslog talk to another daemon that munges a lot of logs from a huge cluster of machines in a business environment and uses the binary format to produce a central archive of logs on the system that can be more easily munged into a sysadmin report.
But leave the logs on the machines ascii. All you need is cat and you can see what happened just before you needed to look at the log.
A possible flag: Are red hat file changes related to a proprietary interest in the failure or capture of Linux?
No, Windows suck as much as people think (if not more).
There's a unit of measure for suckiness:
3.3) Just HOW MUCH does this system suck?
The ASR standard unit of suckiness is the Lovelace (Ll).
This is defined as: One Lovelace is the amount of force (measured in dynes)
it takes to draw a round ball weighing e Troy Ounces down a tube it fits
exactly (in air) at a speed of pi attoparsecs/microfortnight.
Like Farads, this is a rather large measurement. Thus, Plan 9 sucks a few
mLl, for instance, while your average Microsoft product achieves many Ll.
Who is John Galt?
Microsoft loves to put logs in binary format... Please please please... let's not make their same mistakes!!!! Keep it in ASCII text...
because its all tied together with a brilliant OS that people want to use.
So, it's only "brilliant" not at least "super-amazing-brilliant-fabulous-wonderful"? Your praise is *far* too faint, and you will be expelled from the Apple fanboi club forthwith. You are required to destroy all your Apple hardware and report to Redmond for re-programming.
Like how Red Hat started using indecipherable blobs to obscure their contributions to the kernel, I feel like they are just trying to do more and more to differentiate their operating system from linux distros in general while still staying within the framework of freely distributable software. It's not like this move changes much but it does add another layer of uniqueness/expertise that they can sell to their Enterprise clients. An Ubuntu admin is not necessarily going to be able to even read the logs in RHEL and that provides the opportunity for a sale or service charge at some point.
if your life is such a big joke then why should I care?