Slashdot Mirror

No actually they are. Systemd has an export format and a JSON library you can attach to that produces a version of the log designed for combining. Systemd works far better with messaging clients than syslogd because it was written after messaging and networks.

I wouldn't say converting all that metadata to text is great for the network. You have to parse all that JSON on the receiver end.

Your specific example is also wrong. You just setup a Splunk forwarder: http://answers.splunk.com/answ...

I guess you didn't actually read that article, which is about configuring a systemd unit file to start/stop Splunk.

SysD's binary logs have another, serious flaw: they are not designed to be sent over a network.

No actually they are. Systemd has an export format and a JSON library you can attach to that produces a version of the log designed for combining. Systemd works far better with messaging clients than syslogd because it was written after messaging and networks.

Your specific example is also wrong. You just setup a Splunk forwarder: http://answers.splunk.com/answ...

I can see your point but in the cases where I have had to parse binary logs that come to mind i.e utmp files and BSM audit logs, it was significantly more annoying than parsing something like syslog with grep/awk/sed/cut/expr etc etc.

It occurs to me that the problem you are trying to address is only a problem because maybe you havent found the right tools and maybe havent split your logs up into logical files rather than just using syslog.

The tool you want to parse your logs is so good it seems like magic. It is an unbelievable tool. It indexes log files, extracts reports, draws graphs, alerts and keeps your coffee warm. It is http://www.splunk.com/ you can use it for free if you dont index too much information.

Like so many enterprise tools, including all monitoring software, it cant read binary logs.

One word: Splunk

That kind of thing is extremely common. How about this job posting for a tech lead / project manager position at Splunk. "Our development philosophy is similar to that of Joel Spolsky – if you know who that is, and you hold the same beliefs, we’d like to hear from you!" they say. What do you see when you get there? Yep, all of their developers jammed elbow to elbow in a single huge open office that also contains the kitchen/lunch-area, video games, and ping pong table. Oh, but they cater lunch!

I use pfSense too.. great firewall btw.

I thought about sending my logs to a server but haven't had the time to research it. I'm trying to setup a linux box to capture the logs and run it through a web-server to generate the reports I need on the fly. Any ideas? And yes I googled... too many old information out there.

Splunk

Free for home use. I use it and it's nice. It's a CPU usage whore though.

Have a look at this.

J.K.

I know what you mean - but I present to you the balancer that makes the flash ad ok: the best software promotion song ever :-P the software's a bit pricey, though.

You should have Splunked it.

I'd hit it!

I think you are confusing plain old "Splunk" with "Splunk Base" - they are two different things. Both offered by the same people, but different none the less.

http://www.splunk.com/

Yes, I found http://www.splunk.com/ and I even found http://www.splunk.com/base . But when I typed "crashdump" into the search box (and, yes, there is one of them in my log right now even as we speak), I got:

We did not find any pages on Splunk.com that match crashdump

Not a promising beginning.

I can't figure out whether I'm supposed to create an account or not, I can't figure out the site is really supposed to be up and running...

Yes, I found http://www.splunk.com/ and I even found http://www.splunk.com/base . But when I typed "crashdump" into the search box (and, yes, there is one of them in my log right now even as we speak), I got:

We did not find any pages on Splunk.com that match crashdump

Not a promising beginning.

I can't figure out whether I'm supposed to create an account or not, I can't figure out the site is really supposed to be up and running...

Someone should tell those people that they have splunk on their shirts.

Oddly enough, the auto-generated word for this post was "condom".

Insiders can be real threats, the BIGGEST threats. An insider can steal much more than a hacker ever can. And many insiders think they can get away with it. Just look at the porn-billing iBill incident made public last week.

The best policy is to log everything that happens in an enterprise, to a level required to reconstruct past bad behavior. You can't keep your insiders away from information they need to do their jobs. Trust, but also verify! There are products out there like Sensage (http://www.sensage.com/ ) that can collect, centralize, and make available years of log data for an IT organization. While this might not prevent the theft in the first place, a company can crack down on and prosecute current/former misbehaving insiders. Sensage will do very well, as will many other companies in this space (including recent Slashdot heavy banner-advertiser Splunk (http://www.splunk.com/ ) ).

I look forward to seeing how well these products do. It's time one of them went public so we can gauge interest.

I found this on comp.os.linux.announce. no word on IIS files... ...a free Linux/Solaris server that lets you search all your log files... blah ...search and troubleshoot all of their log files...blah... Apache, Jboss... ... http://www.splunk.com?ac=kilroy

Found this on comp.os.linux.announce: ...a free Linux/Solaris server that lets you search all your log files... search and troubleshoot all of their log files...blah...blah...Apache, Jboss... http://www.splunk.com?ac=kilroy

I would agree. Splunk is a cool ajax app (props to SirNick) that's really good at sucking my /var/logs.

//booyakasha

//booyakasha
//Alf Gored

If you haven't seen it yet the splunk log analysis engine is a really neat use of ajax. Theres a live demo up on the site so you can check it out without having to go and install it.

( Fulldisclosure : I do work for splunk but I still think that the gui and engine rock ! )

http://www.splunk.com?ac=secret

OK, not really. It's a cool linux search engine
that is like google for your IT log data. My
mother's son works there.

http://www.splunk.com?ac=secret

OK, not really. It's a cool linux search engine
that is like google for your IT log data. My
mother's son works there.

Splunk has a new search engine which allows you to examine your log files for either security or troubleshooting related issues. The server (free) uses GoogleSuggest-like type-ahead and dynamic graphs to allow you (a sysadmin) to view all the different logs that are in your system (or datacenter) occuring in real time. It uses Ajax and a lot of javascript to make the experience of flipping through log files and finding specific events very fluid. While it won't prevent security issues, it will certainly help make them more detectable.

Splunk has a new search engine which allows you to examine your log files for either security or troubleshooting related issues. The server (free) uses GoogleSuggest-like type-ahead and dynamic graphs to allow you (a sysadmin) to view all the different logs that are in your system (or datacenter) occuring in real time. It uses Ajax and a lot of javascript to make the experience of flipping through log files and finding specific events very fluid. While it won't prevent security issues, it will certainly help make them more detectable.