Slashdot Mirror

If you really think you're not going to seal all the cracks, or that you create new ones as you rebuild your electronic foundation, you need to track what goes on inside the house at all times.

The best way to do this is to log all significant events in your infrastructure:

• network connections
• web server hits
• DB queries
• app server events
• machine syslogs
• ...

Without knowledge of your history you can't see new trends or look back and see how often in the past newly discovered exploits by external attackers and internal were used. The company I work for (Addamark) discusses the log-everything approach to security. It's a tough problem because of the scale of info required. Sorry for the shameless plug but this is the problem we address, and do so rather well at several real-world companies.

Cringely put out an article (Changing the Game: How to Save the World by Taking Back Control of Our Data) a week or so back emphasizing security through recording all activity in any given IT infrastructure. Cryptographic techniques may be great, but social engineering, cracked buffer overflows, and short-sighted or stupid actions can always leave some crucial data exposed.

Rather than throwing your hands up when you've found you've left data exposed, or you've discovered some insider has been poking around documents they shouldn't be looking at, you should be able to track down all access to all information at all layers of your infrastructure. You hopefully can uncover traces of specific incidents, find any other similar unnoticed events that are now part of history, and find the culprits.

So logging and log analysis are key to securing any site. You need to log:

• web servers
• DB access
• app server use
• custom applications
• machine login sessions
• network events
• key card access to buildings
• maybe even disk I/O info
• ... and many others ...

... and you need to do it in a way where you can correlate information from all these disparate sources to uncover patterns of abuse. Cringely mentions that Addamark (he calls them the next "Oracle") is the first company with a viable solution for storing and analyzing the massive logs involved. I've looked at their site, does anybody know anything about this product? Sounds very useful.

Logging public wireless networks at a level of detail required to catch infractors would be a huge undertaking. Most products that could do it would cost a lot and would add a huge burden to those administering those networks.

I've heard good things about Addamark, they claim to have a log capture/query tool at a good price/performance point. Does anyone have experience with large log data volumes and this tool?

• "In order for an IDS to be effective, or in some high-bandwidth cases, even usable, detailed network and business context must be applied to the IDS.

Snort and the other intrusion detection systems perform to varying degrees at monitoring corporate resources resources and alerting personnel when something is amiss, according to the rulesets they've been given. The article assumes the rulesets are known in advance: your work is to take those rulesets and implement them in Snort or your favorite IDS.

The real world isn't so simple. IT personnel can only guess at all the possible security problems with the network equipment, hardware, server software, clients, external network connections, malicious hackers and information thiefs out there -- as well those rare dishonest insiders. A more effective security implementation includes plenty of logging, and subsequent log analysis.

Logs are easy to generate for all varieties of hardware and software. Collecting and centralizing log data lets you:

• track the history of all aspects of IT infrastructure
• analyze patterns of past resource use as personnel understand more about potential threats (have such exploits occurred in the past? what additions to current real-time IDS rulesets will address such exploits?)
• analyze past resource use to see whether newly discovered, real exploits have been used in the past (the organization can take appropriate measures to uncover abuse two months ago, a year ago; what data was compromised then?)

Having the history lets an organization more effectively implement the "detailed network and business context" within the real-time IDS solutions.

Of course, the real problem is the $2 million for the Oracle DB to manage all that log data. And querying all that history is a bear. And the DBAs, the software developers, etc. to manage that log history. I've heard that addamark's log management system (LMS) is a good alternative. Someone told me their product replaced a DB2 cluster at one organization after a two-hour DB2 query took three minutes on an Addamark cluster. The cost savings, storage capacity, and log compression were phenomenal too.

Are there other log centralizing solutions out there you've heard of? Addamark seems to work because it's not a full-fledged traditional DB, but optimized for this log management problem -- can a traditional DB keep pace?

Firewalls are great when you can trust all your insiders. That's rarely the case. Real-time intrusion detection systems also help out, but fail when:

• attacks are diffuse, slow and patient, and seemingly random -- there's no way a real-time detection system will connect the activity
• insiders do the job -- they're not "intruding"

To really address security of corporate data you need to:

• log all activity on all servers and hardware surrounding your vital data
• store that log data in a centralized location
• periodically analyze that data for abnormal patterns of activity within or across logged systems
• some analysis will be boilerplate, other analysis will be highly customized to a specific site's data architecture

This log analysis approach complements the others, and will catch more insidious, long term, and more damaging violations of critical data. Most corporations have the firewall angle covered well, but can't address social engineering or misbehaving insiders.

Of course, the big problem here is storing all that log data. Security analysis companies have been around but either can't perform analysis at the detail required, or charge too much (that log data is huge and Oracle isn't cheap).

Addamark Technologies has a security event logging and analysis tool that seems to address this problem though. They sell a product that uses a cluster of cheap Linux PCs to store all that data, and a SQL/Perl query interface (for those that want to query data directly without web-UI tools), some good web-UI tools. Data loading performance and query performance is out of this world. They've got a great customer list, too.

Data collection is useful only if you can analyze the data. There's no way with millions/billions of records stored this product will manage to expose that information in a useful way.

Unless, that is, they couple it with a high-speed database such as the addamark log management system, a high-compression Linux/SQL/Perl query engine.

The author's analyzing huge volumes of network logs -- he needs a good analysis tool after capturing all that data on SCSI or IDE.

You should take a look at the addamark technologies LMS (log management system) product. It's a distributed, clustered database system that will load huge amounts of log data and run SQL/Perl queries against it much faster than regular commercial databases or other specialized tools. Check it out.

There are tools coming out that will let you store and analyze huge amounts of log data. Check out addamark technologies. A high-compression distributed log storage system with a SQL/Perl interface, it will make storing/analyzing log data cost-effective.