Slashdot Mirror
Applied Security Visualization
rsiles writes "When security professionals are dealing with huge amounts of information (and who isn't nowadays?), correlation and filtering is not the easiest path (and sometimes enough) to discern what is going on. The in-depth analysis of security data and logs is a time-consuming exercise, and security visualization (SecViz) extensively helps to focus on the relevant data and reduces the amount of work required to reach to the same conclusions. It is mandatory to add the tools and techniques associated to SecViz to your arsenal, as they are basically taking advantage of the capabilities we have as humans to visualize (and at the same time analyze) data. A clear example is the insider threat and related incidents, where tons of data sources are available. The best sentence (unfortunately it is not an image ;) that describes SecViz comes from the author: 'A picture is worth a thousand log entries.'" Read on for the rest of rsiles's review.
Applied Security Visualization
author
Raffael Marty
pages
552
publisher
Addison-Wesley Professional
rating
9/10
reviewer
rsiles
ISBN
978-0321510105
summary
Definitely Security Visualization is one of the most relevant present and future topics in the security field, and this book is simply THE reference.
This is a great book that joins two separate worlds, visualization and information security (infosec). The first chapter is an excellent introduction to the human perception system, its basic principles, and how we analyze, discern, and assimilate information. It is an eye opener for those new to the field. Chapter two is similar from an infosec perspective, and summarizes the main challenges and data sources, such as packet captures, traffic flows, and firewall, IDS/IPS, system, and application logs. The third chapter details different graph properties and chart types, including some open-source and online tools for chart and color selection. Although we (infosec pros) are familiarized with link graphs to represent relationships between botnet members or hosts, the book provides a whole set of charts for different purposes; one of the most useful types, and we are not very used too it in the security field, is treemaps. The chapter includes a really useful table to select the right graph based on the purpose of the analysis and the data available.
Then, the previous chapters are smoothly mixed together through a reference methodology that defines what is the problem to solve, and the process to manipulate the available data and generate a (or set of) graph(s) that allow gathering relevant conclusions and answers. The methodology is complemented with an introduction to the standard Unix-based text processing tools (grep, awk, Perl, etc). This methodology is later on applied, with a strong hands-on and how-to spirit, to an extensive set of common security use-cases, such as the perimeter threat, compliance, and the insider threat.
The perimeter chapter offers a deep insight into common attack scenarios, such as worms, DoS or anomaly detection, and operational tasks, like firewall log and ruleset analysis, IDS tuning, or vulnerability assessments. I could never forget how useful were SecViz techniques for anomaly detection on a huge DNS-related incident I was involved about 5 years ago. Thanks to the performance and statistical graphs we had available at that time, we were able to easily identify and solve a very complex and critical security incident.
When I saw this chapter included a wireless section I got really excited due to personal interest. However, I was disappointed as it was just a couple of pages. I think it could be extended to gather a whole set of useful information about complex wireless attacks and client and access points relationships, just by inspecting the different 802.11 management, control, and data frames, and even radio-frequency signals (from a spectrum analyzer). SecViz opens the door to a whole new wireless research area!
The compliance chapter offers a whole methodology to check and manage regulations, control frameworks, auditing, and risk monitoring and management from a visual perspective.
The same applies to the insider threat chapter, as it provides an impressive framework, not only visualization-based, to deal with malicious insiders. It is based on setting up scores for certain behaviors and activities (precursors), generating lists of suspicious candidates, and apply thresholds to accommodate exceptions. It also contains an extensive and directly applicable precursor list at the end to detect suspicious insider activities.
Finally, the book contains a whole chapter, full of references and comparison tables, of open-source and commercial visualization tools and libraries that allow the reader to select the appropriate tool for specific tasks and scenarios.
Although the book hands-on component is very significant, with lots of detailed examples of commands, scripts, and tool options to generate the different graphs, I would have liked to see a thorough usage of the how-to portions, as for some sections there are no specific details about how the graphs have been generated. The book layout makes it the perfect candidate to become a fully interactive technical book. I would suggest to add (for a 2nd edition ;)) practical sections to each chapter where the reader could reproduce all the steps discussed. The book CD is the perfect tool to provide the reader with all the (sanitized) data sets and logs used to generate the graphs, and even allow to include some challenges where the reader needs to analyze the data and answer some questions after generating the appropriate graphs.
To sum up, this book is a mandatory reference for anyone involved in the operational side of infosec, doing intrusion detection, incident handling, forensic analysis, etc, and it can be applied to both, historical analysis and real-time monitoring. Additionally, I found it useful too for auditing and pen-testing professionals, as it provides great tips to generate relevant and efficient graphs for the associated reports.
The accompanying DAVIX Live CD is an excellent resource to start applying the techniques covered throughout the book through open-source tools, SecViz is the Web portal to expand your knowledge on this topic, and AfterGlow is (one of) the most relevant SecViz open-source tools.
You can purchase Applied Security Visualization from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Then, the previous chapters are smoothly mixed together through a reference methodology that defines what is the problem to solve, and the process to manipulate the available data and generate a (or set of) graph(s) that allow gathering relevant conclusions and answers. The methodology is complemented with an introduction to the standard Unix-based text processing tools (grep, awk, Perl, etc). This methodology is later on applied, with a strong hands-on and how-to spirit, to an extensive set of common security use-cases, such as the perimeter threat, compliance, and the insider threat.
The perimeter chapter offers a deep insight into common attack scenarios, such as worms, DoS or anomaly detection, and operational tasks, like firewall log and ruleset analysis, IDS tuning, or vulnerability assessments. I could never forget how useful were SecViz techniques for anomaly detection on a huge DNS-related incident I was involved about 5 years ago. Thanks to the performance and statistical graphs we had available at that time, we were able to easily identify and solve a very complex and critical security incident.
When I saw this chapter included a wireless section I got really excited due to personal interest. However, I was disappointed as it was just a couple of pages. I think it could be extended to gather a whole set of useful information about complex wireless attacks and client and access points relationships, just by inspecting the different 802.11 management, control, and data frames, and even radio-frequency signals (from a spectrum analyzer). SecViz opens the door to a whole new wireless research area!
The compliance chapter offers a whole methodology to check and manage regulations, control frameworks, auditing, and risk monitoring and management from a visual perspective.
The same applies to the insider threat chapter, as it provides an impressive framework, not only visualization-based, to deal with malicious insiders. It is based on setting up scores for certain behaviors and activities (precursors), generating lists of suspicious candidates, and apply thresholds to accommodate exceptions. It also contains an extensive and directly applicable precursor list at the end to detect suspicious insider activities.
Finally, the book contains a whole chapter, full of references and comparison tables, of open-source and commercial visualization tools and libraries that allow the reader to select the appropriate tool for specific tasks and scenarios.
Although the book hands-on component is very significant, with lots of detailed examples of commands, scripts, and tool options to generate the different graphs, I would have liked to see a thorough usage of the how-to portions, as for some sections there are no specific details about how the graphs have been generated. The book layout makes it the perfect candidate to become a fully interactive technical book. I would suggest to add (for a 2nd edition ;)) practical sections to each chapter where the reader could reproduce all the steps discussed. The book CD is the perfect tool to provide the reader with all the (sanitized) data sets and logs used to generate the graphs, and even allow to include some challenges where the reader needs to analyze the data and answer some questions after generating the appropriate graphs.
To sum up, this book is a mandatory reference for anyone involved in the operational side of infosec, doing intrusion detection, incident handling, forensic analysis, etc, and it can be applied to both, historical analysis and real-time monitoring. Additionally, I found it useful too for auditing and pen-testing professionals, as it provides great tips to generate relevant and efficient graphs for the associated reports.
The accompanying DAVIX Live CD is an excellent resource to start applying the techniques covered throughout the book through open-source tools, SecViz is the Web portal to expand your knowledge on this topic, and AfterGlow is (one of) the most relevant SecViz open-source tools.
You can purchase Applied Security Visualization from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
a book that tells you how to make a flowchart.
the tools to populate the flowchart are kind of cool I suppose, but nothing i haven't figured out on my own anyways. oh well, if others find this book useful who am i to judge. but if you're in this industry i would hope you already know most if not all of what looks to be in this book.
viewing first post?
I think not. Unless it is a picture of those log entries.
Come on we've all seen independence day and Jeff looking at the 1s and 0s and then just spotting the pattern and the problem.
Jumped up security people with there fancy visualisation tools. 1s and 0s is where it is at, all you really need is a very very large green screen monitor and the force.
Personally I don't even use the monitor but instead lay hands on the ethernet cables and just squeeze out the bad packets.
An Eye for an Eye will make the whole world blind - Gandhi
I apologize for the shortness of digital temper, I just quit smoking.
May I suggest restarting? I did a double take on "mandatory" but if you flash into dreams of turning things into hamburger after that something is wrong.
he methodology is complemented with an introduction to the standard Unix-based text processing tools (grep, awk, Perl, etc). This methodology is later on applied, with a strong hands-on and how-to spirit, to an extensive set of common security use-cases, such as the perimeter threat, compliance, and the insider threat.
Do you really want someoen who doesn't know grep to be security admin? And is Perl correctly included in that list?
Oh honey look... How cute... an angry slashdotter!
What you do is tail all your logs to a console, sit Neo in front of said console, and have him detect changes in the patterns.
I once worked for a network security startup that had almost exactly this strategy.
They didn't do very well, I'm afraid.
668: Neighbour of the Beast
I prefer the Gibson aproach. It shows cool little green/blue translucent cubes that turn red when something odd happens to them. You then have a whack-a-mole kind of interface to kill those bad processes.
Did I see the word correlation in a summary and then not the "correlationisnotcausation" tag? I'm proud of you /.! So proud!
I did a double take on "mandatory" but if you flash into dreams of turning things into hamburger after that something is wrong.
At what point does the continual erosion of our rights, the continual battery which smashes the individual into a robot of compliance, how low must we sink, before we can revolt?
This is my sig.
A nice port-scanning secviz realtime animation was mentioned (I think here?) back in 2003. See this paper (images and animations are at the bottom) from DOE/LBNL/NERSC.
Space and Computers.
I have ordered one. I have doen quite a bit of SecViz myself, but a survey and reference would be most welcome.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I used to work at a firm called Q1 Labs. Their founders quite brilliantly mapped aggregate network statistical information into visual presentations. In effect, the computer did what it does best: aggregate information. However their product offloaded the detection of anomalies in that aggregate information onto the human, and in particular it presented data in a way that the human brain's visual centre could readily observe patterns and deviations from those patterns (but which patterns and deviations are quite difficult for computers to detect naturally/automatically).
It's kind of offtopic, but I thought that was cool enough to share. :)
I thought netcosm from NetQOS was pretty sweet... if only they'd push it out for end use.
See video here http://www.youtube.com/watch?v=dtC6ZM0_m8U
I have found graphs of log data to be somewhat useful, but my (admittedly limited) experience with visualization tools in security is that they give such a high level view, they are only applicable to certain kinds of activity--DoS is a good example. But at some point, the actual traffic must be analyzed to determine what the activity is. I was once peripherally involved in a "DDoS" that turned out to be Monday morning logins after weekend maintenance cleared DNS caches!
The more serious exploits are typically trying to be stealthy and usually don't show up in the data visualizations. The best analysis tool is access to log data in a format that allows it to be arranged in many ways, such as by timestamp, source or destination address, and also by sorting on multiple fields.
I prefer to spend the CPU cycles on the initial reporting code (IDS sensor, firewall, Email filters, etc.). And as a blatant plug, I have been working on an IDS that does this. It captures both halves of TCP sessions and correlates multiple detects from one or both halves before cutting a report. This allows for fine tuning of rules. The reports are stored in a DB which allows the UI to display many views of the data.
Later . . . Jim
The Realeyes IDS, check it out at:
http://realeyes.sourceforge.net
Did it earlier and better. See the article "The Use of Faces to Represent Points in K-Dimensional Space Graphically" or just skim the Wikipedia article
If you liked the Spinning Cube of Potential Doom and AfterGlow, you should check out NetGrok at http://www.cs.umd.edu/projects/netgrok/. It allows real-time analysis of a live packet capture and is released under the BSD license on Google Code. It was presented at the VizSec 2008 conference on security visualization. Disclaimer: I'm one of the project developers.
No need to sink lower, I already find you revolting.
I haven't read this book yet, but visualization tools ARE a significant part of pattern detection that we've mostly overlooked.
Much as we try to create smarter algorithms that can do feature extraction, clustering, etc., the best pattern-detection engine we have is still the human brain. There are very few systems that can detect patterns when we have NO idea what we're looking for; the brain comes pre-installed. Have you ever tried to do logfile analysis on a few thousand machines? Playing "management by exception" doesn't work at scale; even the rare errors show up a few times a second.
I saw a presentation a few weeks ago by Deb Roy, who's heading the Speechome project at MIT. He's set up a bunch of cameras recording continuous audio and video in his house, in an attempt to map the language development of his son. That's a LOT of data to sift through - some 90,000 hours. Way too much for standard audio scrubbing/speedup, which would be the equivalent of our grep-a-log-file.
So they've had to develop some incredible visualization techniques that let you view higher-level patterns across multiple "rich data" streams - things like frequent patterns of motion (there's baby playing with his toy car with Daddy), eye-gaze focal points (there's baby looking at the car before saying "KA"), etc. that just pop out at you as you view the full data stream. It's truly jaw-dropping stuff, and it's applicable to far more than speech.
Anyone here ever defragged a hard drive (yeah, I know, ext3/HFS/etc.)? Would you get a better feel for the operation if you saw a list of sector numbers that were being relocated, or the usual 2D colored-block graph?
Anyone ever seen TreeMaps for finding large files on your drive?
Anyone ever known when a process is about to crash because the patterns of UI hesitation and hard-drive head-movement sounds change as the core files get written out?
That's all that info-vis is. It's presenting data in a way that lets you use intuition and subconscious cues to find what you're looking for - even if you don't KNOW what you're looking for.
Here's Deb Roy showing how you turn motion patterns from multiple video cameras into a two-dimensional, printable chart:
Visualization generation
mandatory? I don't think that that word means what you think that it means.
Visualization is a natural companion to security metrics. But I'd stress that unless you have sensitive metrics in the first place, visualization is not going to help.
For an excellent, intellectually rigorous treatment see Andrew Jaquith, "Security Metrics", ISBN 0321349989
Parity: What to do when the weekend comes.
No need to sink lower, I already find you revolting
Yes we can!
This is my sig.
I don't work in IT, but I do work with very sensitive data that is high risk, so IT security is an important topic and I try to understand it as much as possible.
the reason I'm shocked is that I'd expect people to at least recognize that as a blindspot in their training before they ever graduate school. Our system is exclusively proprietary so I understand not everyone would need it, but it seems like it would make sense to know it since a good seventy percent of papers I see involve unix solutions. Granted, I might just be operating in a biased environment but I figured everyone would know at least the basics about linux.
and to ducomputergeek, I was infering that I don't see perl as being a unix based language... as far as I know it's platform independent and isn't strictly Unix based.
Oh honey look... How cute... an angry slashdotter!
Ever seen the program called baobab?
It makes a really nice multi-dimensional pie chart representing disk usage.
I'm sure something similar would be very useful for locating security problems such as addresses and subnets with unusual activity levels, etc.
or SPAN port. Just great: http://research.wand.net.nz/software/visualisation.php
"It doesn't cost enough, and it makes too much sense."
"...extensively helps to focus on the relevant data and reduces the amount of work required to reach to the same conclusions."
Unfortunately, most of my bruthuhs in the security biz DO 'reach to conclusions'. And the companies who hire them pay dearly.
A company I work with is on their second outsourced computer and network contract with a very incompetent company, which seems to be par for the course nowadays. They know just enough buzzwords to sucker the technologically illiterate masses and reap the rewards.