Slashdot Mirror
Visualizing Complex Data Sets?
markmcb writes "A year ago my company began using SAP as its ERP system, and there is still a great deal of focus on cleaning up the 'master data' that ultimately drives everything the system does. The issue we face is that the master data set is gigantic and not easy to wrap one's mind around. As powerful as SAP is, I find it does little to aid with useful visualization of data. I recently employed a custom solution using Ruby and Graphviz to help build graphs of master data flow from manual extracts, but I'm wondering what other people are doing to get similar results. Have you found good out-of-the-box solutions in things like data warehouses, or is this just one of those situations where customization has to fill a gap?"
Abstract: We propose a method for characterizing large complex networks by introducing a new matrix structure, unique for a given network, which encodes structural information; provides useful visualization, even for very large networks; and allows for rigorous statistical comparison between networks. Dynamic processes such as percolation can be visualized using animations. Applications to graph theory are discussed, as are generalizations to weighted networks, real-world network similarity testing, and applicability to the graph isomorphism problem.
PtolemyPlot and Java.
http://visualcomplexity.com
Have fun!
If you have access to a plotter, Graphviz gives you a great deal of flexibility with regards to how big these images can physically be. Maybe you could consider posing them up on the wall and having a roundtable session at your office.
This game will waste your life. Don't clicky!
How are you supposed to handle the data if you do not understand it? Sure, there can be too much to see/think about at one time, but if you don't understand it, how can you visualize it usefully?
I am asking because I have a problem: Where I work, I understand the data and I make efforts to visualize it for others. The trouble starts when they don't understand the data and it's sources and limitations, so what they see in my visualization is all they know of it, and they make assumptions about it. I've even had people worry that the network is down because there were holes in the collected data which then showed up in the visualizations.
If anyone has some good URLs for such thinking, I'd be grateful.
I simply do not understand how you can visualize data for people if you yourself do not understand it.
Support NYCountryLawyer RIAA vs People
There was a thread about the R language a couple of weeks ago. Look it up and read it....
The infovis community has been dealing with these subjects for years. There's many different visualisation techniques around. Here's a list of the past conferences and the papers:
http://conferences.computer.org/Infovis/
Plenty of good products out there, but the one that I like most is from Tableau Software (http://www.tableausoftware.com/).
Life is complete only for brief intervals in between toys or projects -- John Dalton
I work in biology, and we use Spotfire DecisionSite to visualize and analyze a lot of our massive genetic data. It's a very powerful program that I barely know how to use. It seems to have packages able to analyze pretty much anything you want, and you can even write your own scripts to help things along.
Wouldn't any everyday cube browser along with any tool to detect base dimentions in a datawarehouse schema do the trick? You may have to add a few custom dimentions on your own depending on how shitty the master data is (I don't think that can be helped, no matter the solution, if a dimention is "these two fields multiplied together times a magic number appended to the value of another table", you need to know, no tool will guess), but aside that?
Thats usually what I do anyway. I dump my data in a datawarehouse, use whatever built in wizard can auto-generate dimensions, then play with them in a cube browser. Works for even pretty archaic home-made multi-thousand-tables-without-normalization ERP systems I had to work with in the past anyhow.
Your ERP isn't supposed to directly analyze the data. You're supposed to use a Business Intelligence software package for that. This being SAP, I believe they'll try to sell you Hyperion.
Just take the first 65k rows and dump them into excel and create a pivot table.
Into a matrix screensaver.
I record my sleeptalking
I have no idea how I stumbled across this, but it looks very pretty...
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
Large, highly complex data sets are best described on the back of four cocktail napkins or on a fixed white board in a shared conference room. ~
Invenio via vel creo
Tufte's ideas are good for presenting simple information. He gets many things right (eg if the visualisation doesn't work in black and white, adding colour won't fix it). However, many in the infovis community are outright sceptical, if not dismissive of his ideas for analysing high dimensional datasets.
Where his ideas really work is once you have "the answer" that you want to present to someone else. However, the basic exploration of the data to find interesting keypoints, is not what he specialises in. There's whole communities devoted to techniques for datamining and presentation, principly infovis/Visual Analytics.
Life is complete only for brief intervals in between toys or projects -- John Dalton
Tableau Desktop is an interactive analysis and visualization product that connects to relational and cube data sources to help people see and understand their data. There was a webinar (slides - PDF) back in November 2008 covering Blastrac Global's success in using Tableau with their ERP system.
Disclaimer: I work at Tableau Software, so I encourage you to see for yourself with a free trial: http://www.tableausoftware.com/products/tour
I had a similar situation to yours recently, except I was trying to detangle a horridly complex product substitution graph for a logistics company.
I used a bunch of Perl to crunch the raw databases into various abstract graph structures, but instead of graphviz or something created by/for developers, I found that the best software for graph visualisation is the stuff that the genetics and bio people use.
The standout for me was a program called Cytoscape which can import enormous graph datasets and then gives you literally dozens of different automated layout algorithms to play with (most of which I'd never heard of, but it's easy to just go through them one at a time till something works)
It's got lots of plugins for talking to genetics databases and such, but if you ignore all that and use Perl/Ruby/whatever for the data production part of the problem, it's a great way to visualise it.
I'm sure it's my mathematics background, but when I saw the headline I assumed the author would be discussing something involving the square root of negative one, to which my response was, "Silly author, you can't visualize four dimensions. (Sober.)"
You have a mathematical background and can not visualize four dimensions? Here is how you do it: Just visualize the problem in n dimensions and then set n=4.
Looks like you work for another company that tried to reimplement their old system word for word and step by step in SAP.
And customization has a specific meaning in SAP that doesn't involve any coding. It appears you don't know that, which doesn't improve your credibility.
If you want that, don't use a package. Now I'm not saying SAP is perfect - there are some companies whose process certainly don't fit SAP. The solution in that case is to write your own from scratch.
Make vs buy is an old dilemma, but doing both isn't really a good solution.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Since you've already bought licences for SAP ERP, you could get a bargain on the Master Data Mgt component. It also offers support to control the Master Data harmonisation process, which you probably need if you have such a large amount of data.
Their priorities are different. I come from three generations of German VW assembly line workers (Wolfsburg plant). I grew up in the US. Last time I saw my now-retired uncle, he asked me about the (then) new VW Bug and why the Americans kept putting flowers in the dashboard-mounted gun rack. Explanation was pointless.