Mapping/Understanding System Complexity?

thesandbender asks: "I've recently inherited a project to 'simplify' the application environment for a company that has 1600+ service offerings (many of these are product 'foobar' that has options (like 'Alpha', 'Bravo', 'Charlie', and so forth) available. I am trying to map out the applications' dependencies from a technological and a business standpoint. I would like to designate a group of applications as depending on concepts, technologies (like SAN, DB2 and AIX), specific customers (like 'Bravo' and 'Charlie') and legacy applications. Basically, I want to define any number of arbitrary dependencies and then be able to map them out in a graphical format. With those maps I can show the business oriented staff how removing one application will affect other applications, and I can show the technically oriented staff how removing one system will affect other systems or applications. Has anyone in the Slashdot community run across such a tool? If you haven't, have you run across the need for such a tool? What would you want from it so that I can fashion a usable tool that addresses everyone's needs and not just my own?" "The most appropriate tool-sets I've found to date are 'mind mapping' or 'concept mapping' tools. All of the tools I've found so far only allow me to create any number of ideas or concepts and don't allow for arbitrary, searchable and/or mappable attributes (e.g. Application 'foo' maps to attributes 'SAN', 'Java', 'Solaris' and 'Buy-Side') that would allow me to create hard and soft groupings that were based on defined attributes (e.g. I could ask for a cloud of all objects that share a specific technical attribute, and another cloud of objects that share a specific business attribute)."

Comments

Although not designed for that problem

[jd]

I'd use a database modelling tool, like ERWin or Dezign for Databases. (Yes, they're both commercial - I've not seen a single good entity-relationship modelling tool that is also Open Source, although it's an obvious tool to write.)

These tools map attributes in records to other attributes in other records. They're designed to then turn these maps into SQL code, but that part isn't important here. What is important is that you can create a full relationship mapping between entities. If you then treat the direction of the relationship as showing the dependency, you can map all the dependencies in the system.

Managers like diagrams to be of a format that are familiar to them, so anything that is "better" from a technical standpoint but "less familiar" to managers from an experience standpoint is, in fact, not as good of a solution.

Re:degfbdsgf

[donaggie03]

What is wrong with a guy doing a little research before implementing an assigned task? I assure you, conversing with colleagues about an issue, whether it is in person or online, and whether you actually know the PERSON or just thier slashdot handle, you are still conversing with colleagues. It sounds to me like you skipped the classes that stressed "teamwork" in your undergraduate curriculum.

Graphviz

[danpat]

You can probably draw the picture you want with GraphViz, found here http://www.graphviz.org/

To use it, you create a text file that defines all your dependencies, it'll look something like this:

    digraph thingies {
        "app1" -> "SAN";
        "app1" -> "Java";
        "client1" -> "app1"; ...
    }

You can then go on to group things together so they show up in meaningful locations on the diagram,
associate pictures with certain nodes, put labels on things, make things in colour, etc.

GraphViz takes care of the laying out parts (where best to put nodes and edges). Sometimes it takes
a while to define everything in format that gets draw neatly, the results are often impressive and
very useful.

On coming to a new job, I've used it to draw all the dependencies between a collection of a couple
of hundred SQL stored procedures in our database. The locals were horrified to have what they all
knew in their gut depicted to them on 35 A3 sheets of paper on a wall :-) It was quite useful for
identifying things recursive calls ("Oh, *thats* why that proc sometimes never ends...").

Simple

[jlarocco]

When I get an assignment like this, I try to take a proactive stance. First, I add the project to my action item list. Then I formulate a list of stakeholders in the project. Then I call a meeting to open a dialog between the stakeholders and myself. After drilling down and making sure we're all on the same page, I draft a scope document. When I'm satisfied with the scope document, I hold a sidebar meeting to touch base with the shareholders and verify the document meets their requirements.

Usually by that time the project gets assigned to someone else.

Re:"Complexity kills"

[QuantumFTL]

The problem with the concept of complexity based on program length, is that program length is highly dependent on the system of encoding used to represent the program. Short programs in functional languages can be quite long in imperative languages, and lamdba calculus functions for even simple things are sometimes so long as to be impossible to represent in a published paper.

I've yet to see any "cannnonical" representation that can be used for this purpose. "Kolmogorov complexity" is not useful for these things for that reason. A more interesting metric, often used in software engineering, is cyclomatic complexity.

Re: "Complexity kills"

[Black Parrot]

> Typically, the Anonymous Coward's contentless sarcasm betrays his shallow grasp of reality. The relevance is clear: When you design your service suite and do not minimize complexity, you aren't just asking for trouble, you are, by definition, producing a low quality suite. You can, in fact, produce a compression of a natural language knowledge base without even using a compression program and have that be an important human accomplishment. Epistemology is virtually defined by such advances. So the fact that the problem is computationally hard is neither here nor there to first order. The important thing is quality of knowledge.

Ignoring the fact that your post was merely a bit of self-aggrandizement unrelated to to the Ask Slashdot question, you're chasing a will-o-the-wisp. There is no universal compression algorithm.

It should be immediately obvious that, when using the same symbol set for plaintext strings and their compressed form, any compression algorithm that makes some strings shorter must make some other strings longer.[*]

Thus the design goal of any useful compression algorithm is to bias it toward the expected properties of the input strings. The algorithm that compresses English text the best probably doesn't compress Latin text the best. The algorithm that compresses Slashdot best probably doesn't compress the New American Standard Bible best. The algorithm that compresses Slashdot stories on astronomy best probably doesn't compress Slashdot stories on biotech best. The algorithm that compresses your post best probably doesn't compress my post best.

What do you expect to accomplish with a prize for best compression of some pre-specified corpus, other than finding out who can do the best job of tuning their algorithm to that corpus?

You certainly won't learn anything about artificial intelligence. Hor help thesandbender with his IT question.

[*] You can get ahead by using different symbol sets for the strings and their compressions, but if you are going to process them with a binary computer and/or store them on binary media, you're stuck with {0,1} under the hood, regardless of what superficial symbol sets you specify.