Mapping/Understanding System Complexity?

thesandbender asks: "I've recently inherited a project to 'simplify' the application environment for a company that has 1600+ service offerings (many of these are product 'foobar' that has options (like 'Alpha', 'Bravo', 'Charlie', and so forth) available. I am trying to map out the applications' dependencies from a technological and a business standpoint. I would like to designate a group of applications as depending on concepts, technologies (like SAN, DB2 and AIX), specific customers (like 'Bravo' and 'Charlie') and legacy applications. Basically, I want to define any number of arbitrary dependencies and then be able to map them out in a graphical format. With those maps I can show the business oriented staff how removing one application will affect other applications, and I can show the technically oriented staff how removing one system will affect other systems or applications. Has anyone in the Slashdot community run across such a tool? If you haven't, have you run across the need for such a tool? What would you want from it so that I can fashion a usable tool that addresses everyone's needs and not just my own?" "The most appropriate tool-sets I've found to date are 'mind mapping' or 'concept mapping' tools. All of the tools I've found so far only allow me to create any number of ideas or concepts and don't allow for arbitrary, searchable and/or mappable attributes (e.g. Application 'foo' maps to attributes 'SAN', 'Java', 'Solaris' and 'Buy-Side') that would allow me to create hard and soft groupings that were based on defined attributes (e.g. I could ask for a cloud of all objects that share a specific technical attribute, and another cloud of objects that share a specific business attribute)."

Comments

"Complexity kills"

[Baldrson]

This request brings to mind the now famous quote from Ray Ozzie regarding reorienting Microsoft to services: "Complexity kills".

There is actually a rigorous definition of complexity that is actually stated in terms of software: Kolmogorov complexity. Kolmogorov complexity is the size of the shortest program that produces, precisely, a given output. The number of bits in that program is the Kolmogorov complexity of the output. This is actually a very useful way of viewing system complexity in the case of software. One way of viewing programming is the compression of all the program's use cases into a program specification which, in theory, is executable. But there is more to it. As it happens -- unsurprising to many of us -- it is now a theorem of computer science that the closer the size of that "executable" gets to the Kolmogorov complexity of the use cases, the better.

This theorem is a major breakthrough in CS and should be learned in every institute purporting to teach IT.

It's important enough that I've proposed a prize award to Ray Ozzie. What follows is my email to him. We'll see if it makes it through his gatekeepers and then gets his attention.

Hi Ray,

I've got a simple and powerful idea that I think, based on your statement "Complexity kills," you'll find interesting.

It is a prize competition that I'm tentatively calling "The C-Prize" that rewards the most succinct representation of a major knowledge base.

Unsurprizingly to some of us, Ockham's Razor is more than a mere rule of thumb -- it turns out to be the foundation of intelligence. Marcus Hutter of the University of Lugano, Switzerland, recently provided a mathematical proof of this link between simplicity and intelligence which has withstood peer review. Dr. Hutter believes the C-Prize to be "an excellent idea".

The criterion is easy enough to state:

Let anyone submit a program that produces, with no inputs, one of the major natural language corpora (such as a Wikipedia snapshot) as output.

S = size of uncompressed corpus
P = size of program outputting the uncompressed corpus
R = S/P (the compression ratio).

Award monies:

Previous record ratio: R0
New record ratio: R1=R0+X
Fund contains: $Z at noon GMT on day of new record
Winner receives: $Z * (X/(R0+X))

The compression program and decompression program are made open source.

At present there is a small prize being administered by Leonid A. Broukhis based on the relatively tiny Calgary Corpus. Matthew Mahoney of the University of Miami has proposed a larger $50,000 compression prize to the National Science Foundation, but experience has shown it is like pulling teeth to get government agencies to fund prize awards -- they generally have to see private parties are doing so first. Marcus Hutter has put up a few small prizes based on some mathematics problems he needs solved for advancement of his theory of intelligence.

Any of these individuals could be a credible locus of control for the C-Prize -- or Microsoft itself could be the locus of control.