Slashdot Mirror
Programming Environment For "Event Correlation"?
sireenmalik asks: "Of late I have become interested in this field of research namely Event Correlation on a Distributed Network System. The more I read about it, the more ignorant I feel. There is so much to it: distributed network systems, databasing, artifical intelligence (neural networks, baysian belief networks, rule based,etc.), software engineering, computer science, telecommmunication....etc. If I were to really attack it from a programming point of view, can somebody tell me what tools and languages should I use? I suppose it will be a realtime environment. Academicians support ADA but I can't figure how the artificial intelligence part will be done. If I use PROLOG/LISP I get into HEAP management business which really is a dragon for realtime systems. C/C++ .. Java....? To add the list I also know about the diverse implementations using JIRO (from SUN), ECDL (from HP), RAPIDE (from Stanford.edu), JAVA Management API, ELAVA, GEM Language, MODEL Language, IF/PROLOG......and the list goes on and on and on! It's interesting as well as confusing (I can't help but agree here). Let's talk about it. Maybe something useful happens here?"
Personally, I'd use some sort of scripting environment, probably Perl, for all of the prototyping, playing around, data translation, and glue work.
And then for serious stuff, I'd use something more structured and formal. Personally, I like Java; it's well known, pretty well supported, good at handling errors, has a reasonable amount of network stuff built in or available, and from the beginning had threading in mind. But I would certainly use some sort of OO language for this, as this kind of work strikes me as well-suited for an OO approach.
Oh, and do your work incrementally! Avoid grand plans; code in an exploratory fashion. If you're doing OO work, the book Refactoring is a great one. And if you're doing incremental design in Java, you will soon grow addicted to unit testing; I have grown to love JUnit, a Java unit testing framework.
But if this is mainly for your own use, use whatever tools you are most familiar with. I find that tackling a new tool and a new problem domain at the same time is generally too much to allow quality work. New problem? Old tool. New tool? Old problem first!
Unless you have an awful lot of processor power to spare, why would you even think about doing this processing in real-time?
Theres several advantages to this approach:
- you don't have to have such a fast machine
- the data collection software can be *simple*
- you don't alter the data collection software when you alter your analysis
- you have the raw data to hand for applying more analysis if you need to do a second pass.
For real-time processing I would look at using an offline analysis to generate state machines for recognizing events. And I would get these machines to *generate* events into the stream as well. That way you can build your analysis hierarchically by recognizing subpatterns and building patterns from them.
In any case, from a practical standpoint 'real-time' processing would not spot some of the most interesting things - such as an event pattern recurring close to a regular period of minutes,hours,days,weeks... - eg network failures due to load and due to incorrect scheduled jobs have a differnt appearance - both occur regularly but the schedule failure would have a more precisely regular period. Unless you plan to accumulate state over long periods of time and watch for such things I reckon you'll miss a lot of important recurrences.
DARPA archives should help you a lot.
[glad I'm out of THAT arena]
/(o\ I'm not a medievalist - I just play one on weekends!
buttfucking flapping anal spew, riding in the creases
jizzum powered lifeboat. tidal wave, a flood of rectal
mucus, jolly laughing bear wipes pickle off his face.
burping brown bubbles, shit smeared nasal hole. getaway
spinning phallus suction cup popping. rusty metal dustpan
scraping dead paint, airborn virus lead paint wall-licking
little boy. "mommy watch me lick the wall" said little
fredrick. he banged his head against the floor, bloody
snot catfood. his mom wiped up the mess with vinegar and
toilet paper, remembering to clean the ears with a fork.
green fog filled the cavern, red rocks flecked with
gold each a blinking nose. hot stuffy air infecting
ashy lungs with speared micromen. toxic waste released
into bloodstream. wild orgy of man fuck apocalypse.
hyperdimensional spear fuck rotten scalp meat blast off
display. the little men jumped out of fleshy gold noses
and fell, hundreds dropping like gassed jews. falling
falling ass spread wide, smack landing on sharp clean
spears. misty yellow and brown. others gag on the
smoke, puking out innards. warpole ass ram fiery phallus
anal splinter. newly birthed micro fetus budding and
popping off eggs. splash of bloody and shit. upturned
noses fill with smog and thick juicy flesh. hyper
vibrate rumbling cavern walls, scalding lava flashing
red, green, blue, steaming. dead fetii hung on a
clothesline to keep them out of the horrid storm. they
cry for more as string is run in rectum and out the
heart, pulsating throbbing yellow. screams of micro
baby fish, countered by puke and shit. crisscrossed
spiders lair net of jewels each shiny anus reflecting
shit splatter. explosion of smog and lights. still the
micro men jump to their anal death, smack on the shiny
spear. glare of spearheads and shiny fish ass. rising
tide, mixed blue green. the walls worn away by waves
of hot jizzum spear smacked erections. brown spots shot
in red walls. melting walls. strings pulled tighter
baby fish sliding back and forth indian burned asshole
heart skinned. slosh slosh slosh. the sky opens up,
golden light pierces the cave, melting rotten bloody
flesh, smog rushes out. golden noses spurt bloody
fireworks. dry baked cracking walls crumble. fault
lines rumble, arise the demons. a swirl of black smoke
and laughter. arms and legs, half-rectums, dead fish,
nose pieces all whip against the walls hyper smack.
flesh dangles from hooks and smacks dead bodies.
equilibrium.
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org
wat is yuor problme you evil evil troal!!!?!?? you stop floodign my site nwo or ill sick my fat webmastre on yuor ass, CaowboyNeel! he'll kike your asse from here to the Lunix confernece...
-- CmdrTaco
Took a huge dump tonight, left a bunch of floaters in my toilet stinking up the place.
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org
Forget multiple languages, if you're not doing it in real time, use Ruby for the "serious stuff" and the glue. It's more OO than Java, has great network support and is also threaded. I forget the link, but there was a genetic programming package posted recently (check the RAA). It also has a unit testing module (RubyUnit).
Other than that, I agree.
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org
stoooooopppppp!!!!!!!!
*throws a childlike fit*
-- CmdrTaco
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org
Am I making poor baby Taco cry?
*shits on your face a little*
Is that better?
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org
J. Wipo Troll, Esq.
Crapflooder Associates
Slashdot.org