Researchers Expanding Diff, Grep Unix Tools

itwbennett writes "At the Usenix Large Installation System Administration (LISA) conference being held this week in Boston, two Dartmouth computer scientists presented variants of the grep and diff Unix command line utilities that can handle more complex types of data. The new programs, called Context-Free Grep and Hierarchical Diff, will provide the ability to parse blocks of data rather than single lines. The research has been funded in part by Google and the U.S. Energy Department."

Comments

Strange names

[gnasher719]

Space characters in the name of a Unix command line tool is asking for trouble.

Re:Strange names

[realyendor]

I expect those are just the spoken names and that the commands will still be single words, similar to:
"GNU awk" -> gawk
"enhanced grep" -> egrep

Re:Strange names

[ripler]

Next thing you know we'll have CSIgrep. (enhance enhance enhance grep)

Re:Strange names

[Longjmp]

Definitely
II mean, where would we end up if unix commands actually give a hint what they are doing ;-)
As a unix novice, if I wanted to search for something, my first choice of course would be grep
Also if I wanted help on something, the first word that jumps to my mind would be man

heh.

Re:Strange names

[mytec]

According to this paper, they are called bgrep and bdiff.

Re:Strange names

[Anne Thwacks]

CSIgrep would take 30 mins to get the result! (With ad breaks)

Re:Strange names

[iluvcapra]

If you don't like a tool's name, export an alias.

It's not about typing commands as much as it's about making these work:

$ find . -name ".txt" | xargs wc
$ for file in $*; do
mv $file old/$file
done

Versus these:

$ find . -name ".txt" -print0 | xargs -0 wc
$ for file in $*; do
mv "$file" "old/$file"
done

A lot of scripts you run into are just broken because of braindead assumptions.

Re:Strange names

[toadlife]

"I have only been able to come up with one algorithm for creating Unix command names: think of a good English word to describe what you want to do, then think of an obscure near- or partial-synonym, throw away all the vowels, arbitrarily shorten what's left, and then, finally, as a sop to the literate programmer, maybe reinsert one of the missing vowels."

Rachel Padman

Re:Strange names

[mfnickster]

There's nothing that says the name of the tool and the command you type must be the same

Very true. Unix programmers seem to follow these rules:

1. delete any spaces in the name
2. delete any vowels in the name
3. delete any superfluous consonants
4. chuck the entire thing and just abbreviate it to the first letter of each word in the name

So these tools will likely be run as "ctxtfrgrp" and "hierdiff" or just "cfgrep" and "hdiff"

Re:Strange names

[jd]

You have to figure in two's complement notation. If it's sufficiently counter-intuitive, the sign bit flips over and it becomes totally intuitive.

Re:Strange names

[morgauxo]

GP was a joke I am sure.

As to yours though.. I wouldn't want spaces in my commands. How do you tell where the command ends and the arguments begin?

As for man... man is the MANual. That's not that bad is it? Ok, help might be a little better but it's not a big deal unless you are very closed minded. It's really a history thing. Man wasn't just somebody's idea of a help command. Unix (or Unics as it was called back then) originally actually had a manual. As in dead trees paper! It got big. Real big. One day Dennis Ritchie accidentally dropped a copy and killed his dog. Flattened the poor girl like a pancake. After that he decided it needed to be digital. Man is a digital copy of that original dog killing book plus decades of additions and updates. Thus it is man(ual).

Now should manual have been "manual" or maybe the real whole title "Unix Programmers Manual"? It might be easier to remember. 5 years after you learned that command and you are still typing it 5 times a day would you still appreciate the ease of using real whole English words? Are you that abc? (abreviationally challenged) Or do you just really love typing. Is your r/l name Mavis Beacon?

That's how a lot of Unix commands are, they make plenty of sense with history. I'm sure grep and the others all have their own stories. Well.. not all. How much of a story does it take to come to ls is a lazy way to type list? Oh, yah, you are AbC. Sorry about that.

Yes, the history of decades old programming decisions isn't really something you want to learn to use an OS (or any other software). But what's the alternative? Throw everything out x number of years and start over? It sounds great when you are a hopeless newbie but once you actually learn something do you want to do it all over again every 10 years just to make it easy for the next batch of basement kiddies? Your clock is ticking too you know! Now get off my lwn!!!! (lawn)

P.S. Ok, Ok, I made up the dog part of the story. But it COULD have happened! The rest was real. Actually, I don't KNOW that it didn't happen... hmm....

Re:Strange names

[rk]

Unix is user-friendly, it's just picky about who its friends are.

Link to one of their papers on these tools

[treerex]

http://www.cs.dartmouth.edu/reports/TR2011-705.pdf

Re:Follow the money...?

[Tanktalus]

Context-free grep/diff can be used to search for data/changes in arbitrary non-line-record-based files. Such as XML, HTML, JSON, SQLite databases, other databases, Apache configs, and many other pieces of data. Heck, even most programming languages are not line-based, but statement terminated/separated. Imagine being able to grep for a function name, and getting its entire prototype/usage even when it spans multiple lines (very common in standard glibc headers). And, depending on the plugin's capabilities, you could grep for a function name as a function name and not get back any usage of that text as a variable or embedded in a string, or a comment (skip commented-out calls!).

If there's sufficient configurability, you could ask for the entire block that given text is in, and such a grep would be able to display everything in the corresponding {...}. Makes grep that much valuable.

So, my question is, why aren't more IT-heavy corporations/government departments not involved?

RTFA?

[DragonWriter]

funded in part by Google and the U.S. Energy Department

I wonder what's the interest of these two in this.

FTFA:

Google's interest in this technology springs from the company's efforts in cloud computing, where it must automate operations across a wide range of networking gear, Weaver said. The DOE foresees that this sort of software could play a vital role in smart grids, in which millions of energy consuming end-devices would have connectivity of some sort. The software would help "make sense of all the log files and the configurations of the power control networks," Weaver said.

Terrible idea

[deblau]

This violates so many rules of the Unix philosophy that I don't even know where to begin...

FTFA:

Grep has issues with data blocks as well. "With regular expressions, you don't really have the ability to extract things that are nested arbitrarily deep," Weaver said.

If your data structures are so complex that diff/grep won't cut it, they should probably be massaged into XML, in which case you can use XSLT off the shelf. It's already customizable to whatever data format you're working with.

FTFA:

With [operational data in block-like data structures], a tool such as diff "can be too low-level," Weaver said. "Diff doesn't really pay attention to the structure of the language you are trying to tell differences between." He has seen cases where dif reports that 10 changes have been made to a file, when in fact only two changes have been made, and the remaining data has simply been shifted around.

No, 10 changes have been made. The fact that only two substantive changes have been made based on 10 edits is a subjective determination. That is, unless you want to detect that moving a block of code or data from one place to another in a file has no actual effect, in which case good luck because that's a domain-specific hard problem.