Solving the /etc Situation?

mrfibbi asks: "/etc is a mess, plain and simple. Each program has its own (incompatible) config file format and the naming scheme/hierarchy is left almost solely to the author. Furthermore, package updates are a mess, either choosing to replace the entire config file, reject any updated versions (which leads to inconsistencies), or, as is the case with etc-update, asking the user to manually merge the files, which takes forever after a big update. We've revamped /dev with udev, but we've still failed to come up with a universal, duct-tape-free solution for the problem. Though solutions exist, there has been little or no adoption, either due to a personal dislike for the idea or API, or just an indifference to the problem. Should we work toward migrating to an Elektra-like system? Something else? Or do most simply find it not worth the trouble?"

Comments

I got an ... _angle_

[Tumbleweed]

Simply port over the Windows registry, problem solved!

Re:I got an ... _angle_

[tzanger]

You are on crack. Seriously cheap crack. There are so many problems with your idea I can't even figure out where to start.

One badass file that everything clamours to get access to, even through a well defined API is going to be hell. Contention abounds. A bug can kill the entire system configuration. And then you throw the saviour of the digital world into it: XML. Why don't you just make it a binary file and have every single problem the Win32 registry has.

A unified configuration scheme is a great idea. XML really isn't IMO the solution, and one big badass file is certainly the wrong way to go about it.

I use et al

[christopherfinke]

Have you ever thought of using et al instead of etc?

This is just how OSS is...

[JacquesPinette84]

OSS is about choises, and /etc really highlights this. People have different ideas on how apps should be configured, and I think putting all configurations in one place was a good compromise.

Messes are inevitable

[squiggleslash]

Whatever solution you come up with, it's unlikely to remain clean for long. I'm sure the inventors of the Windows Registry thought they'd come up with a clean solution, and I know the GConf people did too. Are they clean? Do you like them?

I don't.

Leave /etc as it is. It works. It's done the right job now for coming on 30 years. It's had one major clean up (executables moved to /sbin), and otherwise it's nice. You can grep it. Help is a manpage away. With the exception of a handful of recent apps from programmers who don't "get it" and think XML is better than simplicity, the config files are, by and large, consistant, within a certain mindset.

There's lots of things that could be improved about Unix. Destroying /etc doesn't really improve it.

Re:Messes are inevitable

[Just Some Guy]

I agree, with the exception that I think XML is infinitely preferable to some of the hand-rolled workalikes I've seen, and you can use a standard set of tools to parse any particular XML file sitting around.

What's the conceptual differences between

[TopSection]
Setting1 = true
Setting2 = yes
Setting3 = no
Setting4 = false

and

<section name="Top">
<Setting1>true</Setting1>
<Setting2>true</Setting2>
<Setting3>false</Setting3>
<Setting4>false</Setting4>
</section>

except that the latter can be browsed and edited with any editor that understands XML (as opposed to only Vim with the file-format-specific highlighter), can be deterministically validated by generic tools, and doesn't require the program's author to implement his own configuration file parser?

XML isn't a perfect fit everywhere, but /etc is one of the places I think it works brilliantly.

Re:Messes are inevitable

[walt-sjc]

One is clean and readable, the other wordy and cluttered. I hate xml config files. I don't WANT to be forced to use some bizzaro XML editor, or deal with manually edited XML.

Re:Messes are inevitable

[Anonymous Coward]

Unbelievable. Just this little example you show demonstrates the utter insanity of XML.

The XML version is harder to read than the .ini version. It's hard to tell at a glance if the config is a valid config. Would you really be able to spot an error in a closing tag, for instance?

It is also harder to edit using command-line tools. It is harder to edit with ANY common unix tool, actually. You *must* use editors that understand XML (do you have editors like that on every machine by default? And no, syntax highlighting doesn't exactly count).

How do you diff/merge? How do you push out changes to *some* of the values to your farm of 100 machines? Remember, your XML file can look like this too, and still be equivalent:

<section
name='Top'
>
<Setting1>true</Setting1< Setting2>true</Setting2>
<Setting3
>false</Setti ng3><Setting4
>false</Setting4>
</section>

(the extra spaces here and there are /.'s doing)

Say goodbye to the entire arsenal of simple well-established unix text-munging tools. Say hello to multi-megabyte XML parsers, just to slurp up key/value pairs.

XML is definitely NOT what I'd want to see on my machines. Especially ones that are meant to be reliable.

Here's how I would write that hypothetical program. I'd create a directory /etc/my-app/top. The "true" settings can be set like this:

cd /etc/my-app
touch top/setting1 top/setting2

Done. Now my app can just check for the existence of the file as a true/false (or yes/no) flag. Do I need a new setting "top/send-alerts-to" that takes a string? No problem:

cd /etc/my-app
echo me@example.com > top/send-alerts-to

Am I use a high-availability system? Here's how to do the above atomically:

cd /etc/my-app
echo me@example.com > top/send-alerts-to.tmp
mv top/send-alerts-to.tmp top/send-alerts-to

Do I need to synchronize this config with another machine? But not the "send-alerts-to" setting? No problem:

rsync -a --exclude top/send-alerts-to /etc/my-app/ othermachine:/etc/my-app/

Do I need to temporarily change the config and leave a comment for another admin? Here I go:

cp -a /etc/my-app /etc/my-app.2005-03-02
touch /etc/my-app/top/setting6
echo "Temporarily activating top/setting6 because...." > /etc/my-app/README

Want to merge changes? No problem, just use common version control tools like CVS, SVN, Darcs, whatever, that you probably already have laying around. You don't need a special XML merge tool.

Want to quickly double-check a setting? Use grep, find, a GUI tool, whatever you want. NEVER underestimate the power of line-oriented text files. Searching XML with grep is a nightmare.

Am I just showing off my 'leet command line skeelz here? No I'm showing how simple and flexible this scheme is. It's pretty hard to make the configs unparseable or inconsistent. They can be automated, or edited with vim, or whatever. Heck, in your XML examples, you have to type each tag *twice* (open/close)! Yeesh. That doubles the chance of mistakes, right there.

as usual, complexity is the problem

[Anonymous Coward]

* Using XML is ABSOLUTELY NOT a solution. It is a problem. You can't assume XML-based tools available everywhere. I'm not going to link a 2MB XML library into my 1KB daemon. If you tell me I have to, I will put my config someplace else. I'm not going to put a 2MB XML library on my 8MB flash card. XML makes absolutely no sense for a mission-critical config.

* The config files are way too complex NOW, even without XML. Everybody invents pointless file formats. For instance if you need a flag for a program, just use the abscence or presence of a file in a directory in the filesystem! Don't create a format, write a parser, deal with error checking, etc, etc, just to set a damn flag. Another advantage to using the filesystem as a database (or "registry" if you prefer) is that it becomes super-easy to automate. Automation, simplicity, and guaranteed correctness are GOOD things.

Something to think about: your filesystem is a key/value database, for example "/etc/hostname" is a key, and the contents of that file is the value. There, you don't have to write a parser or a new config file format. Use what's already there.

I remember when Red Hat and other distros started setting up Apache (and other programs) so that they read extra config files out of a directory, instead of trying to automate in-place edits using sed or something. Suddenly everything got 10x easier. Need to install a new config? Just link or copy to the directory.

Taking this to the logical conclusion, you have DJB's software (qmail, djb-dns, etc). Ultra-minimal config using only environment variables or a few line-oriented files. Very flexible, easy to automate, updates can be atomic, merges/updates can done with ssh/rsync, and so forth. Configs can be placed anywhere you want, not just in a special spot in /etc.

Gentoo is on the right track in this department. They use filesystem-based configs for some things (for instance the bash-completion package can be configured by linking things into a directory).

* If you absolutely need a complex config, use a shell script that sets environment variables. Many distros (BSD, gentoo again) also use configs that are shell scripts. I.e. the config is just a script that sets some variables. If you want, you can add some code in the script to do calculations or have other logic. This is very nice and flexible. And you don't have to write a new parser, you can assume /bin/sh and basic Unix tools are present.

* Merging updates: this is impossible to automate. This is the same problem as branching/merging in a version control system. The best thing we can do is 1) KEEP THE CONFIGS SIMPLE as mentioned above 2) use version control to track changes and assist with the merge and 3) again, keep it simple! (Imagine what a nightmare XML would be for merging, by the way). You as an admin should know how to merge the configs, and the machine should make it as simple as possible.

Again Gentoo is on the right track here too. It makes it easy for you to diff/merge your new configs, and can even keep old revisions in version control with RCS.

* The name of the directory (/etc) is bad, it should be /config or something, but we're kinda stuck with it I think.

So in conclusion, /etc is not broken, it's not a problem, there's nothing to "solve". Just focus on simplicity. If your ego wants you to invent yet another config file format, DON'T. If your software has a bunch of useless config settings, trim them down, use command-line flags, do something, don't just pile them into a config file. Make your software work in some capacity WITHOUT a config. Look at DJB's software for ideas on how to keep your config simple and powerful. Use the power of pipes, shell scripts, line-oriented text files, and so forth. THINK about the busy admin who has to deal with your crap.

And please, no XML!

Use the filesystem!

[spitzak]

How many times do people have to come up with crazy ideas like XML and Registries or even elektra.

It's easy: make a file for each key and put the literal value (not encrypted or encoded or anything) into the file. Want the "registry key" foo/bar/baz? Look in ~/configuration/foo/bar/baz, and if it is no there look in /configuration/foo/bar/baz. Read the contents of the file. DONE!

The filesystem already supports a totally arbitrary name hierarchy, and the data is allowed to be blocks of bytes of arbitrary size. YOU DO NOT NEED ANYTHING ELSE!

Comments could be supported by renaming the file baz to baz.comment. You could also document the expected structure of each directory with a .template file in the parent directory. Many other ideas could be added. The important thing is all this structure can be IGNORED by a simple program that just wants to read configuration.

Elektra is close, but the files are not raw data.

I don't understand why this solution is not absolutely obvious to everybody. What is the problem? Why is anybody proposing anything other than this?