Solving the /etc Situation?

mrfibbi asks: "/etc is a mess, plain and simple. Each program has its own (incompatible) config file format and the naming scheme/hierarchy is left almost solely to the author. Furthermore, package updates are a mess, either choosing to replace the entire config file, reject any updated versions (which leads to inconsistencies), or, as is the case with etc-update, asking the user to manually merge the files, which takes forever after a big update. We've revamped /dev with udev, but we've still failed to come up with a universal, duct-tape-free solution for the problem. Though solutions exist, there has been little or no adoption, either due to a personal dislike for the idea or API, or just an indifference to the problem. Should we work toward migrating to an Elektra-like system? Something else? Or do most simply find it not worth the trouble?"

Comments

Simple

[ambrosen]

/HKEY_LOCAL_MACHINE/ /HKEY_CURRENT_USER/ /HKEY_CLASSES_ROOT/ /HKEY_CURRENT_CONFIG/
& /HKEY_USERS/

should do it, I'd have thought. Then restrict it to a tree structure only under there and everybody should be happy.

Re:I got an ... _angle_

[Anonymous Coward]

Now there's a good solution. I will take the chaotic state of /etc over the Windows Registry NIGHTMARE any day. Hell, I'd rather get a good swift kick in the nuts than deal with the Windows registry nightmare

A simple request

[HotNeedleOfInquiry]

Ok, I'm just a humble hardware engineer that has to support a couple Linux servers. So when I have to muck with /etc I make sure I'm close to a google window where I can run for help.

I'd be very grateful for nothing more than a README file, in /etc, for each application and service that tells what the files are named and what they do. Is that so much to ask?

Re:A simple request

[jon787]

A well commented config file with inline examples is even better. Sure they might end up being 10x longer than necessary, but they are readable.

Re:A simple request

[Curtman]

A well commented config file with inline examples is even better.

I don't see the problem with the man pages. Except when they are non-obvious. But anything that simply doesn't have one should be filed as a bug IMHO.

Re:A simple request

[jon787]

Well with the larger config files knowing where to start can be an issue, so a well commented default config file is nice.

Re:I got an ... _angle_

[Tumbleweed]

Okay, the serious answer is that yes, 'a' registry would be a very useful thing - just not implemented in a half-assed, non-secure, non-manageable manner ala Windows.

So, we make it a nicely-formatted XML text file, obviously.

Each user should be able to access only their portion of the file.

Each application can access only their portion of the file, and are _signed_ (by the system) as to what changed what and when.

The amount of changes backed-up could be left to the user or sysad when the file is changed ("Do you want to backup the previous version?", etc.)

As long as each user & app can only access their section, then modifications can be made predictable. Applications can't "hide" data in the registry, and since things can only access their data or the public data, information can be hidden from other users/apps who don't have a need to know.

In short, I think a registry is a good idea, but I just hate that the name has gotten such a horrible rap because of how Windows implements one.

Re:I got an ... _angle_

[bersl2]

The user registry should be in the user's home directory.

And do you know how disorganized a single file can get? Much more than a single directory, that's for sure. I like having a separate config file for things because file system operations are so much easier to handle than text editing alone.

Some of you might find /etc messy, but I think it's a flowerbed compared to the jungle that is the Windows registry.

Re:I got an ... _angle_

[BrynM]

So, we make it a nicely-formatted XML text file, obviously.

This sounds dangerous to me. Suppose and RPM writes a garbled line to the file by mistake. That could hose every app on the system. Don't get me wrong - a single file is a good idea, but it would take quite a bit of error checking for me to trust it.

When it comes down to it, I would prefer some heirarchal system. I think the habit of some apps (like Apache) having their conf files in a subdirectory of /etc might be a good way to go.

Messes are inevitable

[squiggleslash]

Whatever solution you come up with, it's unlikely to remain clean for long. I'm sure the inventors of the Windows Registry thought they'd come up with a clean solution, and I know the GConf people did too. Are they clean? Do you like them?

I don't.

Leave /etc as it is. It works. It's done the right job now for coming on 30 years. It's had one major clean up (executables moved to /sbin), and otherwise it's nice. You can grep it. Help is a manpage away. With the exception of a handful of recent apps from programmers who don't "get it" and think XML is better than simplicity, the config files are, by and large, consistant, within a certain mindset.

There's lots of things that could be improved about Unix. Destroying /etc doesn't really improve it.

Re:Messes are inevitable

[walt-sjc]

One is clean and readable, the other wordy and cluttered. I hate xml config files. I don't WANT to be forced to use some bizzaro XML editor, or deal with manually edited XML.

Re:Messes are inevitable

[bairy]

This isn't flamebait, it's a good point. xml files are cluttered with tags (I might add the tags are usually longer than the data they hold). Standard ini-style files are easily editable by notepad without hassle, and it would be very easy to design an editor for them if people wanted one.

Re:Messes are inevitable

[Anonymous Coward]

Unbelievable. Just this little example you show demonstrates the utter insanity of XML.

The XML version is harder to read than the .ini version. It's hard to tell at a glance if the config is a valid config. Would you really be able to spot an error in a closing tag, for instance?

It is also harder to edit using command-line tools. It is harder to edit with ANY common unix tool, actually. You *must* use editors that understand XML (do you have editors like that on every machine by default? And no, syntax highlighting doesn't exactly count).

How do you diff/merge? How do you push out changes to *some* of the values to your farm of 100 machines? Remember, your XML file can look like this too, and still be equivalent:

<section
name='Top'
>
<Setting1>true</Setting1< Setting2>true</Setting2>
<Setting3
>false</Setti ng3><Setting4
>false</Setting4>
</section>

(the extra spaces here and there are /.'s doing)

Say goodbye to the entire arsenal of simple well-established unix text-munging tools. Say hello to multi-megabyte XML parsers, just to slurp up key/value pairs.

XML is definitely NOT what I'd want to see on my machines. Especially ones that are meant to be reliable.

Here's how I would write that hypothetical program. I'd create a directory /etc/my-app/top. The "true" settings can be set like this:

cd /etc/my-app
touch top/setting1 top/setting2

Done. Now my app can just check for the existence of the file as a true/false (or yes/no) flag. Do I need a new setting "top/send-alerts-to" that takes a string? No problem:

cd /etc/my-app
echo me@example.com > top/send-alerts-to

Am I use a high-availability system? Here's how to do the above atomically:

cd /etc/my-app
echo me@example.com > top/send-alerts-to.tmp
mv top/send-alerts-to.tmp top/send-alerts-to

Do I need to synchronize this config with another machine? But not the "send-alerts-to" setting? No problem:

rsync -a --exclude top/send-alerts-to /etc/my-app/ othermachine:/etc/my-app/

Do I need to temporarily change the config and leave a comment for another admin? Here I go:

cp -a /etc/my-app /etc/my-app.2005-03-02
touch /etc/my-app/top/setting6
echo "Temporarily activating top/setting6 because...." > /etc/my-app/README

Want to merge changes? No problem, just use common version control tools like CVS, SVN, Darcs, whatever, that you probably already have laying around. You don't need a special XML merge tool.

Want to quickly double-check a setting? Use grep, find, a GUI tool, whatever you want. NEVER underestimate the power of line-oriented text files. Searching XML with grep is a nightmare.

Am I just showing off my 'leet command line skeelz here? No I'm showing how simple and flexible this scheme is. It's pretty hard to make the configs unparseable or inconsistent. They can be automated, or edited with vim, or whatever. Heck, in your XML examples, you have to type each tag *twice* (open/close)! Yeesh. That doubles the chance of mistakes, right there.

Re:Messes are inevitable

[drinkypoo]

Hahahaha!

I guess I don't see how "tag: value" is inherently easier than "<tag&gtvalue</tag>".

You might not be able to see how, but thanks to your example, the rest of us have had an excellent demonstration.

Re:I got an ... _angle_

[Tumbleweed]

See, I don't think an app should be writing to the file; it should be a system service that writes the data to the file, otherwise, there's no way to enforce what and where an application does to said file.

Re:I got an ... _angle_

[tzanger]

You are on crack. Seriously cheap crack. There are so many problems with your idea I can't even figure out where to start.

One badass file that everything clamours to get access to, even through a well defined API is going to be hell. Contention abounds. A bug can kill the entire system configuration. And then you throw the saviour of the digital world into it: XML. Why don't you just make it a binary file and have every single problem the Win32 registry has.

A unified configuration scheme is a great idea. XML really isn't IMO the solution, and one big badass file is certainly the wrong way to go about it.

as usual, complexity is the problem

[Anonymous Coward]

* Using XML is ABSOLUTELY NOT a solution. It is a problem. You can't assume XML-based tools available everywhere. I'm not going to link a 2MB XML library into my 1KB daemon. If you tell me I have to, I will put my config someplace else. I'm not going to put a 2MB XML library on my 8MB flash card. XML makes absolutely no sense for a mission-critical config.

* The config files are way too complex NOW, even without XML. Everybody invents pointless file formats. For instance if you need a flag for a program, just use the abscence or presence of a file in a directory in the filesystem! Don't create a format, write a parser, deal with error checking, etc, etc, just to set a damn flag. Another advantage to using the filesystem as a database (or "registry" if you prefer) is that it becomes super-easy to automate. Automation, simplicity, and guaranteed correctness are GOOD things.

Something to think about: your filesystem is a key/value database, for example "/etc/hostname" is a key, and the contents of that file is the value. There, you don't have to write a parser or a new config file format. Use what's already there.

I remember when Red Hat and other distros started setting up Apache (and other programs) so that they read extra config files out of a directory, instead of trying to automate in-place edits using sed or something. Suddenly everything got 10x easier. Need to install a new config? Just link or copy to the directory.

Taking this to the logical conclusion, you have DJB's software (qmail, djb-dns, etc). Ultra-minimal config using only environment variables or a few line-oriented files. Very flexible, easy to automate, updates can be atomic, merges/updates can done with ssh/rsync, and so forth. Configs can be placed anywhere you want, not just in a special spot in /etc.

Gentoo is on the right track in this department. They use filesystem-based configs for some things (for instance the bash-completion package can be configured by linking things into a directory).

* If you absolutely need a complex config, use a shell script that sets environment variables. Many distros (BSD, gentoo again) also use configs that are shell scripts. I.e. the config is just a script that sets some variables. If you want, you can add some code in the script to do calculations or have other logic. This is very nice and flexible. And you don't have to write a new parser, you can assume /bin/sh and basic Unix tools are present.

* Merging updates: this is impossible to automate. This is the same problem as branching/merging in a version control system. The best thing we can do is 1) KEEP THE CONFIGS SIMPLE as mentioned above 2) use version control to track changes and assist with the merge and 3) again, keep it simple! (Imagine what a nightmare XML would be for merging, by the way). You as an admin should know how to merge the configs, and the machine should make it as simple as possible.

Again Gentoo is on the right track here too. It makes it easy for you to diff/merge your new configs, and can even keep old revisions in version control with RCS.

* The name of the directory (/etc) is bad, it should be /config or something, but we're kinda stuck with it I think.

So in conclusion, /etc is not broken, it's not a problem, there's nothing to "solve". Just focus on simplicity. If your ego wants you to invent yet another config file format, DON'T. If your software has a bunch of useless config settings, trim them down, use command-line flags, do something, don't just pile them into a config file. Make your software work in some capacity WITHOUT a config. Look at DJB's software for ideas on how to keep your config simple and powerful. Use the power of pipes, shell scripts, line-oriented text files, and so forth. THINK about the busy admin who has to deal with your crap.

And please, no XML!

/etc is a mess?

[ConceptJunkie]

There's something that's much better. It's called "\Documents and Settings".

Very organized, consistent, and above all easy to understand.

Re:I got an ... _angle_

[Anonymous Coward]

Uhm, are you joking? What you describe has already been invented, it's called "the filesystem".

So, we make it a nicely-formatted XML text file, obviously.

This is the "joke" part. Let's ignore it.

Each user should be able to access only their portion of the file.

Unix filesystem permissions? Check.

Each application can access only their portion of the file,

Open only the files you need? Check.

and are _signed_ (by the system) as to what changed what and when.

Not 100% sure what you mean (how can the system enforce this). I'll assume you mean version control. We have that, it's called RCS (and of course more sophisticated ones but I use RCS on my servers). Check.

The amount of changes backed-up could be left to the user or sysad when the file is changed

Available with RCS. Check. Actually with RCS you have a complete history of the file going back to day 1.

Applications can't "hide" data in the registry

I can search the filesystem with find, locate, etc. Check.

In short, I think a registry is a good idea, but I just hate that the name has gotten such a horrible rap because of how Windows implements one.

Us Unix folks just call lightweight hierarchical key/value databases "the filesystem". Easy to back up, copy, merge, store, read, edit, understand, etc..

Why reinvent the wheel?

Re:The good, the bad, and the ugly

[Anonymous Coward]

Those different file formats exist because it is appropriate to have a file format that suits the purpose of the application. /etc/password and /etc/group are designed for exactly the purpose for which they are used, for example. It doesn't make much sense for apache to have a file format similar to, say, exim; they are applications with entirely different purposes.

You have conflated "syntax" and "data model" into "file format". It doesn't make sense for apache to have a file format similar to exim because their data models are different. But that doesn't mean that they shouldn't share the same syntax.

You might as well say that there's no reason for Slashdot and Google to have the same syntax (HTML) because they serve completely different purposes. You are looking at the problem at the wrong level.

Re:so.. clearing the mess..

[Ogerman]

to clear /etc/ you would create another one that would need programmers to comply to that?

Elektra-style configuration could easily be phased in over time. That's part of the beauty and simplicity of it.

wouldn't it be actually easier to get them to place stuff in /etc/ so that they're logically placed and findable?

It wouldn't solve the problem -- which is that it takes too much time and effort to administer a system using today's /etc mess. It has very little to do with logical /etc organization. It has everything to do with trying to figure out what the options are, what the valid values are, and how to integrate the slew of packages necessary to make a working system for real-world needs. Yes, the real world needs standard, Open Source GUI admin tools.

The Elektra design is on the right track. The only thing it is missing is optional XML meta-data that can describe the key-value hierarchy to a new generation of admin tools. Everything else is perfectly taken care of by the filesystem. Imagine if, instead of editing long, heavily commented config files, you could browse through a logical hierarchy of the options. Whenever you select an option, a side panel lists the valid values, describes their purpose, and gives usage examples. Upon edit, the admin tool validates the value entered and logs the change with a timedate stamp. If you want, you can add notes too.. these are stored in your home directory or optionally within the metadata as well. (though a different file)

Beyond configuring individual packages, switching a standardized config hierarchy would enable a new generation of intelligent, modular admin tools. As example, imagine if the software could automatically "wire up" all the tools needed for a complete mail or file server. You choose what packages you want and the admin tool looks up a default "recipe" for making them work together. Then all you have to do is configure the specifics for you installation. Sure beats endless HOWTO's for every combination of Linux distro, MTA, IMAP server, webmail, spam filter, virus filter, etc.

I've been administering Linux for 8 years now and this would absolutely be a dream come true.

Use the filesystem!

[spitzak]

How many times do people have to come up with crazy ideas like XML and Registries or even elektra.

It's easy: make a file for each key and put the literal value (not encrypted or encoded or anything) into the file. Want the "registry key" foo/bar/baz? Look in ~/configuration/foo/bar/baz, and if it is no there look in /configuration/foo/bar/baz. Read the contents of the file. DONE!

The filesystem already supports a totally arbitrary name hierarchy, and the data is allowed to be blocks of bytes of arbitrary size. YOU DO NOT NEED ANYTHING ELSE!

Comments could be supported by renaming the file baz to baz.comment. You could also document the expected structure of each directory with a .template file in the parent directory. Many other ideas could be added. The important thing is all this structure can be IGNORED by a simple program that just wants to read configuration.

Elektra is close, but the files are not raw data.

I don't understand why this solution is not absolutely obvious to everybody. What is the problem? Why is anybody proposing anything other than this?

Re:Use the filesystem!

[mikrorechner]

I don't understand why this solution is not absolutely obvious to everybody. What is the problem? Why is anybody proposing anything other than this?

Sounds interesting, but editing a program's configuration would be a royal pain in the ass with this system. Right now, I can open the conf file, browse through it, read the commentaries and change what needs to be changed. With your proposal, I would have to open a new file for each setting, plus one for the commentary. Unless, of course, there is a special editor for this.

Re:This is just how OSS is...

[Ogerman]

OSS is about choises, and /etc really highlights this. People have different ideas on how apps should be configured

OSS is about choices. But not all choices bring value to the community. As a long time Linux admin, I can tell you that peoples' different ideas on how apps should be configured does NOT bring any value whatsoever to Open Source. It does quite the opposite. We absolutely must find a compromise and bring sanity to the situation.

So far, Elektra is the best compromise. It doesn't require any new tools. It doesn't require any drastic changes in most software. It doesn't create any single points of failure. It retains the human readable and human editable nature of /etc. It makes security more granular. And most importantly, it opens the door for a new generation of GUI admin tools that will make Unix administration accessible to the masses of existing Windows admins who don't have time to re-learn everything they know. Of course, it'll also improve the lives of seasoned Unix admins as well. At least those who value their time..

Elektra + XML meta-data now! (:

Why I don't have a problem with /etc

[phaze3000]

Personally, I don't consider /etc to be a problem. /etc is a directory to be edited by the system administrator, and as a system administrator, one should damn well know what one is doing editing config files. End users shouldn't have to touch these files.

I fail to see how a 'common format' is going to help anyone - it's not the format that causes head-scratching, it's working out how to achieve what one wants using the options available.

Because of the simple text file nature of /etc one can easily place the directory into some sort of revision control system (RCS, CVS and the like), making tracking of changes extremely simple.

That looks cleaner than my home folder...

[ultramkancool]

my home folder has like 352 items in it + my other 2 home folders (for myself with differnt usernames (because of all my distro changing)) it's 690. now /etc only has 210 it's no mess

Re:The good, the bad, and the ugly

[gseidman]

Those different file formats exist because it is appropriate to have a file format that suits the purpose of the application. /etc/password and /etc/group are designed for exactly the purpose for which they are used, for example. It doesn't make much sense for apache to have a file format similar to, say, exim; they are applications with entirely different purposes.

You have conflated "syntax" and "data model" into "file format". It doesn't make sense for apache to have a file format similar to exim because their data models are different. But that doesn't mean that they shouldn't share the same syntax.

You might as well say that there's no reason for Slashdot and Google to have the same syntax (HTML) because they serve completely different purposes. You are looking at the problem at the wrong level.

It isn't quite that simple, though you raise a good point. While it would be possible to represent all of the same information in XML, for example, that is represented in /etc/password, the exim config, and the apache config, it doesn't make sense to do so. While it might even make sense to take the larger and more complicated configs and unify their syntax, it is highly desirable to be able to use standard unix tools such as grep and cut to be able to get information from /etc/password and the like.

P.S. If I were a moderator, I'd mod the parent anonymous post up.

My view...

[agraupe]

Config files should be in separate directories of /etc based on the package/program they are for. On gentoo, for example, I have /etc/portage/. This is how it should work. It makes everything easy to find. I don't find etc-update a terrible amount of work, because I know that, if I didn't touch the config file in question, it is safe to update, because I'll be moving from a default to a default. As it is, it's not a big problem, because most programs have .conf files with descriptive names. As for the idea of a consistent API, I think it is doomed to failure because applications have such vastly different requirements from their config files. There are config files that just have the default settings for a program, all the way to something like the apache config file that is, essentially, the only way to control the program. As for the idea of one file, like the Windows Registry: no. Aside from insecurity, this means that a group (that, say, admins the HTTP server) can not be allowed to modify apache's config, but nothing else.