Solving the /etc Situation?

mrfibbi asks: "/etc is a mess, plain and simple. Each program has its own (incompatible) config file format and the naming scheme/hierarchy is left almost solely to the author. Furthermore, package updates are a mess, either choosing to replace the entire config file, reject any updated versions (which leads to inconsistencies), or, as is the case with etc-update, asking the user to manually merge the files, which takes forever after a big update. We've revamped /dev with udev, but we've still failed to come up with a universal, duct-tape-free solution for the problem. Though solutions exist, there has been little or no adoption, either due to a personal dislike for the idea or API, or just an indifference to the problem. Should we work toward migrating to an Elektra-like system? Something else? Or do most simply find it not worth the trouble?"

Comments

I got an ... _angle_

[Tumbleweed]

Simply port over the Windows registry, problem solved!

Re:I got an ... _angle_

[Anonymous Coward]

Now there's a good solution. I will take the chaotic state of /etc over the Windows Registry NIGHTMARE any day. Hell, I'd rather get a good swift kick in the nuts than deal with the Windows registry nightmare

Re:I got an ... _angle_

[BrynM]

So, we make it a nicely-formatted XML text file, obviously.

This sounds dangerous to me. Suppose and RPM writes a garbled line to the file by mistake. That could hose every app on the system. Don't get me wrong - a single file is a good idea, but it would take quite a bit of error checking for me to trust it.

When it comes down to it, I would prefer some heirarchal system. I think the habit of some apps (like Apache) having their conf files in a subdirectory of /etc might be a good way to go.

Re:I got an ... _angle_

[Tumbleweed]

See, I don't think an app should be writing to the file; it should be a system service that writes the data to the file, otherwise, there's no way to enforce what and where an application does to said file.

Re:I got an ... _angle_

[tzanger]

You are on crack. Seriously cheap crack. There are so many problems with your idea I can't even figure out where to start.

One badass file that everything clamours to get access to, even through a well defined API is going to be hell. Contention abounds. A bug can kill the entire system configuration. And then you throw the saviour of the digital world into it: XML. Why don't you just make it a binary file and have every single problem the Win32 registry has.

A unified configuration scheme is a great idea. XML really isn't IMO the solution, and one big badass file is certainly the wrong way to go about it.

Re:I got an ... _angle_

[Anonymous Coward]

Uhm, are you joking? What you describe has already been invented, it's called "the filesystem".

So, we make it a nicely-formatted XML text file, obviously.

This is the "joke" part. Let's ignore it.

Each user should be able to access only their portion of the file.

Unix filesystem permissions? Check.

Each application can access only their portion of the file,

Open only the files you need? Check.

and are _signed_ (by the system) as to what changed what and when.

Not 100% sure what you mean (how can the system enforce this). I'll assume you mean version control. We have that, it's called RCS (and of course more sophisticated ones but I use RCS on my servers). Check.

The amount of changes backed-up could be left to the user or sysad when the file is changed

Available with RCS. Check. Actually with RCS you have a complete history of the file going back to day 1.

Applications can't "hide" data in the registry

I can search the filesystem with find, locate, etc. Check.

In short, I think a registry is a good idea, but I just hate that the name has gotten such a horrible rap because of how Windows implements one.

Us Unix folks just call lightweight hierarchical key/value databases "the filesystem". Easy to back up, copy, merge, store, read, edit, understand, etc..

Why reinvent the wheel?

Re:I got an ... _angle_

[Foolhardy]

The only things centralized about the registry are that it has a single database server that does the actual file writes while providing a high level API, and it has a single-root hierarchy.
Data in the registry is stored in hives mounted at various places in the registry's hierarchy. The registry's root is a key named \REGISTRY in the Object Manager's namespace. Win32 confuses things a little by renaming \REGISTRY\MACHINE to HKEY_LOCAL_MACHINE and \REGISTRY\USER to HKEY_USERS, and providing some pseudo keys, like for the current user, but the idea is the same. The RegLoadKey, RegUnLoadKey and RegSaveKey functions can be used to dynamically mount, unmount and copy hives in the registry. The key \REGISTRY\MACHINE\SYSTEM\CurrentControlSet\Control \hivelist contains a list of all the hives to be automatically mounted at startup. Go ahead, divide the registry into as many hives as you want, and put them where ever you want.

What I'm trying to say is that the registry is no more centralized than the VFS is in Linux. The registry consists of various hives mounted at different locations, just like the VFS consists of various filesystems mounted at different locations.

As for messyness, is this due to the structure of the registry itself, or Microsoft's usage of it: do you really think that if Windows used an /etc it would be any better documented or organized?
Personally, I don't think the layout is all that bad.

Simple

[ambrosen]

/HKEY_LOCAL_MACHINE/ /HKEY_CURRENT_USER/ /HKEY_CLASSES_ROOT/ /HKEY_CURRENT_CONFIG/
& /HKEY_USERS/

should do it, I'd have thought. Then restrict it to a tree structure only under there and everybody should be happy.

I use et al

[christopherfinke]

Have you ever thought of using et al instead of etc?

so.. clearing the mess..

[gl4ss]

to clear /etc/ you would create another one that would need programmers to comply to that?

wouldn't it be actually easier to get them to place stuff in /etc/ so that they're logically placed and findable?

because going over to elektra seems actually more work. even elektra's web page admits.. "It is much more an agreement then a piece of software. Relation is 99% to 1%.".

do I have a problem with /etc/blaablaa? no. it's hell a lot better than registry still and at least I got SOME idea where the config file is.

Re:A simple request

[jon787]

A well commented config file with inline examples is even better. Sure they might end up being 10x longer than necessary, but they are readable.

This is just how OSS is...

[JacquesPinette84]

OSS is about choises, and /etc really highlights this. People have different ideas on how apps should be configured, and I think putting all configurations in one place was a good compromise.

Re:This is just how OSS is...

[Ogerman]

OSS is about choises, and /etc really highlights this. People have different ideas on how apps should be configured

OSS is about choices. But not all choices bring value to the community. As a long time Linux admin, I can tell you that peoples' different ideas on how apps should be configured does NOT bring any value whatsoever to Open Source. It does quite the opposite. We absolutely must find a compromise and bring sanity to the situation.

So far, Elektra is the best compromise. It doesn't require any new tools. It doesn't require any drastic changes in most software. It doesn't create any single points of failure. It retains the human readable and human editable nature of /etc. It makes security more granular. And most importantly, it opens the door for a new generation of GUI admin tools that will make Unix administration accessible to the masses of existing Windows admins who don't have time to re-learn everything they know. Of course, it'll also improve the lives of seasoned Unix admins as well. At least those who value their time..

Elektra + XML meta-data now! (:

I have the answer

[HawkingMattress]

Let's turn all those files into XML, integrate Xalan into the kernel, and change the cd command to have it evaluate XPATH requests. Then some magical thing will probably happen !

--
<runs_away type="duck" reason="run forest, run !"/>

How about changing the GPL?

[WebHostingGuy]

Insert a clause in the GPL stating that all programs using any part of any component or system must prepare their config files in the following format and must have the following directory structure /etc/program/bin, /etc/program/config, etc.

Seriously though. The best way would be to set up a standard and then begin to push the agenda across the linux distributions asking them to ask the developers to abide by the standard.

Another offbeat solution is that: (You can have a kernel module which reads the program files and if not in the correct format will tell the user it is wrong and therefore it will not run the binary. This would get users to 1. hack a solution or hopefully 2. complain to the developers to fix the problem.)

Or maybe make programs use a specific installer which does this...

Or you could chmod all the other directories to prohibit the programs from writing someplace else...

Messes are inevitable

[squiggleslash]

Whatever solution you come up with, it's unlikely to remain clean for long. I'm sure the inventors of the Windows Registry thought they'd come up with a clean solution, and I know the GConf people did too. Are they clean? Do you like them?

I don't.

Leave /etc as it is. It works. It's done the right job now for coming on 30 years. It's had one major clean up (executables moved to /sbin), and otherwise it's nice. You can grep it. Help is a manpage away. With the exception of a handful of recent apps from programmers who don't "get it" and think XML is better than simplicity, the config files are, by and large, consistant, within a certain mindset.

There's lots of things that could be improved about Unix. Destroying /etc doesn't really improve it.

Re:Messes are inevitable

[Just Some Guy]

I agree, with the exception that I think XML is infinitely preferable to some of the hand-rolled workalikes I've seen, and you can use a standard set of tools to parse any particular XML file sitting around.

What's the conceptual differences between

[TopSection]
Setting1 = true
Setting2 = yes
Setting3 = no
Setting4 = false

and

<section name="Top">
<Setting1>true</Setting1>
<Setting2>true</Setting2>
<Setting3>false</Setting3>
<Setting4>false</Setting4>
</section>

except that the latter can be browsed and edited with any editor that understands XML (as opposed to only Vim with the file-format-specific highlighter), can be deterministically validated by generic tools, and doesn't require the program's author to implement his own configuration file parser?

XML isn't a perfect fit everywhere, but /etc is one of the places I think it works brilliantly.

Re:Messes are inevitable

[walt-sjc]

One is clean and readable, the other wordy and cluttered. I hate xml config files. I don't WANT to be forced to use some bizzaro XML editor, or deal with manually edited XML.

Re:Messes are inevitable

[Anonymous Coward]

Unbelievable. Just this little example you show demonstrates the utter insanity of XML.

The XML version is harder to read than the .ini version. It's hard to tell at a glance if the config is a valid config. Would you really be able to spot an error in a closing tag, for instance?

It is also harder to edit using command-line tools. It is harder to edit with ANY common unix tool, actually. You *must* use editors that understand XML (do you have editors like that on every machine by default? And no, syntax highlighting doesn't exactly count).

How do you diff/merge? How do you push out changes to *some* of the values to your farm of 100 machines? Remember, your XML file can look like this too, and still be equivalent:

<section
name='Top'
>
<Setting1>true</Setting1< Setting2>true</Setting2>
<Setting3
>false</Setti ng3><Setting4
>false</Setting4>
</section>

(the extra spaces here and there are /.'s doing)

Say goodbye to the entire arsenal of simple well-established unix text-munging tools. Say hello to multi-megabyte XML parsers, just to slurp up key/value pairs.

XML is definitely NOT what I'd want to see on my machines. Especially ones that are meant to be reliable.

Here's how I would write that hypothetical program. I'd create a directory /etc/my-app/top. The "true" settings can be set like this:

cd /etc/my-app
touch top/setting1 top/setting2

Done. Now my app can just check for the existence of the file as a true/false (or yes/no) flag. Do I need a new setting "top/send-alerts-to" that takes a string? No problem:

cd /etc/my-app
echo me@example.com > top/send-alerts-to

Am I use a high-availability system? Here's how to do the above atomically:

cd /etc/my-app
echo me@example.com > top/send-alerts-to.tmp
mv top/send-alerts-to.tmp top/send-alerts-to

Do I need to synchronize this config with another machine? But not the "send-alerts-to" setting? No problem:

rsync -a --exclude top/send-alerts-to /etc/my-app/ othermachine:/etc/my-app/

Do I need to temporarily change the config and leave a comment for another admin? Here I go:

cp -a /etc/my-app /etc/my-app.2005-03-02
touch /etc/my-app/top/setting6
echo "Temporarily activating top/setting6 because...." > /etc/my-app/README

Want to merge changes? No problem, just use common version control tools like CVS, SVN, Darcs, whatever, that you probably already have laying around. You don't need a special XML merge tool.

Want to quickly double-check a setting? Use grep, find, a GUI tool, whatever you want. NEVER underestimate the power of line-oriented text files. Searching XML with grep is a nightmare.

Am I just showing off my 'leet command line skeelz here? No I'm showing how simple and flexible this scheme is. It's pretty hard to make the configs unparseable or inconsistent. They can be automated, or edited with vim, or whatever. Heck, in your XML examples, you have to type each tag *twice* (open/close)! Yeesh. That doubles the chance of mistakes, right there.

Re:Messes are inevitable

[drinkypoo]

Hahahaha!

I guess I don't see how "tag: value" is inherently easier than "<tag&gtvalue</tag>".

You might not be able to see how, but thanks to your example, the rest of us have had an excellent demonstration.

The good, the bad, and the ugly

[gseidman]

What's the real problem with /etc anyway? If you said "all the different and wacky file formats" then you're missing the forest for the trees. Those different file formats exist because it is appropriate to have a file format that suits the purpose of the application. /etc/password and /etc/group are designed for exactly the purpose for which they are used, for example. It doesn't make much sense for apache to have a file format similar to, say, exim; they are applications with entirely different purposes.

The real problem is that the user doesn't automatically know how to correlate a process, configuration file, startup script, etc., and it affects more than just /etc. Suppose I am administering a Debian machine with CUPS, since I happen to have one handy. Something is wrong with printing. I have to somehow know all of the following:

1) Printing services are managed by CUPS.
2) To start and stop printing services, I need to run /etc/init.d/cupsys
3) If that doesn't work properly, the process to kill is named cupsd.
4) The config file for specific printers for CUPS is /etc/cups/printers.conf
5) The config file for the cups service as a whole is /etc/cups/cupsd.conf
6) To manage most things about CUPS (though not browsing and sharing), I don't really want to touch the files in /etc/cups but, rather, connect to http://localhost:631/ with a web browser.

Just for kicks, let's consider the same situation under MacOS X (10.3):

1) I can control pretty much everything about printing services (including browsing and sharing) from the Print & Fax section of the System Preferences
2) If I really need to mess with the config files, they are in /etc/cups, but I never do.
3) If I really need to manually start and stop printing services I use /System/Library/StartupItems/PrintingServices/Prin tingServices, but I never do.
4) If I really need to use the web interface to CUPS, it is still at http://localhost:631/

The really important thing here is that on a Mac, I don't even need to know that printing is handled by CUPS. The down side is that on any *nix, unlike Windows, I don't have a way of seeing all the services my machine is providing (to itself or the net). Of course, under Windows it's pretty hard to tell what a service is really for, or how to configure it. When you get right down to it, each system has some good ideas, but none of them have it right.

I want a system where I can get a list of running services (not just all processes), the ports on which they are listening (I don't care if they are TCP, UDP, Unix domain, whatever passes for IPC on Windows, or even Mach ports), and be able to trivially turn a service on, or off, or configure it with a simple commandline or a click of the mouse.

Give me the Windows services manager (with a console equivalent) with the ability to see through what interfaces (e.g. ports) that service is provided, the ability to configure when and how it is started (e.g. at which runlevel), the ability to configure it directly from there (e.g. bring up a custom configurator, a web browser with the appropriate URL, or a text editor on the right files with a double-click in the list). I don't care where the configuration is on disk (though this better integrate well with backup software), nor how it's stored (though there had better be a human-readable text format that can be exported and imported, if necessary).

Furthermore, there should be a separation of default configuration of user applications and configuration of services. Why does /etc/profile belong in the same place as /etc/inetd.conf? That gets back to the central question, of course, and I think the answer there is that each package should be responsible for managing its own (default and user) configuration and, similarly

Re:A simple request

[jon787]

Well with the larger config files knowing where to start can be an issue, so a well commented default config file is nice.

as usual, complexity is the problem

[Anonymous Coward]

* Using XML is ABSOLUTELY NOT a solution. It is a problem. You can't assume XML-based tools available everywhere. I'm not going to link a 2MB XML library into my 1KB daemon. If you tell me I have to, I will put my config someplace else. I'm not going to put a 2MB XML library on my 8MB flash card. XML makes absolutely no sense for a mission-critical config.

* The config files are way too complex NOW, even without XML. Everybody invents pointless file formats. For instance if you need a flag for a program, just use the abscence or presence of a file in a directory in the filesystem! Don't create a format, write a parser, deal with error checking, etc, etc, just to set a damn flag. Another advantage to using the filesystem as a database (or "registry" if you prefer) is that it becomes super-easy to automate. Automation, simplicity, and guaranteed correctness are GOOD things.

Something to think about: your filesystem is a key/value database, for example "/etc/hostname" is a key, and the contents of that file is the value. There, you don't have to write a parser or a new config file format. Use what's already there.

I remember when Red Hat and other distros started setting up Apache (and other programs) so that they read extra config files out of a directory, instead of trying to automate in-place edits using sed or something. Suddenly everything got 10x easier. Need to install a new config? Just link or copy to the directory.

Taking this to the logical conclusion, you have DJB's software (qmail, djb-dns, etc). Ultra-minimal config using only environment variables or a few line-oriented files. Very flexible, easy to automate, updates can be atomic, merges/updates can done with ssh/rsync, and so forth. Configs can be placed anywhere you want, not just in a special spot in /etc.

Gentoo is on the right track in this department. They use filesystem-based configs for some things (for instance the bash-completion package can be configured by linking things into a directory).

* If you absolutely need a complex config, use a shell script that sets environment variables. Many distros (BSD, gentoo again) also use configs that are shell scripts. I.e. the config is just a script that sets some variables. If you want, you can add some code in the script to do calculations or have other logic. This is very nice and flexible. And you don't have to write a new parser, you can assume /bin/sh and basic Unix tools are present.

* Merging updates: this is impossible to automate. This is the same problem as branching/merging in a version control system. The best thing we can do is 1) KEEP THE CONFIGS SIMPLE as mentioned above 2) use version control to track changes and assist with the merge and 3) again, keep it simple! (Imagine what a nightmare XML would be for merging, by the way). You as an admin should know how to merge the configs, and the machine should make it as simple as possible.

Again Gentoo is on the right track here too. It makes it easy for you to diff/merge your new configs, and can even keep old revisions in version control with RCS.

* The name of the directory (/etc) is bad, it should be /config or something, but we're kinda stuck with it I think.

So in conclusion, /etc is not broken, it's not a problem, there's nothing to "solve". Just focus on simplicity. If your ego wants you to invent yet another config file format, DON'T. If your software has a bunch of useless config settings, trim them down, use command-line flags, do something, don't just pile them into a config file. Make your software work in some capacity WITHOUT a config. Look at DJB's software for ideas on how to keep your config simple and powerful. Use the power of pipes, shell scripts, line-oriented text files, and so forth. THINK about the busy admin who has to deal with your crap.

And please, no XML!

Use the filesystem!

[spitzak]

How many times do people have to come up with crazy ideas like XML and Registries or even elektra.

It's easy: make a file for each key and put the literal value (not encrypted or encoded or anything) into the file. Want the "registry key" foo/bar/baz? Look in ~/configuration/foo/bar/baz, and if it is no there look in /configuration/foo/bar/baz. Read the contents of the file. DONE!

The filesystem already supports a totally arbitrary name hierarchy, and the data is allowed to be blocks of bytes of arbitrary size. YOU DO NOT NEED ANYTHING ELSE!

Comments could be supported by renaming the file baz to baz.comment. You could also document the expected structure of each directory with a .template file in the parent directory. Many other ideas could be added. The important thing is all this structure can be IGNORED by a simple program that just wants to read configuration.

Elektra is close, but the files are not raw data.

I don't understand why this solution is not absolutely obvious to everybody. What is the problem? Why is anybody proposing anything other than this?

Why I don't have a problem with /etc

[phaze3000]

Personally, I don't consider /etc to be a problem. /etc is a directory to be edited by the system administrator, and as a system administrator, one should damn well know what one is doing editing config files. End users shouldn't have to touch these files.

I fail to see how a 'common format' is going to help anyone - it's not the format that causes head-scratching, it's working out how to achieve what one wants using the options available.

Because of the simple text file nature of /etc one can easily place the directory into some sort of revision control system (RCS, CVS and the like), making tracking of changes extremely simple.

My view...

[agraupe]

Config files should be in separate directories of /etc based on the package/program they are for. On gentoo, for example, I have /etc/portage/. This is how it should work. It makes everything easy to find. I don't find etc-update a terrible amount of work, because I know that, if I didn't touch the config file in question, it is safe to update, because I'll be moving from a default to a default. As it is, it's not a big problem, because most programs have .conf files with descriptive names. As for the idea of a consistent API, I think it is doomed to failure because applications have such vastly different requirements from their config files. There are config files that just have the default settings for a program, all the way to something like the apache config file that is, essentially, the only way to control the program. As for the idea of one file, like the Windows Registry: no. Aside from insecurity, this means that a group (that, say, admins the HTTP server) can not be allowed to modify apache's config, but nothing else.