How to Fix the Unix Configuration Nightmare

jacoplane writes: "There's an interesting article on freshmeat talking how sorting out some kind of standard for configuration could really help Unix systems could be more user friendly. The article points out that since Apple has managed to build a quite usable system on top of NetBSD, it should be doable to do the same for open-source interfaces."

Comments

There are valid reason

[evilviper]

There are situations were I've wondered why they went off and created their own configuration file format just for that one application, but that's not too common. The problem with a universal format is that each application need to do different things. Typing ALL apps to one format would in many cases make configuration changes more difficult.

i.e. instead of changing one thing, you have to make the change for every interface/terminal/etc it is running on. Just an example mind you.

Maybe the author should try webmin

[evil_roy]

Have a look. webmin

Don't confuse Syntax and Standards

[osolemirnix]

I like the idea, such a tool would certainly be nice. Unfortunately the article only focuses on the Syntax part, the file format and access. I would think that is rather easy, use some kind of XML and you're already halfway there.

The part that makes such a system really useful however, is a standard agreement of which information is stored and what it means. This is where the Windows Registry falls down. And Unix is even worse, because all it has is some common soft-of-agreed-upon shell variables, like $EDITOR etc.
Apple is able to do this better because they set the standards for the OS (even more than MS). The can have one central "registry" for something like default associations of MIME-types with particular applications and define an API so every application can use it and a user doesn't have to change his settings in his browser AND his mail client AND his ftp client, etc.

Given the diversity of the unix crowd, the latter seems difficult to me. Maybe they can include it as part of LSB for a start?

Re:Don't confuse Syntax and Standards

[j7953]

I would think that is rather easy, use some kind of XML and you're already halfway there.

No no no. XML is not the magic solution to all problems in communication, data storage and usability. Using XML does have many advantages for some applications, like communicating over the internet, but I don't see how

<DocumentRoot> /some/path </DocumentRoot>

is so much more usable than

DocumentRoot /some/path

The only thing XML will do if widely used for configuration files is make maintaining them harder for those who'd like to do that by using shell scripts or simple text editors. As a very basic example, using XML makes configuration files harder to grep. The above example might also be written like this:

<DocumentRoot>
/some/path
</DocumentRoot>

This is completely legal XML, but if you grep DocumentRoot this file, you'll get the following result:

<DocumentRoot>
</DocumentRoot>

which is probably not what you wanted.

XML as a starting point perhaps?

[andykuan]

In the writer's outline section he has a few bullet-points that scream "XML!" (I'm paraphrasing here):

1. A core system to handle parsing, verification, etc. -- If app configs were based around XML, you could use any of a dozen XML parsers out there.

2. A configuration format description file. -- Hmmm, sounds like a DTD...

3. OS-neutral. -- XML was meant to be portable from the get-go.

The other items lined up pretty well with what XML is all about as well. Anyway, that said, I'm not so sure this'll ever happen since, from a developer's point of view, it's a lot simpler to slurp in a bunch of lines from a text file with fgets and chop it up into words than to have to somehow link in a third-party XML parser.

Re:XML as a starting point perhaps?

[khuber]

In the writer's outline section he has a few bullet-points that scream "XML!"

I'd agree that XML is a good basis, but "XML" really doesn't provide much by itself. It's just a file format that is human readable. If you just use XML with a bunch of proprietary tags, your own XML language so to speak, you really don't gain much over the existing different syntax config files.

An automated tool has no clue what your ipaddress (or whatever) tag means at all. You need to provide additional context for tools to understand the semantics of the configuration data. To make configuration files understandable in a more intelligent sense, you need to either restrict the tags you use to your own configuration language, or you need to provide metadata of some sort.

Why is this intelligence necessary? Well there are all sorts of dependencies and relationships in configuration files. You might want a GUI to let you know if you change something that may break another setting, and so on. Plus ideally you would only allow legal values to be set. Data typing could be done with W3C XML Schema Definition Language, or RELAX NG schemas.

Which brings me to RDF, which I think would be better suited to this task than XML alone. If you use RDF (see http://www.w3.org/RDF/ ) you make it much easier to have a self-describing format that tools can do more intelligent things with than raw XML. While I don't think RDF, DAML+OIL, et al is enough to create a Semantic Web as Tim Berners-Lee is hoping, it _is_ a step in a higher-level direction that will support more intelligence in processing data.

Mozilla already uses RDF for various configuration files and I'm sure there are other applications that do too. Mozilla has a whole bunch of stuff about their RDF here.

XML is just a tree of "stuff" in human-readable format. RDF lets you set up properties and relationships in the data in a standardized way. I don't have a brilliant example to prove this to skeptics, but really it is a better way to represent a lot of types of data you want to be able to query. There are many knowledge bases, expert systems and other query engines already out there using RDF and even higher-level languages like DAML+OIL.

-Kevin

Other side to this

[larien]

This is something I think would help linux become more widely accepted, but there's more to it.

I was thinking this morning how I hadn't udpated my Apache server in a while, and wondering whether I should apt-get the latest version (Apache is kind of important for security, as it's the only open port on my system from the internet). However, I've done various tweaks to config files which would get overwritten if I accepted the Debian standard file, but I don't want to miss out on any new settings that could be important. I know I can do a diff, but that's effort I'd rather not have to go through.

This kind of situation is where a registry-like interface is useful; the install program just has to do a quick 'if-not-exist then add' to any new settings and leave the rest alone (or ask if you want an overwrite of all settings, with appropriate disclaimers).

This kind of thing is difficult to do in a flat file split into different sections (if there isn't a concept of sections, you can just tag the setting on), but trivial in a registry structure, especially when the tool (dpkg-upgrade/apt/rpm) has to handle all the different file formats. However, linux/Unix users would rebel en masse if the registry got inflicted on them (with damn good reason! I like being able to fix problems from single user mode using vi!), but some form of layer between the text file and settings may provide the best of both worlds (programmatic ease and editability). apt/dpkg/rpm could use the interface to add/modify settings without splatting your custom tweaks while still adding the new required settings.

Unfortunately, we're starting from a difficult point, with thousands of applications with many different requirements for their settings. Hopefully we can get some covergence over time.

Apple fixed it...

[Adnans]

Because there is only one captain on the ship, Apple. Good luck fixing it in the Linux world. The only way it might have a chance of working IMHO is if such a proposal gets included the Linux Standard Base. Here's a bold idea, why not copy the way Apple does it??? No need to reinvent the wheel...

-adnans

WebMin...not the Right Thing but damn good

[Spoing]

I've mentioned this before, so I'll just quote myself;

1. Get Webmin. Setup Webmin. Use Webmin. Show Webmin to all other administrators. Teach them Webmin. Eventually, when the time is right, show Webmin to management. Drive home the idea that Webmin is a one-stop-shopping, simplified, and robust server management web app that anyone can grasp (if not master). Point out that Webmin supports Unix (including OSX) but not Windows servers (except for Windows with Cygwin). Over time, point out that on the server side, the few remaining Windows servers take much longer to manage and are less reliable then the multitude managed under Webmin.
2. www.webmin.com/webmin

The complaints about Webmin is that it isn't perfect. It doesn't solve the desire for a universal config mechanism or encourage the editing of files directly. OK. It doesn't. Yet, it exists now and is a lifesaver when using multiple UNIXs. Anyone who wants to know what any Webmin module is doing probably also knows how to use find -cmin, ls -lart, or read the module itself since they tend to be in Perl.

When a universal config comes along, I'll use it and guess what -- it'll work with WebMin.

Webmin might not be the right thing, but it is good enough for a vast number of uses.

I hope this goes somewhere

[Twylite]

A brilliant article, and a subject which I have tried before (with little success) to broach with various OpenSource developers. Unfortunately the comments demonstrate the lack of understanding that is all too common in the free software world: an inability to distinguish between apis/formats and software implementations; and an unwillingness to move to a more user friendly system.

GConf and Webmin, for example, are not solutions that address the problem posed by the article. Other comments completely lose the plot: this is about a common configuration format which can use metadata to provide for configuring just about any application, it is not about creating an all-encompassing configuration that will give settings to all applications everywhere for all time.

Gconf is half of the solution at best. It provides a library (common abstraction) for accessing the data. But an application still has to have explicit knowledge of the data in order to act on it. A configuration framework describes this knowledge to allow any compliant application to act on it.

Sometimes an example is worth 1000 pictures (and hence about 1000000 words), so consider a simple packet filtering application with an ACCEPT list and a DENY list, which can listen on several interfaces.

Such an application would have a meta-data configuration file (say /etc/conf/meta/filter.cfg) which specifies: section name=interface count=multiple type=ipv4 ; section name=accept count=multiple type=cidr ; section name=dent count=multiple type=cidr. Now any application can access the packet filter's meta-data file, and the real configuration file (say /etc/conf/filter.cfg), and understand the meaning, to a degree, of the configuration. The types "ipv4" and "cidr" will have to be primitives for the configuration framework.

Thus a general configuration application could present a tree which has three branches (interface, accept, deny), each of which displays a screen with a list. Appropriate entries can then be added, modified or removed, reconfiguring the packet filter without the configuration application having any prior knowledge of the filter's configuration or requirements.

With adequate forethough (for the categorisation and types required) such a system can be extremely powerful. A single configuration library can be used by the application, as well as GUI and CLI configuration clients. You could easily have a command in a script: config pktfltr add -section interfaces -value 1.2.3.4 .

A common library like this can also provide powerful features that a lot of applications lack, like local (user) settings which override global defaults. The library can maintain global, group and user configurations for each application, and getting/setting preferences will take the overlays into account.

Implementing such a system would also provide an ideal opportunity to introduce configuration versioning. CVS (for example, or arch, which includes directory versioning) could be used to maintain various versions of the configuration files in /etc/conf.

Arguments about diversity are largely moot. The data format is independant of the data in any well-written program. .INI files, Bind-style files, XML files and SAMBA files can all represent the same information, but they look different. Some are arguably more efficient in some situations. Even the feared Windows registry has the same functionality ... it just makes the (fatal) error of storing everything in a mostly inseparable datastore (sortof like an /etc/conf directory in which you can't treat the various files separately).

This is a suggestion which needs to be followed through, and supported by the community.

Install/configure

[autopr0n]

Sure, installing software is easy. Unpack and compile. We are talking about configuring software.

Like getting sendmail to rout and forward mail, apache to serve up web pages you want, BIND to bind names to IP address, etc.

These tasks arn't "hard" or anything, but they do require a lot of reading on the part of a Newbie. In the windows world (don't know much about OSX) Most of that stuff can be done via am intuitive GUI.

Flame me if you want, but I'd greatly prefer a system that didn't require me to learn diffrent config file formats for each service I want to have running... or deal with a hodgepoge of 'easy config' program hacks.

A simple, standard configuration system is definetly the way to go.

Re:Install/configure

[walt-sjc]

This is a defeatist attitude. Did you even read the article? If MS or Apple can do it, we can do it, and we can do it better.

Now a GUI won't make people understand what they are doing or do the work for them (ever configure MS Exchange? That's a bitch....) but what it DOES do is make the configuration easier to navigate.

By using an XML based file format (as I proposed in a previous comment) you get the best of both worlds. The Apache config file is a close example of what I was thinking (it's not XML, but it is XML "like".)

An application would actually have 2 configuration files: one that actually contains the configuration data (like httpd.conf for apache) and the other to describe the options, help for the options, defaults, etc. to the configurator GUI. Both would be XML, would be extensible as hell, and provide a single tool for configuring any application, current or future.

As I mention in a previous post, a set of importers / exporters would provide a way to migrate until the various applications were updated to use the new format (sendmail may NEVER change...)

So you get what you want. Hell it would probably be better as you would no longer need to memorize all the differet quirks / rules for different applications's configuration language.

Some Stupid Questions

[heideggier]

Would it be possible to get rid of dotfiles? While alot of people like them I don't know why you couldn't have a ~/etc in the home dir. I get lot's of weird looks when I say, you keep all the configurations in /etc but user config's in dotfiles. There's must be a good reason for this could someone please explain

On that thread what the hell is /usr/etc used for, or /usr/local/etc? won't it make more sense to move /etc here instead?

Do we really need a central registry type file like the article proposes, surly isn't a front end enough, I can't understand what difference this would make. Having a central file which contain's everything like System.dat seem's to be a major security floor in windows since if you can crack that file you pretty much have root on the machine. Not being able to set different permissions for something like fstab or shadow would be pain. Although to be fair the article proposes it as a backup. If it ain't broke don't fix it.

Apart from these questions I think that this was a pretty good/fair article, although I thought that it was a bit unfair to compare linux to windows/OSX since they are completely different OS and something like creating a central config app might work to make linux less customisable IMHO. Most distro's tend to have app's along these lines anyway, what the hells wrong with DrakConf?.

It seems you're missing the point

[Nailer]

The problem isn't with the lack of a standardized configuration tools, its a lack of standardized configuration file formats. This allows:

• Systems administrators to work with new applications config files without worrying about how exactly a comment should look
  
• Automated parsers that don't have to be re-written for every config file - you'd have a whole buynch more webmin modules if this was the case.
  
• GUI folk to be able to use their graphical interfaces knowing that they won't screw up a config file, because there's a very rigidly defined format for the config file and a standard parser that's very well tested
  
• Better manipulation, imports, exports, etc of information into config files, as a popular format allows for the development of tools to perform these tasks across many different applications
  

Some *correct* information about MacOS X config

[lcracker]

Yes, I know this is a comment on freshmeat (I wrote it).. but it pains me to see so much incorrect/incomplete information here that people just believe for lack of experience with OS X.

---

MacOS X's solution to this problem is a couple levels
deep. There are three (?!) systems for configuration:

(1) /etc (which is a symlink to /private/etc):
It's still there, used for a few things. But not much.
More or less just stuff that runs before everything else,
or stuff that apple doesn't have much control over, like
Apache.

(2) Netinfo database:
basically takes a lot of stuff out of /etc and puts it in a
"registry", this registry is accessible over a network
connection for remote administration and such. You
can also do some funky server/client stuff with this too.
There's a command line interface to it that lets you
read/change 'keys' and also dump out certain keys in
well known /etc style formats

(3) Property Lists (.plist):
This is the way that just about all configurable
information in MacOS X is stored. There are a couple
different kinds of plists that the Core Foundation
libraries understand, but the most prevalent of which is
the XMLified version, the others are either vestigal or
are used primarily for localization of strings. Every
application (well, bundled app) on the system has an
info.plist file that has application settings that the OS
reads from on app startup, but you can also put your
own tags in there. Applications also have the option of
putting their own tags in the Resources subdirectory in
their bundle, which is the preferred way. OS specific
preferences (requires admin privs) go in /System/
Library/Preferences, System-wide app preferences go
in /Library/Preferences, Network-wide app prefs (only
applicable if you have OS X Server on the network) are
in /Network/Library/Preferences, and User app prefs go
in ~/Library/Preferences.

The Core Foundation framework manages more or less
all of the work. They'll find the plist for you, let you
work with the data structures, and serialize/deserialize
it to plist format. It's really quite nice to use. Just
about all of the data structures in Core Foundation are
serializable, even the property list data structures, so
sometimes you'll see property lists with serialized
property lists inside them.

As far as unified configuration interfaces go, there
really isn't a need for any in Mac OS X.

You're on your own with /etc, as nothing there is meant
to be novice-tweaked anyways (In OS X).

There is a GUI NetInfo Manager and command line tools
to configure the stuff in that database (nothing a
novice needs to touch).

Configuring the plists is primarily done inside the
applications that you use (since it's so easy for the
developer to use Core Foundation to read/write
property lists), there are no separate configuration
programs, but there are command line tools for working
with plists. If you're using the developers kit, you get a
little (not very poweful and pretty inefficient) program
to edit plist files graphically, but otherwise you either
do it by hand with a text editor or the command line
tools and risk breaking things or let the apps take care
of it themself.

BTW, all of the Core Foundation stuff is available in two
flavors: C, and Objective C. The C libraries are pretty
straightforward and work in all situations, but if you're
writing an app using the Cocoa framework in Obj C, it's
way easier.. though there are a few little peculiarities
about the Obj C implementations. The bonus is that
you can use both at the same time, and Core
Foundation types / Cocoa instances are interchangable
by some voodoo that Apple did. If the Cocoa
implementation doesn't work right (As with serializing a
plist that has NSNumber Boolean types), you just use
the Core Foundation implementation and a typecast..
voila, one line of code changed and everything is
happy.

Re:Some *correct* information about MacOS X config

[Jordy]

NetInfo exists for the same reason NIS, NYS, YP, etc exist; to centralize system configurations across a network of machines. Things like printer configurations are typically the same in a corporate environment.

NetInfo is actually Open Source. Though I'm not sure how up-to-date it is.

As for /etc... some of those are synchronized from NetInfo, not the other way around. A good number of them are just stub files, but there are a few that do not exist at all in NetInfo at all.

Re:I /like/ the Unix Configuration Nightmare

[Shanep]

that is UNIX. Take it or leave it.

I agree. I'm reading all this wondering why people are complaining so much. How much time do people spend in /etc anyway?

I need to get something set up, I go to /etc and either see a file or directory named something along the lines of what I'm configuring. Get it up and if I don't know it well, either quickly read the comments or the blah.conf man page... configured... working... I'm done in here for a while until I think of some other cool thing to do.

When Win95 went to a single, non-text config file (registry), I instantly thought this was going to be a really bad move. And it proved to be. Registry rot is incredible. Anyone ever notice that if you reinstall Windows about once every year or so, you get a massive boost in performance? I installed 98 the other day (just for D2:LOD, Starcraft and RainbowSix), it boots (PII-300, 512MB) in about 30 seconds and shuts down in 1-2 seconds. Prior to this, 98 was taking an age, *if* it would shut down at all, and I had all sorts of stability problems. BTW, I put the C: on a 700MB partition and used a D: of 2.5GB for the games. I burn the C: to a CDR and when things get bad, just dd it straight back to hda1.

My point is (back to the OSes that are more fun than a Frost Nova up Andariels arse), *I* want to mess with those config files, rather than have some app(s) messing with one really important file.

I like it too.

A few things

[HalfFlat]

The current system for starters isn't too bad - the configuration files for important system tools are typically well documented (in man pages if nowhere else), and aren't too hard to edit in a text editor. Certainly there is a problem with the use of automated GUI tools for system configuration, but I can't really see why GUI tools are so important.

Consistency though would be a nice thing, if only so that as more programs are used and require configuration, we don't all have to learn more and more ad hoc configuration schemes. There is the potential for the current system to get well out of hand. Consistency is nice from an aesthetic point of view - ad hocery is rarely pretty.

Tools such as linuxconf or webmin don't help the problem, they just obscure it. Often badly! (Don't get me started on linuxconf, ugh!)

The earlier suggestions of XML, or a recognised subset of XML, together with XML schema to describe the formats, seem ideal as far as file formats go:

• It's standard and well-documented.
• There are a large number of parsers that work with a useful large subset of XML, for many platforms and languages.
• In a pinch, it's not hard to write a quick and dirty and correct-enough parser from scratch on an ad hoc basis, to parse your own particular XML-based configuration files.
• It's human editable (or should be, if used wisely).
• It is rich enough to encompass all the text-format configuration files currently in common use (under the Linux-based operating systems I know of at least).

The big problem of course is inertia: changing configuration files is a big change! Personally, I think the cost though would be worth it in the long term.

The important thing to me is that there is no API specified as mandatory. While a standard API for accessing and manipulating configuration data would be nice, requiring it to be used will not only make adoption of a standard config format slower and harder, but will also limit software developers to the tools for which the API has an implementation. I'd go for:

1. Use XML + schema as the mandatory config file format in the new scheme.
2. Have a preferred mechanism for signalling to applications that configuration data has changed (eg SIGHUP) so that changes can be made atomically.
3. Provide an API for manipulation of config data, but only as a convenience to developers and automated config tools.

I also agree with the earlier comment promoting the use of ~/etc over dot-files. Makes good sense! Again, it's only inertia that will make such a change difficult. Though if other changes are going through ... in for a penny, in for a pound!

Lastly, whatever scheme is adopted, it seems to me essential that there be some flexibility in where the configuration files are located. Standard places are important, but there must also be some way to say to an application that their config data is in /usr/local/etc, or in /opt/package-name/etc, or wherever. This is necessary for sane package isolation and management, especially if not all software on a system is maintained by someone with superuser priveleges.

It took this long?

[Ageless]

It took MacOS X for people to realize that this was a problem in UNIX? Please.

The reason UNIX and UNIX applications are hard to configure, in most cases, is because Open Source programmers are lazy.

This is obviously a blatent generalization so I will explain.

The old adage is that an Open Source program gets written when a programmer has an "itch" they decide to scratch. The problem is that very few people are itched by configuration. You may write the best web server in the world (Apache!) but by time it comes to writing the configuration manager for it the volunteers start falling away.

It isn't very fun writing a bunch of dialogs, windows, buttons and such to make a nice configuration for a program. It's kinda like documentation (and we all know the state of docs for many UNIX programs).

I see examples of this every day. I have a Mac OS X using friend who sends me the URL of every new program he decides to use. It's incredible how many of them are UNIX ports with a beautiful configuration manager stuck on. Mac programmers hold themselves to a higher level of user experience and UNIX people need to get on the boat.

What's needed isn't a global, all dancing, all singing configuration system. What is needed is responsibility in programming.

P.S. Everyone always whines about the Windows registry because it's binary, you can't edit it blah, blah, blah... But the fact is: It works. The average user never cares to edit it because they config their programs from WITHIN their programs. If something is truly needed, do the Windows registry in text file format. Make it /etc/registry.conf. There is no reason it has to be binary.

Re:No

[MaxVlast]

Do you know what that means? "Based on NeXT" doesn't mean anything. NeXT was a company.

Mac OS X is based on OpenStep 4.2, which, itself, was based on NEXTSTEP 3.3. NEXTSTEP is a BSD operating system running on a modified version of the Mach microkernel. OpenStep is a API specification and a set of libraries that conforms to that API. OpenStep 4.2 (the operating system) is an implementation of those libraries on top of NEXTSTEP.

When Apple bought NeXT, they planned to build on top of OpenStep. They first produced Rhapsody for PPC and Rhapsody for Intel. They were the same OS running on two hardware platforms. On top of Rhapsody, Apple put the Blue Box, which was a Macintosh compatibility environment. At no time was there any need for a "BSD compatibility layer." It was all software running on top of BSD. Apple then killed Rhapsody for Intel (and the Yellow Box, but that's tangential.)

What was left was released as Mac OS Server.

Mac OS X 10.0 and Mac OS Server 10.0 (and further versions) are also BSD operating systems. They have the Cocoa (OpenStep) and Carbon libraries available, and the imaging system is called Aqua (replacement for Display PostScript.) At no point in any of this is there a need for any UNIX compatibility layer, as it is all real UNIX. The only compatibility environment necessary is for Mac OS 9 (Classic.) Only certain older applications (Carbon) can run natively on OS X, so for running non-Carbon apps, Mac OS 9 is run in a compatibility environment (similar, but not the same as VMWare.)

I hope that clarifies things.

Screw XML, use Python code

[Tony Hammitt]

We should use actual Python code as the configuration file format. It's callable directly from C, and by extension, all other languages. It's nice and clean. It supports heirarchial inclusion of other configuration files. It has easily readable comments. In short, it's the perfect configuration file specification language.