Slashdot Mirror

I have been using Courier for over two years now. No remote roots ever or problems of any kind (I am amazed!). It's open sourced and a full package (esmtp, pop, imap, webmail and a thousand other things). It gets my vote.

I used it for a couple months because I wanted to have Maildir type mailboxes and wanted an IMAP server, it would crash all the time and give me all kind of troubles. I then switched to Binc IMAP (Binc is not courrier), which claim to be better than Courrier, but it was actually worse. It wouldn't last one week without crashing and send a lot of junk in syslog. I finally settled for dovecot with qmail. I have been running it for 6 months now without any problem.

Currently version 6 is in beta, probably to be released real-soon-now, and it includes the dovecot IMAP server. This is proving to be a champ of an IMAP server, particularly when integrated with the e-smith automation. It runs great on even low-end hardware, is proving robust yet easy to manage; a real winner.

My suggestion is to download a copy and take a look. Also check out the add-ons that take advantage of the e-smith templating & web management systems. There's even a marvelous set of Lazy Administrators command line tools for making bulk changes to accounts and settings. For a nearly turn-key solution it is quite impressive.

For those looking for support Mitel has a commercial version of e-smith called the Mitel Networks 6000 Managed Application Server which offers more groupware features and other nice things.

I was using Courier which works pretty well. But the configuration is a mess, and there have been some performance issues when using SSL (Outlook Express was ornery, for example).

Switching to Dovecot was pretty easy and I noticed an immediate performance increase. The support of Maildirs is a must-have.

I use Cyrus in a number of my packaged configurations, but for ease of migration and security Dovecot seems promising, although it lacks many of the advanced features that Cyrus has. It would probably be helpful to know exactly how many users you'll be serving and what your mail volume is. You might drop by #cyrus on irc.freenode.net and chat with people there.

You could, of course, look around and hire a Linux consultant to set it up for you.

I have been using Courier [courier-mta.org] for over two years now. No remote roots ever or problems of any kind (I am amazed!). It's open sourced and a full package (esmtp, pop, imap, webmail and a thousand other things). It gets my vote.

I used it for a couple months because I wanted to have Maildir type mailboxes and wanted an IMAP server, it would crash all the time and give me all kind of troubles. I then switched to Binc IMAP (Binc is not courrier), which claim to be better than Courrier, but it was actually worse. It wouldn't last one week without crashing and send a lot of junk in syslog. I finally settled for dovecot with qmail. I have been running it for 6 months now without any problem.

Two things:

1) There is an internal Mozilla preference to tell Mozilla to check *all* IMAP mailboxes, rather than just your INBOX. Enter ``about:config`` in the address bar and create the following boolean pref:

mail.check_all_imap_folders_for_new

and set it to ``true``.

2) IMAP is not client-side check only -- the protocol allows for server-side checking and notification. Have a look at the new GPL IMAP server called Dovecot which has support for this.

Although it's not been released as stable yet, it's current version number is 0.99.9.1 (sound familiar? :) Once we're happy that the last bugs are squished, it will most likely replace wu-imapd on our site IMAP servers.

Cheers,

David

Here's something to read for IMAP client authors: IMAP Client Coding HOWTO.

NOTE: This is a temporary location for this document, selected with slashdot people specifically in mind.

Copyright (C) 2002 Timo Sirainen <tss@iki.fi>

• Introduction
• Memory Allocations
  • The Old Ways
  • Data Stack
  • Memory Pools
  • Memory Pool API
• String Handling
  • The Old Ways
  • String API
  • String API with Memory Pool API Support
• Buffer Handling
• Real World Usage

Everyone knows about buffer overflows nowadays. Everyone knows some ways to prevent them. But most people still do it in a way that requires them to be unnecessarily careful while coding - even a single carelessly written part of code can be a security disaster.

None of these ideas I've written about is new, but most of them are very rarely used with C programs. I think it's mostly because many people don't know about them or just haven't realized how they could be easily used. Sure, there are also people who will never change their way of coding. And there's also the problem that using non-libc functions make the program bigger, which is pretty annoying with otherwise very small programs.

Besides not only making your code more resistant to buffer overflows, I think many of these features will actually make writing C-code a lot easier and more fun.

I'm not a writer and I'm not too good at english, so sorry about all the spelling and grammar errors :) All of this stuff was written at sunday morning, tired after being awake the whole night and not being able to do anything useful..

• Use static buffers - fast, but lack the ability to grow when needed.
• malloc() - slighly slower than static buffers, possible memory fragmentation, and most importantly it requires the memory to be freed, which can be sometimes very annoying. It's easy to forget to free the memory and cause memory leaks. free()ing already freed memory may also be an exploitable a security flaw.
• Garbage collector - this would be the best way to manage memory. For example OCaml's garbage collector is quite smart by treating long-living allocations differently than temporary allocations. However with C it's not really possible - you can't go moving the allocated memory elsewhere unless you write your program in special way. So there's a few simple non-portable garbage collector implementations for C, but they're not much more different from malloc()ing other than that you don't need to free() memory. They're not fool proof either.

What I haven't yet seen used anywhere outside my own software and some programming languages internals (eg. calling Perl code from C), is using data stack for temporary memory allocations. It has the most important advantage of garbage collectors; allocate memory without worrying about freeing it. It also has a few gotchas, but I'd say it's advantages are well worth it.

The way it works is simply letting the programmer define the stack frames. All memory allocated within the frame are freed at once when the frame ends. This works best with programs running in some event loop so you don't have to worry about the stack frames too much. Here's an example program:

Advantages over control stack:

• Functions can return a value allocated from data stack.
• We can portably specify how much data we want to allocate at runtime, and in some cases we can grow it without copying data.

Advantages over malloc():

• FAST, most of the time allocating memory means only updating a couple of pointers and integers. Freeing the memory all at once also is a fast operation.
• No need to free() each allocation. This has a huge impact on code simplicity. For example: fprintf(f, "'%s'\n", escape_string(input)); vs. char *str = escape_string(input); fprintf(f, "'%s'\n", str); free(str);
• No memory leaks.
• No memory fragmentation.

Disadvantages:

• Allocating memory inside loops can accidentally allocate a lot of memory if the loops are long and you forgot to place stack frame there.
• The memory could accidentally be saved somewhere and accessed after the end of stack frame.
• Debugging for buffer overflows can be difficult, since the regular bound checkers don't work. To kludge around this, I've made an compile time option to allocate all memory using malloc(), but this of course hurts the performance and grows memory usage. Of course, by coding good code you should never need this :)

The second disadvantage is the most problematic one. It's actually two problems; first it shouldn't be mixed with permanent data, and second it shouldn't be temporarily stored to location where it could be accessed outside the stack frame.

The first one is easier to handle. The ideal solution would be to have some tags in C language that would give warning if being lost, for example temporary char *str = t_strdup("str"); char *str2 = str; where the str2 assignment would give a compiler warning about the missing temporary tag. But since this isn't possible we can do almost as well using const keyword, which has exactly that behaviour but restricts us from modifying the memory. Luckily that's not usually needed.

The second one is much more difficult, I haven't found any good way to handle it other than by a) don't even try it, b) fill the freed memory when freeing stack frame so the corrupted value is easily noticed at runtime.

And why ever even do that? Because it's simpler in some situations. For example I have this message header parser, which calls a callback function for each header line. Suppose I only want to find the message's Content-Type, so save it with context->content_type = t_strdup(content_type); Then later I just read it without need to worry about freeing it. Now, the only problem with this is that it relies on the parser not to create a stack frame around the callback which would invalidate our saved return value. While writing the callback it's easy to check if it's possible, but what if the parser was later modified without remembering the special needs of the one callback..

Alloc-only pools can be quite useful for storing larger amount of return values from some function, especially when data stack can't be used (the second problem case above). They still provide relatively easy memory management since you only need to free the pool once.

Alloc-free pools could be useful with some applications to prevent memory fragmentation and memory leaks by grouping related data together into single malloc() block. They could also make the program run faster due to (possibly) better CPU cache utilization. Besides performance reasons, they could also be useful for statistics to find out where exactly the allocated memory goes. They'd however need their own internal malloc() implementation so they're not very simple.

When you have a function that could return a large return value, it's not very efficient to first store it into data stack and then copy it somewhere else. The traditional way would be to just give a (void *result, size_t size) as parameters. This again has the problem that we might not know at all what buffer size is large enough.

So, one way to deal with this is to give the function a memory pool object which can be used by the function to allocate the correct amount of data. Besides the dynamically created alloc-only and alloc-free pools there could be global pools as well: data_stack_pool and system_malloc_pool. Here's an example:

This is where most of the buffer overflows have happened. Nowadays people are more aware of the problem, but many still they do it in unnecessarily hard way.

• strncpy(), strncat(), snprintf() - only snprintf() of these is easy to use safely but it's still somewhat unportable (Windows). strncpy() doesn't necessarily NUL-terminate and many people misunderstand how strncat() works (ie. in very stupid and difficult to use way).
• strlcpy(), strlcat() - much better replacements to above by OpenBSD. Very unportable, but you can easily create your own ones. But these can still be used unsafely if the buffer size parameter is wrong or if the programmer goes playing around with the buffer indirectly, by eg. appending single characters and missing size checks (yes, I've seen this in software that contained "secure" in it's name).
• Dynamically allocating the amount of wanted memory and then using strcpy(), strcat(), sprintf() and direct accessing. This requires you to be very careful with the string size calculations. I don't understand why so many people think that's not a problem, they have this "If you can't calculate the sizes correctly, you're stupid and you shouldn't be coding at all" attitude. Why bother wasting time with that at all when you could be doing more important things?
• Dynamically growing buffers, used by for example GLIB, vsftpd, qmail, djbdns and Postfix. This is definitely the right way; string manipulation is done through API which discourages - or even disallows - direct buffer manipulation.

There's several slight variations how to implement the dynamically growing buffers. Most work by allocating a new string object, using it and then freeing it. qmail and djbdns uses statically created buffers which are reused and never freed - that's pretty dangerous unless you can be sure the buffer isn't in use anymore. It's actually pretty much the same as having static char bigbuf[8192] which is used by several functions.

There's two ways how to manage a buffer - first being the explicit way done by eg. djb:

GLIB supports doing this much more easily:

Or if you wanted a modifyable buffer:

Some people don't seem to like the printf-style or believe it's unsafe. GCC however gives very good warnings about parameters with incorrect type so I don't think there's any need to worry about.

Nothing fancy here really. Just by being able to specify the used memory pool we can easily allocate strings from data stack. I've made several wrapper functions so that instead of p_strdup_printf(data_stack_pool,...) I can just use t_strdup_printf(). Here's a list of few useful functions:

t_strdup_noconst is there mostly to avoid casting const away in those few situations where it's needed. Because it returns char * it could be more easily mixed with permanent data so it's usage should be kept minimal.

t_strconcat() is one function that I also copied from GLIB. It's a bit dangerous though, the terminating NULL is too easy to forget. I've been thinking about removing it entirely, but it's much more efficient than t_strdup_printf() so I haven't yet had the heart :)

Many people concentrate only on string related buffer overflows. Yes, they've been the most common ones but they're quite known now and there's more to buffer overflows than just them. The most obvious one is just another type of buffer handling. All the other data that couldn't be called strings. Often people just hack away separate memory allocations and/or size checks for them. The problems are exactly the same as with strings.

If we create a buffer API and write to buffers only through it, we would prevent almost all buffer overflows, even integer related ones, simply because the buffer API implementation is the only part of code that directly writes to memory, and that of course should be well audited not to overflow in any situation.

Buffer API would be somewhat similar to string API, ideally the string API should be created using the buffer API, or they could even be the same. My buffer API currently looks like this:

Then there are several write, append and truncate functions and a few functions to temporarily limit the accessible buffer range. An example:

There is of course also the possibility of buffer overflows while reading data. That's a bit more difficult to prevent except by careful coding, but usually it's not such a big deal anyway. It might crash the program, or at worst could expose some private information to attacker. If the later is a real problem, you should use multiple processes and IPC to hide the private data.

I've written a fully featured IMAP server using pretty much these techniques. The code still needs some cleanups and there's probably a couple of bugs left, but mostly I think it's not too bad :) The library functions in it are MIT licenced, so go ahead and rip them to your own programs. I could try to create some minimal library out of them if there's enough interest.

vsftpd uses it's own string API for pretty much everything. In only few of the files it's even possible to create a buffer overflow with the used coding style.

Well, probably too late for anyone to actually read this but lets try anyway.

I've been writing for the last few months a secure and fast IMAP server, named Dovecot. Suggestions and other feedback appreciated :)

Other projects seem to be creating all-in-one products which probably are easy to install and maybe to maintain, but much less powerful than the products that focus on just one thing. I have no plans on creating yet another useless SMTP daemon, Postfix and Qmail will do very well already. Of course, some people can merge those into some packages that are easy to install and administrate.