What Setup Is Best For Large Mailing Lists?

Super_Frosty asks: "I run a large mailing list that has expanded beyond my means. Perhaps you have heard of it - its headquarters is at this site. I currently have about 10,000 subscribers, which I am running from a small group of sad Macintoshes. I can hardly send out the daily message anymore. Anyhow, I can't find anybody who knows how to administer a list this large. I am beginning to have embarassingly close calls with people flooding and spamming the list due to security problems. My hardware can't handle it, and most software packages aren't meant for it, either. Can anybody reccommend a hardware and software solution for this list? Surely, some Slashdotter has administered a large list - I hope!"

Comments

qmail + ezmlm

[Matts]

The only mailing lists I'm on that have near instant response time are running ezmlm with qmail (including some large lists like the mod_perl list and the perl5-porters list). I can't recommend this setup enough, I use it for a very low traffic mailing list for AxKit, but besides the low traffic, administration, albeit by command line utilities, is a breeze (there are web admin tools available, and admin by email is possible too).

Every other list I'm on, run on a variety of setups (the most common appears to be sendmail + majordomo), has really poor latency of around a couple of hours. For a daily email that might not be so bad, except to realise that the latency transfers directly into list delivery speed.

There's not much more to say about it. Try it - I never regret it.

Don't really need list software?

[ldspartan]

From the look of the URL you gave us, it seems that this isn't a real list like most people are thinking... It looks like a daily message from one source sent out to a bunch of people, without any feedback/discussion from the subscribers. If this is true, qmail alone can handle it, using some custom perl and qmail's dot-qmail file format. Using this method, the distribution can easily be split amongst several servers.

If the subscribers do need to have discussions, then I would suggest qmail + ezmlm. If I recall, it too allows lists to be split amongst many locations, albeit with more trouble.

Now, none of these suggestions are going to help if the problem is bandwith, or anything other than server speed/power/etc. What exactly seems to be the bottleneck?

Re:Don't really need list software?

[anticypher]

My recommendations follow most of the others. Get Majordomo for handling the list, and any decent MTA like sendmail or qmail for the actual sending. Make sure your machine is fairly reliable, and keep logs of everything you do. If you have problems, you probably are better off to not send out a joke one day than to send 2 or more jokes in a day trying to fix the problem.

Since you are rightly worried about security, I'd recommend going out and reading up on everything you can. bugtraq archives, dejanews, risks archive are a few of the places to start. You've obviously put some time and effort to grow a lame joke list to 10k subscribers, so put some effort into protecting it.

I noticed your emails coming into my /. only account starting soon after you posted this question. Did you notice someone loading up your mailing list with the complete user list scammed from /. ? Are you planning on cleaning up your list, or are you going to wait for each person to go clean it out themselves?

the AC

Hardware isn't that hard

[randombit]

As long as you choose reasonably decent hardware and a good OS (FreeBSD or Linux seem like good choices for this), you should be OK. Lots of memory (128M+) would probably be useful, though.

Mailman has a great user and admin interface (all web based, very intuitive to use), but it gets kind of hard to admin once the mailing list reaches any size at all (it's kind of a pain to work with only 150 members on one list I admin). I don't know much about MTAs; sendmail seems to work OK but I've heard that postfix or qmail are faster. YMMV.

Postfix + mailman

[draziw]

Postfix for mail and Mailman for the list software.

Sendmail + Majordomo

[silvwolf]

I run a discussion list with about 350 subscribers (225 in 'direct mail', another 125 in digest) that gets about 100 messages from subscribers per day. It runs on sendmail & majordomo on an AMD K6-233 w/ 64MB RAM running RedHat 6.1.

Neither myself, nor the guy who actually owns the machine, are sendmail experts, so our problems stem from sendmail taking so long to deliver messages to places having trouble recieving mail. Creates a small backlog, but messages are delivered to subscribers in under an hour usually. Subscription requests and admin things by email usually take under a minute to get confirmation.

We also keep archives using MHonArc and glimpse, searching is a little slow, but that's to be expected considering the machine speed, and tens of thousands of messages.

It really isn't a bad setup, considering all the software is free, the machine was sitting in the basement collecting dust, and it's run off a static IP cable modem.

qmail + ezmlm

[TheLink]

qmail should be able to take your load.

When I last checked, Hotmail and Yahoo mail used qmail. I doubt your load is more than theirs.

For discussion style mail lists you can use ezmlm. But if it's just one way, one to many broadcast, just qmail will do.

However if the problem is bandwidth, then you need to either get a larger pipe, OR find someone willing to redistribute the mail for you- e.g. you send them a single email with tons of addresses and they handle the delivery. Do NOT use someone's mail server for that without their permission.

Good luck,

Link.

If I were you....

[Tony Shepps]

I'd stop worrying about the hardware and software to run something like this, and hire a firm to worry about it for you.

Say for example Listbox or there's always Egroups. My own company's mailing list services aren't really ready for prime-time yet, but I do operate the announcements-only mail list for the Philadelphia Eagles, and that is about 11000 subscribers although it is one-way only.
--