OpenLDAP: Replication has worked fine for me since 2.0 and up.
Qmail:
Tried it, loathe it. I won't go into it here for fear of DJB suing me...;)
MD4:
I did tests, asked around, general opinion was to use MD4 over MD5 if you're going to use the first 4 characters as a directory hash.
Netapp:
I won't sell you on them. I'll leave Netapp's salesmen to do that for you. I will say that any issues about redundancy or scalability are pretty much addressed.
WRT using FreeBSD/Linux NFS servers - I guess its the same reason why I don't buy a machine with many PCI-X slots and fill it with quad fast ethernet cards to build my own switch. Its just esaier (and more reliable) to get that component from a vendor thats done the work on it.
Hardware load balancers:
Same reason as above. They are cheap, they work, and they can do cool things like DSR which I'm pretty sure that the software load balancers can't do. DSR is pretty much a requirement for a high performance environment.
The servers are IBM x306s with 2GB RAM, 3.2Ghz CPUs and dual 80GB SATA drives, mirrored.
I think the 2GB ram and the fast back-end NFS is key here. Lots of cache.
I *think*, from memory, the Courier-IMAP indexes a lot of data now. At least, it leaves a bunch of cache looking files in the Maildirs. I have mailboxes with 1000s of messages and they don't seem to cause lots of NFS IO.
Check out the newer versions, maybe its addressed the issues you described.
Spamassassin is actually fine, you just have to tweak it a lot to get it running well on a busy system.
With regards to configuration of the MTA, thats where an experienced Exim admin helps. Or at least, someone who is able to read the Exim docs. Not a big task really.
I have 8 machines doing 500,000 messages a day, and their load is under 1 even during the busiest periods, using the above config.
I'm pretty certain it would scale out to 1,000,000 users just by adding more hardware. I don't think it would scale linearly, either; I saw a more than double increase in performance going from 4 to 8 machines. Though, I wouldn't expect that curve to continue all the way through to 1M.
1) So you don't have 50,000 domains sitting in one directory. Can get a little slow.
2) So if you need to have more than one NFS server, which you will if you have a large customer base, you can separate the base level into different NFS servers.
My job is building systems like this. Current mailserver system I designed and built is hosting 80,000 email accounts, and will scale out to a million quite cheaply by just adding more machines.
OpenLDAP
You need a central configuration repository to store the email accounts, their passwords, etc. OpenLDAP is perfect for this, and you can replicate it out for scalability. Be prepared to learn about LDAP schemas.
Exim
Use Exim because it has a simple process model (a single binary that does all the work, like sendmail) but has a human readable configuration file and has to be the most flexible MTA out there. You will have customers with weird requirements sometimes, and Exim will be able to meet those. Plus, it has Exiscan-ACL built-in these days, which allows you to do virus scanning and spam scanning at the DATA stage, before the mail is actually accepted by the MTA. It means you can make the sending MTA deal with the bounces if the mail is a virus or is obvious spam.
Courier-IMAP for POP3 and IMAP access.
Yeah its written by a sociopath, but nothing else works as good in the field. It works out of the box with sensible LDAP schemas and is fast, reliable and secure. Handles SSL, all the different authentication methods, what have you. Maildir compatible.
Maildir message store.
Store the mail in maildirs. Don't put them in/maildirs/domain.com/user/Maildir - split the domains up with a 2 level deep hashing algorithm (if you're virtual hosting domains, which is what it sounds like to me), so make it something like/maildirs/xx/xx/domain.com/user/Maildir, where xx/xx might be something like 3f/6b (depending on the hash). Use MD4 for the hash because its more balanced than MD5.
NFS mount the maildirs from a fast NFS device like a Netapp. Netapps are recommended because you can plug them in, and they just work, plus they are easy to scale by adding more trays.
Linux NFS servers set up with heartbeat and shared disk also make a nice HA NFS, and would be cost effective, but you'll have to buy an array anyway (probably fiber channel) so it might be better just get something thats completely integrated like the Netapp.
Spamassassin.
Can be configured to scan make at DATA time in the SMTP conversation. A LOT of configuration work here to make it play nice on a massively scaled platform, but it can be done. Mostly it needs to have things like the auto whitelisting and bayseasn filtering turned off, as the extra DB file work is a bit excessive.
Actually, I'm sure there is a way to make it work with a less resource intensive repository, but using the standard SA rules seems to work well for my environment. *shrug*
ClamAV.
Free antivirus, it works, and integrates well with Exiscan-ACL. Set it up to scan via the daemon, and configure it to update every couple of hours from cron, and bob's your uncle.
Scaling out
Make every box the same. Make every box an MTA, a POP3/IMAP server, etc. Use something like Kickstart to automate builds so that you can build a machine in 10 minutes, and all you have to do is configure the IP address and plug it in. If you want to be REALLY sexy, you could make the machines boot off the network, and mount / from a shared NFS area, and make/var/spool/exim the internal mirrored disks. DHCP them, then all you do is plug a machine in and set it to PXE boot. Pretty trivial to do.
Load balancing
Hardware load balancers are pretty much a necessity. Don't touch cisco stuff. Its not very good. Go with Foundry Networks ServerIrons. The XLs can handle 1 billion requests/day if you configure them in Direct Server Return mode (also known as DSR/Foundry switchback). Use it. It makes all the return traffic go directly out to the net, meaning your ServerIrons have to switch less traffic and track less sessions. I would recommend however for a million users a pair of the ServerIron 450GTs, or bigger. Maybe one per VIP/Service.
Now, if this is all looking pretty daunting, you could always hire me to build it for you:)
Is it just me or is the new SG1 and Atlantis shows really taking good shape? I mean, they have like, the hot chick from Andromeda, Crichton from Farscape AND the hotchick from Farscape... got rid of that plain jain Amanda Tapping!
HOT CHICKS GALORE!!!!1111one!!!
OK. Calm down:(
Next I'll start mentally masturbating over Boomer from BS: Galactica.
Now there's an army of clones that are allowed to abduct me! Rowr.
The problem is that most people work off the maximum wattage draw of all the components in their system, and add it up, and think "Ooh, I need a 900 watt power supply!".
Its complete bollocks.
A mate and I went to Akihabara to buy him a new PC. He had loads of money to burn on it, and burn he did. Dual core Athlon 64, 8 - yes, Eight, SATA-II 320GB drives, a raid card, 2 x GeForce 7800s (I think thats the model?), a SLI capable motherboard, etc etc...
And the guy came over and tried to sell him this really ugly loud monster PSU (700 watts) for it. We looked at it, and then at the 420 Watt power supply that had all the SATA power we needed, plus the power for the SLI, plus everything else.
It came with some software to see what the power draw is.
He set it all up. How much its drawing? Even when he is hammering the RAID5 volumes as hard as he can, he still only draws about 300watts.
Do we need 1KW PSUs? no. I don't think so. Not unless your machine has something like 30 drives in it, and good luck finding a case that fit that many.
This can't be right. Right now, I manage a system that does 500,000 messages/day running of 8 IBM 306 boxes with FC3 & Exim and a NFS backend. Its pretty small, most of the companies I know here in japan are 100 times bigger than us and must easily break 50M messages/day... docomo alone must do 500M/day...
And the idea that one would need commercial software to do this is laughable...
A jumphost is what someone else called 'SSH Bastion hosts'. Basically, the only box(es) that you allow direct SSH access to from the outside. You have to go through it to get to your other machines.
I recommend Kerberos simply because when you want to disable an account, its as simple as nuking it on the kerberos DC. If you have 200 machines, all with local authentication and RSA keys, you'll have to go through all 200 machines.
Its not ideal, of course. You still need some way of managing user accounts. But its a good fit for the auth side of things.
It IS a fucking nightmare to get working properly cross platform though.
Oh, and as an addendum to this, 99% of ALL the intrusions I see hitting the network right now are SSH dictionary attacks. At the very least, you should get rid of password auth, and go with RSA key auth. Carrying the key around on a USB keychain isn't such a bad idea, though not perfect. It should be considered a temporary solution until you've had time to implement a full blown security policy.
The biggest problem facing anyone looking at implementing an IDS into an existing system is the size of the network.
If you're doing 500mbit/sec+ of traffic, it requires a somewhat beefy snort box just to process that data let alone do something about anything that looks like an attack.
Snort CAN do it, it just takes a lot of effort to pair down the ruleset to the point where it can handle your traffic. But, pairing down the ruleset has some drawback...:)
Or, if you can segregate your network, that can help a lot too. But unfortunately, a lot of networks suffer from a lack of design and you end up with huge VLANs that span thousands of hosts, and other nightmares.
IMHO If you're worried about intrusion, start with host security. If you have a huge farm of linux boxes, then great. Use iptables and keep everything up to date. If you MUST have sun boxes, try not to put them on the edge of your network - NAT specific ports via linux NAT firewalls. Same goes for windows machines. Don't bare them to the internet for any reason.
Have some aggressive ACLs on your border routers. Don't allow SSH into all your machines directly. Use jumphosts. Consider using token based authentication, like SecurID. Consider Kerberos to replace the use of public key auth in your ssh infrastructure.
once you have that down, putting in an IDS can wait:)
But couldn't I just as easily say "There are many, many good original news feeds out there now, people who want to have original news WILL USE AN ORIGINAL NEWS FEED. Slashdot should be news aggregator."
No, not really.
The whole point about/.'s news submission system is that there are thousands of people every day submitting stories. Of those thousands, 15 or 20 get picked. My main complaint is that rather than picking stories that have had thought put into them, the editors choose stories that are just dupes of old stories or, worse, clickfests from other news sites. And I'm sick of needing to use bugmenot to log into the NYT.
The point you hit on there about the discussions. Yes. Thats one good point./. discussions invariably have one or two incredibly interesting posts that are relevant and contribute something to the whole. Its worth scanning through and looking for the more interesting comments at times.
I guess this is a problem with age. As you get older you look on the past with rose tinted glasses and think "back when I was a lad,/. was actually cool and relevant! Now its just a sycophantic clickfest of dupes!". *sigh*.
I use a mac for work every day, but really this shit is getting ridiculous.
Its just not... news yet! Its not even been confirmed, or announced. Its just rumours! So why feed the rumour mill?
The things that are beginning to make me stop coming here, after what, ever since/. started?
The dupes. C'mon. If you, as an editor, can't be bothered to review the submissions properly (remember, there is only 10 or so stories a day!!!) then you, the editor, should fucking resign and let someone else who is more capable and, better yet, INTERESTED in doing the job properly.
The sycophantic Apple loving. I like Apple kit - as I said, I use a powerbook every day - but stop stop stop stop stop reporting on every little bloody thing they do. Wait until there is actually NEWS. The whole rumour on the shift to intel had how many stories
The quality of the stories. At the top of the page it says, "News for nerds. Stuff that matters.". Not "News linked to from The Register" or "News linked to from Anandtech" or "News aggregator for anime stuff". There are many, many good news aggregators out there now, people who want to have aggregated news feeds WILL USE A NEWS AGGREGATOR. Slashdot should be original news. Stuff that everyone else isn't reporting on. The stories on the ethanol (while a dupe from a couple of weeks back, and nothing had changed since then) were interesting, for example, and provoked really quite interesting discussion. But no, we have to see more "Apple suck!" comments because someone has helped feed Apple hypemachine yet again.
I've had enough. I don't know why I keep coming back, but I think it's because sometimes, once in 20 or 30 stories posted, there is actually something interesting there that someone has gone out of their way to research and follow through on.
Slashdot Mirror
Its for performance reasons, not security.
OpenLDAP: Replication has worked fine for me since 2.0 and up.
... ;)
Qmail:
Tried it, loathe it. I won't go into it here for fear of DJB suing me
MD4:
I did tests, asked around, general opinion was to use MD4 over MD5 if you're going to use the first 4 characters as a directory hash.
Netapp:
I won't sell you on them. I'll leave Netapp's salesmen to do that for you. I will say that any issues about redundancy or scalability are pretty much addressed.
WRT using FreeBSD/Linux NFS servers - I guess its the same reason why I don't buy a machine with many PCI-X slots and fill it with quad fast ethernet cards to build my own switch. Its just esaier (and more reliable) to get that component from a vendor thats done the work on it.
Hardware load balancers:
Same reason as above. They are cheap, they work, and they can do cool things like DSR which I'm pretty sure that the software load balancers can't do. DSR is pretty much a requirement for a high performance environment.
Courier-IMAP seems fine here.
The servers are IBM x306s with 2GB RAM, 3.2Ghz CPUs and dual 80GB SATA drives, mirrored.
I think the 2GB ram and the fast back-end NFS is key here. Lots of cache.
I *think*, from memory, the Courier-IMAP indexes a lot of data now. At least, it leaves a bunch of cache looking files in the Maildirs. I have mailboxes with 1000s of messages and they don't seem to cause lots of NFS IO.
Check out the newer versions, maybe its addressed the issues you described.
In my environment, its never been an issue.
Spamassassin is actually fine, you just have to tweak it a lot to get it running well on a busy system.
With regards to configuration of the MTA, thats where an experienced Exim admin helps. Or at least, someone who is able to read the Exim docs. Not a big task really.
I have 8 machines doing 500,000 messages a day, and their load is under 1 even during the busiest periods, using the above config.
I'm pretty certain it would scale out to 1,000,000 users just by adding more hardware. I don't think it would scale linearly, either; I saw a more than double increase in performance going from 4 to 8 machines. Though, I wouldn't expect that curve to continue all the way through to 1M.
Couple of reasons,
1) So you don't have 50,000 domains sitting in one directory. Can get a little slow.
2) So if you need to have more than one NFS server, which you will if you have a large customer base, you can separate the base level into different NFS servers.
My job is building systems like this. Current mailserver system I designed and built is hosting 80,000 email accounts, and will scale out to a million quite cheaply by just adding more machines.
/maildirs/domain.com/user/Maildir - split the domains up with a 2 level deep hashing algorithm (if you're virtual hosting domains, which is what it sounds like to me), so make it something like /maildirs/xx/xx/domain.com/user/Maildir, where xx/xx might be something like 3f/6b (depending on the hash). Use MD4 for the hash because its more balanced than MD5.
/var/spool/exim the internal mirrored disks. DHCP them, then all you do is plug a machine in and set it to PXE boot. Pretty trivial to do.
:)
OpenLDAP
You need a central configuration repository to store the email accounts, their passwords, etc. OpenLDAP is perfect for this, and you can replicate it out for scalability. Be prepared to learn about LDAP schemas.
Exim
Use Exim because it has a simple process model (a single binary that does all the work, like sendmail) but has a human readable configuration file and has to be the most flexible MTA out there. You will have customers with weird requirements sometimes, and Exim will be able to meet those. Plus, it has Exiscan-ACL built-in these days, which allows you to do virus scanning and spam scanning at the DATA stage, before the mail is actually accepted by the MTA. It means you can make the sending MTA deal with the bounces if the mail is a virus or is obvious spam.
Courier-IMAP for POP3 and IMAP access.
Yeah its written by a sociopath, but nothing else works as good in the field. It works out of the box with sensible LDAP schemas and is fast, reliable and secure. Handles SSL, all the different authentication methods, what have you. Maildir compatible.
Maildir message store.
Store the mail in maildirs. Don't put them in
NFS mount the maildirs from a fast NFS device like a Netapp. Netapps are recommended because you can plug them in, and they just work, plus they are easy to scale by adding more trays.
Linux NFS servers set up with heartbeat and shared disk also make a nice HA NFS, and would be cost effective, but you'll have to buy an array anyway (probably fiber channel) so it might be better just get something thats completely integrated like the Netapp.
Spamassassin.
Can be configured to scan make at DATA time in the SMTP conversation. A LOT of configuration work here to make it play nice on a massively scaled platform, but it can be done. Mostly it needs to have things like the auto whitelisting and bayseasn filtering turned off, as the extra DB file work is a bit excessive.
Actually, I'm sure there is a way to make it work with a less resource intensive repository, but using the standard SA rules seems to work well for my environment. *shrug*
ClamAV.
Free antivirus, it works, and integrates well with Exiscan-ACL. Set it up to scan via the daemon, and configure it to update every couple of hours from cron, and bob's your uncle.
Scaling out
Make every box the same. Make every box an MTA, a POP3/IMAP server, etc. Use something like Kickstart to automate builds so that you can build a machine in 10 minutes, and all you have to do is configure the IP address and plug it in. If you want to be REALLY sexy, you could make the machines boot off the network, and mount / from a shared NFS area, and make
Load balancing
Hardware load balancers are pretty much a necessity. Don't touch cisco stuff. Its not very good. Go with Foundry Networks ServerIrons. The XLs can handle 1 billion requests/day if you configure them in Direct Server Return mode (also known as DSR/Foundry switchback). Use it. It makes all the return traffic go directly out to the net, meaning your ServerIrons have to switch less traffic and track less sessions. I would recommend however for a million users a pair of the ServerIron 450GTs, or bigger. Maybe one per VIP/Service.
Now, if this is all looking pretty daunting, you could always hire me to build it for you
I had a good chuckle when a friend spotted these:
http://www.stopurban4x4s.org.uk/shop.htm#parking
Thank YOU for actually doing some research! Good man.
You know your config is similar to mine. I was wondering about the Reserators. Very tempted to buy one. Worth it?
BWAHA! TROLLED!! Hee hee! ;)
Actually, she's cool.
Yeah, I know. :(
:)
Still, show me an adult, and I'll show you someone who still behaves childishly even when they're old and crusty, like me.
Also, I hope "growing up" doesn't mean I'll stop being interested in hot chicks. That would suck..
Is it just me or is the new SG1 and Atlantis shows really taking good shape? I mean, they have like, the hot chick from Andromeda, Crichton from Farscape AND the hotchick from Farscape ... got rid of that plain jain Amanda Tapping!
:(
HOT CHICKS GALORE!!!!1111one!!!
OK. Calm down
Next I'll start mentally masturbating over Boomer from BS: Galactica.
Now there's an army of clones that are allowed to abduct me! Rowr.
No, there is a peak power output and the rated output. The peak power output is for when you turn the system on.
Look, the guy built it, and is using it with no problems. *shrug*
I think these 1KW power supplies are all pns envy like another poster said.
Well, maybe.
:)
The guy isn't having any problems *shrug*. I guess we'll know if it explodes!
The problem is that most people work off the maximum wattage draw of all the components in their system, and add it up, and think "Ooh, I need a 900 watt power supply!".
Its complete bollocks.
A mate and I went to Akihabara to buy him a new PC. He had loads of money to burn on it, and burn he did. Dual core Athlon 64, 8 - yes, Eight, SATA-II 320GB drives, a raid card, 2 x GeForce 7800s (I think thats the model?), a SLI capable motherboard, etc etc...
And the guy came over and tried to sell him this really ugly loud monster PSU (700 watts) for it. We looked at it, and then at the 420 Watt power supply that had all the SATA power we needed, plus the power for the SLI, plus everything else.
It came with some software to see what the power draw is.
He set it all up. How much its drawing? Even when he is hammering the RAID5 volumes as hard as he can, he still only draws about 300watts.
Do we need 1KW PSUs? no. I don't think so. Not unless your machine has something like 30 drives in it, and good luck finding a case that fit that many.
This can't be right. Right now, I manage a system that does 500,000 messages/day running of 8 IBM 306 boxes with FC3 & Exim and a NFS backend. Its pretty small, most of the companies I know here in japan are 100 times bigger than us and must easily break 50M messages/day ... docomo alone must do 500M/day ...
...
And the idea that one would need commercial software to do this is laughable
this just makes me go HUH!?
I mean, what kind of demented, bored, halfwit lawyer decides it would be a good idea to SUE a guy making FURNATURE out of PACKING MATERIALS?!
I mean, COME ON! Give the guy a break!
Hey, that guy is so poor, he obviously needs more problems, so lets slap a lawsuit onto him! Yeah!
Great idea!
Bastards.
A jumphost is what someone else called 'SSH Bastion hosts'. Basically, the only box(es) that you allow direct SSH access to from the outside. You have to go through it to get to your other machines.
I recommend Kerberos simply because when you want to disable an account, its as simple as nuking it on the kerberos DC. If you have 200 machines, all with local authentication and RSA keys, you'll have to go through all 200 machines.
Its not ideal, of course. You still need some way of managing user accounts. But its a good fit for the auth side of things.
It IS a fucking nightmare to get working properly cross platform though.
Oh, and as an addendum to this, 99% of ALL the intrusions I see hitting the network right now are SSH dictionary attacks. At the very least, you should get rid of password auth, and go with RSA key auth. Carrying the key around on a USB keychain isn't such a bad idea, though not perfect. It should be considered a temporary solution until you've had time to implement a full blown security policy.
The biggest problem facing anyone looking at implementing an IDS into an existing system is the size of the network.
... :)
:)
If you're doing 500mbit/sec+ of traffic, it requires a somewhat beefy snort box just to process that data let alone do something about anything that looks like an attack.
Snort CAN do it, it just takes a lot of effort to pair down the ruleset to the point where it can handle your traffic. But, pairing down the ruleset has some drawback
Or, if you can segregate your network, that can help a lot too. But unfortunately, a lot of networks suffer from a lack of design and you end up with huge VLANs that span thousands of hosts, and other nightmares.
IMHO If you're worried about intrusion, start with host security. If you have a huge farm of linux boxes, then great. Use iptables and keep everything up to date. If you MUST have sun boxes, try not to put them on the edge of your network - NAT specific ports via linux NAT firewalls. Same goes for windows machines. Don't bare them to the internet for any reason.
Have some aggressive ACLs on your border routers. Don't allow SSH into all your machines directly. Use jumphosts. Consider using token based authentication, like SecurID. Consider Kerberos to replace the use of public key auth in your ssh infrastructure.
once you have that down, putting in an IDS can wait
Well, I agree with you there.
:)
Like I said in a previous comment, I think my main issue here is looking back at things with rose tinted glasses.
Still, there is room for improvement. A site redesign wouldn't bloody hurt either. Gah!
No, not really.
The whole point about /.'s news submission system is that there are thousands of people every day submitting stories. Of those thousands, 15 or 20 get picked. My main complaint is that rather than picking stories that have had thought put into them, the editors choose stories that are just dupes of old stories or, worse, clickfests from other news sites. And I'm sick of needing to use bugmenot to log into the NYT.
The point you hit on there about the discussions. Yes. Thats one good point. /. discussions invariably have one or two incredibly interesting posts that are relevant and contribute something to the whole. Its worth scanning through and looking for the more interesting comments at times.
I guess this is a problem with age. As you get older you look on the past with rose tinted glasses and think "back when I was a lad, /. was actually cool and relevant! Now its just a sycophantic clickfest of dupes!". *sigh*.
Maybe that one story in 10 or 20 is why I keep coming back.
I just guess I'm expecting too much from the editors to get that signal to noise ratio modified upwards a little :)
hahaha :)
:)
/. these days, I keep coming back for punishment ... unfortunately its a habit now.
No
As much as I'm hating
I use a mac for work every day, but really this shit is getting ridiculous.
Its just not ... news yet! Its not even been confirmed, or announced. Its just rumours! So why feed the rumour mill?
The things that are beginning to make me stop coming here, after what, ever since /. started?
I've had enough. I don't know why I keep coming back, but I think it's because sometimes, once in 20 or 30 stories posted, there is actually something interesting there that someone has gone out of their way to research and follow through on.
I know nothing of this airgapping of which you speak... :)
Don't forget the 'might'.
Like I said, I'm not convinced.