Slashdot Mirror
What Goes into an Enterprise Network?
Komi asks: "I work for a big semiconductor company, and I'm part of a group that is spear heading the Linux movement here. Right now everyone uses Sun machines to design, but you can get a cheaper Linux x86 machine that is four times faster. So it is my job to prove that Linux works. The problem is that I'm an analog circuit designer stuck in the role of sysadmin. So I need some advice on what goes into a network. It won't be that large right now, but it has to be scalable for up to a couple of hundred machines. If this works, then hopefully we'll convince all designers at my company to make the switch."
"Here's the hardware that I am planning on getting:
- 2 servers:
These would hold the home accounts and tools, as well as serve out NIS, NTP, etc. I know I'll need a lot of hard drive space (2x72GB SCSI each), but do I need a lot of memory? (It's 4GB RDRAM max.) Should the processor be fast, or dual?
-
3 batch machines:
These would be a small compute farm running LFS or something. Jobs would get queued up and run continuously. So these should be dual CPU with lots of memory, probably 4GB each. Any other particular details?
- 10 desktop machines:
These would be on the designers and developers desktops. These should be reasonably fast (~2GHz) single CPU machines with probably need at least 2 GB RAM. The simulations we run do not benefit from dual CPUs. They probably don't even need SCSI. I'm thinking a $2k PC should work.
- 1 Itanium server:
This would be to play around on to test our 64-bit applications. The only advantage of 64-bit is applications using huge amounts of data.
For a network with centralized file stores you will want some sort of automated backup system. Probably an LTO tape drive/autoloader.
Obviously your ROI will be based upon two metrics:
1 > Time savings versus average hourly rates for computing & employee time costs. This would be an agressive ROI metric.
2 > A more conservative metric would be the cost of replacements of Sun systems over time versus costs of , say, a small farm of Dell Optiplex PCs.
You could then also compute the value of gigaflops per dollar, showing the clear advantage of the PCs.
-- We live in a world where lemonade is artificial and soap has real lemon.
So I need some advice on what goes into a network. It won't be that large right now, but it has to be scalable for up to a couple of hundred machines.
...you can get a cheaper Linux x86 machine that is four times faster.
1) You had better find some damn fine PCs to replace those Suns, because a couple hundred PCs can make your life miserable due to lots of random breakage.
2) This is not true (unless you found Pentiums with SPECfp of over 3000!). If you buy the right-sized computers for your task, the hardware costs won't be a dominating part of your budget. Human costs and non-OS commercial licensing will be, regardless of your platform choice.
Whenever people say that Linux is absolutely outright cheaper then commercial UNIX, then I'm pretty convinced they haven't figured out all the costs involved. Also, I'm not convinced they understand just how simple maintaining a Solaris box can be, for example, due to sunsolve.sun.com, ample documentation, optional support out the wazoo, etc.
Before you go blazing these new trails, just stop and think for a minute. Put aside the zealotry and really think hard about what is and is not cost effective. Regardless of your choice, you really need to be convinced it is the right one.
Healthcare article at Kuro5hin
Agreed.
Hire a good sysadmin and the job will get done much better and faster.
Itanium also doesn't sound like the way to go for you.
Think about it. Hammer is going to be out in a few weeks which will give you better 64-bit & 32-bit performance than the itanium for a fraction of the price. It's not the solution for everyone, but for you.. it sounds like it.
Now about redhat... You could consider other distros as well like Gentoo which give you added benefits like better package management especially if you are going to have a lot of your own source around. You can write ebuilds for them and easily install your source packages on all your machines. It could also give you a nice performance benefit.
But the distro might best be picked by the sysadmin you hire. (who needs to be a specialist in linux, but maybe not already tied down to a specific distribution)
Liberty.
It's a big troll, sure. However it is also a chance to dispense some good advice:
1) There's a difference between PC's and 'Server Class' hardware. The biggest is testing. It will work, and it can be supported easily. Drivers are nice and available (generally speaking). Usually dual proc, usually RAID enabled. You can use RAID to speed up read access, but almost no one does. They use it for redundancy (in case a disk flakes out on you). How much money you spend depends on how much downtime costs. If it really costs, you need RAID 1+0 or 0+1. Go with hardware based RAID if you can.
2) Sun hardware. There are many more advantages to sun hardware than what's obvious. Never over look what a good support organization can do. You pay for it, but if something fried, I can have a part in my hands 4 hours later. Sun's low end desktop's are nothing to write home about. However, if you've got Ultra60's or SunBlade 1000 or 2000's, that's some really class hardware. You can do some supprising things with it. [1]
3) Dual procs. On Desktops, even if your simulations don't benifit from dual proc, if they take a while, and they eat that 1 CPU, you'll be happy to have a second (web browsing, etc). On your servers, it's effectivly a must.
4) RAM. On the servers, crank it. On the desktops, you should probably crank it.
5) Cost. If your work is anything like mine, you have 'capital' money, and 'O and M' money. When in doubt, over spec the machines, so that your less likely to have to request more money from the 'capital' pool than you initially quoted. "Going back to the well" viewed poorly.
6) NIS. NIS is evil and the plague. If your in a relativly local office with good connectivity, it's alright. If you try to spread it over WAN links, you're going to get hurt at some point.
7) NTP? Why run a seperate server when you don't have too. Leverage what's already in use in the company. This leads to my last point (and what was the best point of the parent)
8) Get yourself a real sysadmin. These are decisions that s/he is experienced in, and paid to do. Your trial by fire that would come from this will probably drive you insane. Good sysadmins are a rareish breed. I know, I am one. There are a fair number of good ones out of work now. Find one.
[1] The reason largely has to do with cache. Sun chips made in the last 2ish years have 8 MEGS of cache on them (that's even mirrored so it's 16 in total, but you can only use 8). We built a GIS app, and field tested it on Sun and Intel hardware. The intel hardware could deal with 1 to 4 users with less resources than the sun box could. However, the the sun box kept growing up to several hundered users, while the intel box started thrashing hard after 10 or so. We compared a dual US3 box to a dual Xeon P4.
Zapman
What I would concentrate in is:
Just a quick overview, to sum it up I would second the advice somebody else gave you in a previous posting: hire a decent sysadmin and plan things with him.
Sorry for the horrible formatting, slashdot forces me to hit a certain line count, and I am tired of messing with coding HTML just to post to a damn msg board .
...
.
.
...They will have RAID5 with complementary back ups of each server.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
...well
yeah sure, but for the cost of one 4 processor
Netra 1400t with 16 gig of RAM
.
.
.
They need to get with the future
I have had some killer boxen I have built that have worked well for years and have passed on thru hands of other ppl.
PC hardware like one poster pointed out is cheap and is gonna break Make up several extra PC's ready to go with a "image" if identical hardware is used
Keep several ready to go and working in a storage closet out of site and keep their
existence little known or else they will get appropriated just because ppl "feel" they need an extra boxen
Don't tell anyone either they will slip and tell someone and then they will never stop pestering
you til you have no extra boxen .
They will even stoop to calling in favors of ppl in authority to try to scrounge them an extra boxen . They are snakes !!!
Users should keep all their files on the servers, BECAUSE
If one server catches on fire, the other is backing it up during "low load" times, or at pre-scheduled cron times
Monitor load usage of network and servers, plan back ups and other simlar tasks off peak.
IDE based raid is now cheap and reliable and you can get awesome amounts of storage for reasonable money
Ex.: 12 channel IDE Raid 5 controller with 12 - 120 gig drives pushing 1.4 Tera prior to losing 33% due to overhead of parity .
Keep several extra IDE drives laying around, use all the same size and order them in bulk factory direct if you can
Hot swap trays are essential, read reviews and get the best RAID
Alot of ppl on slashdot have used 3ware and promise, Adaptec is always damn good too .
Ex: order several cases of drives from the manufacturer . In IDE stay away from Seagate, and Maxtor drives that were Quantum's .
Alot of ppl I know generally like Western Digital, IBM, and the better Maxtors
Again read reviews online, learn to form your own opinion . learn from the pain of others, serach news groups for model#'s you are considering buying
Never buy the newest, just got on the shelf products, alot of the time they are buggy and need BIOS updates.
I know I just bought one.
Tried and true is what should go in a server. If it is not the pillar of praise, you do not want it in your server .
If you want to be 100% sure, go with SCSI, but be prepared to pay hideous amounts of money for equal storage
Set the 2 Raid 5 arrays to snapshot each other every day , and you can restore a backup in minutes this way or incrementally .
The sheer volume of volume will let you do these monster backups, cheaply, and quickly if you use 64 bit controllers, and 64 bit PCI slots
Dual Xeon's for the Servers is most likely best . As for waiting for AMD's hammer,
that is postponed damned near indefinitely, I have heard 3rd or 4th quarter
When I worked for cisco this is how they did it, and they snapshotted the desktops too .
The servers, build to the teeth, MAX RAM, Dual or Quad Ethernet NIC's . Then bond the NIC ports as needed , load balance as needed . Set up some basic SNMP package with an e-mailer to let you know when boxen are burping
Careful not to over do it on the SNMP it can burden your servers or your network, just the essentail info, the books will clue you in on this
Don't bother with the expense of RDR RAM , go DDR, use the extra money to buy more of it.
Hell use the extra money for an extra server .
Fast RDR costs almost triple what DDR does, and RDR only outperforms in select apps .
price compare here : http://www.pricewatch.com
I'd recommend a top of the line Ethernet switch, after all what good is your servers if the network is crap
Consider fiber GBIC's from the servers to to a blade on a nice cisco switch
Giga-bit ethernet over fiber is a beautiful thing to behold
Consider a Giga-bit link from server to server to the backups so they do not load the network .
You can just use a crossover cable if you you use Giga-bit over copper
Cisco is expensive as hell, but they are good . Juniper and Extreme are good too as long as you are just running one protocol and not trying to make a hybrid multi-protocol network.
The "Hire a real sysadmin" statement is true, unless you are one to like new HUGE challenges .
If you are stuck with this, you need to do ALOT of reading, O'reilly has some good books, but there are others you will need as well .
Don't skimp here, read the highly recommended Unix Bible and any books it recommends .
Unix Admin's guide too, but these alone will not be enough .
You are about to read several thousands of pages of material, you might point that out to the ppl that dumped this on you .
Software for the servers, I'd do alot of research, I have no recommendations, I am a hardware guy . Linux of course, I am partial to Redhat
As for Sun boxes beating x86 boxes
you can build many x86 boxes and use somthing like a beowulf cluster or www.mosix.com .
When it comes down to $$$'s, x86 is gonna win, if you want support, and someone to hold your hand and be there 24x7x365 go sun .
Sun support, parts, and just about you name it is mucho deniro . I think if you get your learn on, you can better spend the money elsewhere.
The learning curve on this is going to look like the combined eliptical orbits of every planetary body in our galaxy
Network Security ??? Call in a well known expert and have them set up a plan , follow it religiously or get hacked.
Security is almost becoming a science unto itself , a good firewall, well setup and maintained.
IP access lists in your Cisco, or other managed Router or layer 3 switch
Oh, and if your religious, you might pray .
If you have any specific questions just e-mail me at my addy on the webpage below
If I do not know it, the *nix wizards that taught me will for sure . I am still learning myself, but if your a REAL IT person you always will be.
Peace...
Ex-MislTech
http://www.geocities.com/duanenavarre
google "32 trillion offshore needs IRS attention"
Sysadmins will be hired for this once money gets freed up and we can prove to groups that linux works. A later post was correct that there are really two issues. a) Getting everyone to switch to linux, and b) getting designers to put linux on their desktop. We really only care about b), but by the nature of the deal, we have to prove a) and b). Also we don't care about the cost of switching a design group over to linux either. That's someone else's job. We just show that the end result works.
And finally, we do need 64-bit machines. Some of the programs we run use huge ammounts of data that need 64-bit to address them. So if we're getting free loaner equipment, then why not play with an Itainium? :)
I appreciate the advice from everyone.
Thanks,
Komi
The ultimate goal of science is to unify all forces of nature to a single law that can be silk-screened onto a T-shirt.
Er, every commonly-used RAID level will speed up read access...
If it really costs, you need RAID 1+0 or 0+1.
You want RAID 10 (1+0, stripes over mirrors), not 0+1.
There are many more advantages to sun hardware than what's obvious. Never over look what a good support organization can do. You pay for it, but if something fried, I can have a part in my hands 4 hours later.
Dell can have a part in my hands in two hours - often with an on-site tech included. There are certainly reasons to buy Sun hardware, but hardware support isn't a major one.
However, the the sun box kept growing up to several hundered users, while the intel box started thrashing hard after 10 or so. We compared a dual US3 box to a dual Xeon P4.
And how did the intel boxes go when you spent the same amount of money on them (purchasing multiple machines and scaling horizontally) as you had spent on the Sun box(es) ? Or didn't your app scale horizontally ? (that *would* be a good reason to buy Sun - or some other lots-of-CPUs-in-a-machine). Similarly, if you have an app that really benefits from massive amounts of L2/L3 cache, then machines that have CPUs with lots of cache might give disproprtionately better results. "It all depends".
The idea with going intel over Sun is to use the enormous price difference to buy lots of intel machines and cluster them. This works only if your application can efficiently scale across multiple machines and doesn't benefit disproprtionately from things that simply aren't available on intel - like massive amounts of fast cache memory.