Slashdot Mirror
Rugged Linux Server For Rural, Tropical Environment?
travalas writes "Last year I moved to Rural Bangladesh. My work is pretty diverse, everything from hacking web apps to designing building materials. Increasingly a Linux VM on my MacBook Pro is insufficient due to storage speed/processing constraints and the desire to interface more easily with some sensor packages.
There are a few issues that make that make a standard server less than desirable. This server will generally not be running with any sort of climate control and it may need to move to different locations so would also be helpful if it was somewhat portable. The environment here is hot, humid and dusty and brutal on technology and power is very inconsistent so it will often be on a combination of Interruptible Power Supply and solar power. So a UPS is a must and low power consumption desirable, so it strikes me that an Integrated UPS a la Google's servers would be handy. Spec wise it needs to be it needs to be able to handle several VM's and some other processor storage intensive tasks. So 4 cores, 8GB of ram and 3-4 TB of SATA storage seems like a place to start for processing specs. What sort of hardware would you recommend without breaking the bank?"
Yeah, not breaking the bank isn't going to be an option here, I'm afraid.
Ever heard the expression "Fast, reliable, cheap (pick two)"?
It applies here.
Of course, you were fairly specific with the processing specs you need, but not your budget. So it's hard to say what "breaking the bank" is for you. also, you called it a UPS, but you also called it an "Interruptible Power Supply". I'm assuming a brain-fart, but the "U" stands for Uninterruptible.
Just picking nits.
Touch everywhere, even when inappropriate.
Low power consumption, quad core, 8 gigs of ram, a UPS and a few TB of storage? 1. Not gonna be cheap, though I'm not sure what your budget is, this will be somewhat pricey. 2. You might want to get a few UPS's, because I doubt, unless you get a very large solar array, that you will be able to run it on that. Expect power loss, disable write caching on the disks, etc. Also, a UPS isn't meant to be used as a constant power source, just as a way to keep you from losing work if power goes out, and if you're lucky, hold you over till it flickers back on. 3. This will NOT be portable, those UPS's will be a pain to move. Good luck, I certainly don't mean to be so negative, but this is a somewhat unreasonable thing to look for.
http://CryoLANparty.com/ A lan I'm staff on!
Get a laptop or 3.
Portable - check
UPS - check
Able to handle no climate control - check
4 cores & 8GB - check
4TB of storage - Get an external drive bay. (Do you really need that much storage? really?)
Some of the XPS line from Dell or other 'Gaming' laptops should do the trick.
"The price good men pay for indifference to public affairs is to be ruled by evil men." ~Plato (427-347 BC)
While, unfortunately, any "real" solution to dealing with hot, dusty, humid, and otherwise unpleasant environments is going to cost a bloody fortune; the relatively modest specs that you are looking for should help.
4 cores, 8GB of RAM, and 3-4 TB of HDD is, these days, a slightly nicer than average; but hardly exotic, desktop computer. The nice thing about desktops is that, unlike servers, they are designed to deal with human environments, rather than datacenter ones. No AC, cat hair, cigarette smoke, that sort of thing. Plus, they are cheap and almost exactly the shape of a small rolling suitcase.
Since the environment is nasty, you'll want to make sure that the system has enough fans to keep things cool even if one conks out when you aren't there, and you'll want to have at least one spare drive in your RAID.
There is a good solution to your problem, probably manufactured to mil-spec by General Dynamics, that costs 50 times as much as you can afford; but, in this case, you might well be able to get away with doing it the cheap way, since your computational requirements are actually fairly modest.
get a toughbook, a satellite uplink, and colo a server somewhere controlled.
seriously. finding the computational strength you want with the power restrictions is not going to happen.
my company just shipped some units to a desert in the middle east (can't mention where). we bought an entire trailer and powering units (generators, solar, etc) to provide the juice to run the servers and air conditioning. it was _not_ cheap. you can do that or you can remote to a controlled area.
If they're big enough there's the potential of changing the entire climate of Bangladesh.
You sound like you want a high power server (multiple VM's) with significant storage (multiple TB's) to run on no power in an unconditioned environment. And you want it affordable. Those are rather contradictory requirements, rather like having cake and eating it too.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
You could get an external drive and possible cluster them together for the enhanced processor power? Dont know but this taptop seems to be able to handle the enviroment you want it to. Also UPS plus Solar Panels = headache so be prepared!
Wouldn't it be easier to just hook up a dehumidifier and use normal (non-rugged) parts?
... in rural Bangladesh that requires 4 cores, 8GB Ram, and 4 TB of storage? I can understand if you're in the city and involved in some company, but you make it sound like you want some serious number crunching to occur in the middle of the jungle.
How about offloading all of your processing requirements to a co-located server and just getting a cheap rugged laptop to access and control the processing.
Choose two
If you can read this... 01110101 01110010 00100000 01100001 00100000 01100111 01100101 01100101 01101011
That's going to be a performance limiter at any price, especially if you need x86-compatibility.
At least for now.
There are some nice low-power architectures out there if you don't mind having to use a free operating system. For them, the degree ruggedization will be a driving cost factor. -40C-+85C costs more than 0C-50C, but it costs less than -70C-+120C.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Are you protecting against heat and cold or rapid temperature changes? Shock? Humidity and liquid spillage? For each, how much ruggedness are you willing to pay for?
You can build a computer that can handle -40-85C and both high and low humidity that won't survive a 5-foot drop onto concrete, and you can build a device that will survive a 10-story fall while operating, but that has no extra protection against humidity and temperature extremes.
State your requirements, and for a price, someone will build it, or at least try.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Let me summarize your requirements
-> Runs cool and quiet
-> Heat, humidity, dust resistant
-> Portable
-> Low power requirements
-> Integrated UPS
-> Very beefy server
-> Cheap
If you find one drop me an email, I want to install Duke Nukem Forever on it.
Sounds like you want plain ole standard commercial grade server hardware mounted in a tiny RV.
Extensively shock mount a relay rack, put in somewhat bigger AC/batteries/genset than usual, and you're good to go.
You can use the living quarters to house the armed guard, which will be required for expensive equipment in that corner of the world.
Trying to buy super tough server hardware will simply be more expensive than a RV and much harder to replace / maintain when it breaks.
Admittedly I'm mystified what you'd do with such immense computing power in a rural area without electricity. Maybe a really nice mythtv backend? Educate the locals using SimTractor?
You do realize that Bangladesh is like 1 foot above sea level, so no need to engineer this to last forever when its going to get washed into the sea every couple years by storms etc. Using a RV could help in the evac, assuming there is any place safe to evac to...
Alternately, split your workload transparently across maybe 50 smaller machines, and start purchasing replacements when attrition nears 75%.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
I know some guys who were running some wi-fi gear on a roof with a small linux server etc andto beat the elements (many days a year of driving horizontal rain and gale force windows) they submerged some low power components in a metal tool chest filled with mineral oil. Their set up had 4gb CF and USB keyfobs for storage. There was a 12VDC input power car-PC-style supply that handles variable input (goes as low as 6v) and they ran long wires down to a small 240v/12v transformer in the building. This meant that even if moisture got in, the components were very well protected as water would sit at the bottom of the oil, and there was utterly no dangerous voltage exposed to the outdoors. They later they went with a smaller o-ring sealed aluminum box filled with proper transformer oil, but the original hack was working fine after 1 year.
From my own experience with dunking rigs in oil, you only need to watch out for a few things, one being the mineral oil leaching plasticizers out of wire insulation - they eventually become brittle. You also need to seal your electrolytic caps with a little epoxy so the rubber seal doesn't get eaten alive. Interestingly most caps seem to survive a long time like this, but personally I'd recommend motherboards with solid aluminum caps.
However these things don't become a problem for months, so you'd likely get away with just dunking your rig and leaving it. You also cannot dunk a HDD, as the oil will get inside it and foul things up. I haven't tried it, but it would be possible to seal up in a box or 'pot' a mechanical HDD in epoxy, but best to stick with SSD / Compactflash.
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
I would actually go for a sealed box with cooling fins and then some water cooling with an external radiator and fans.
Sealed box - mostly to keep any kind of bugs out of the box and also to try to keep the humidity down. Add some silica gel inside to keep it dry.
Remember that silica gel can be re-used, you just have to dry it in some way.
I'm assuming that the box doesn't have to be deluge-proof, so just make it reasonably sealed. Add thermometers and possibly a small radiator/fan inside for general cooling of the PSU air.
With water cooling you will get a stable temperature and be able to get rid of a lot of heat - and be able to vent the heat outdoors.
Also select the most power-efficient PSU you can get your hands on to avoid unnecessary heat.
And for UPS - that shall be located in a separate compartment to avoid catastrophic problems in case you get a battery leak.
Mounting the whole box on inflatable rubber wheels would be a good idea - not only for moving it, but the rubber wheels can also provide vibration dampening when transporting.
Rugged things get heavy.
And don't forget - mount the hard drives using extra shock-proofing in some way. Mirrored drives is also a good idea since it may save you from some agony.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Buy a used 1U rack Dell server with redundant power supplies, Pentiums, ethernets and HDs on a RAID. Then replace the HDs with Flash SSD. Then put the whole thing in a plywood box with an air conditioner mounted on top, tubes blowing cold air in and three .00 grade nylon layers over the out vents, the upper layer removable. Seal all cracks, especially around cable slots, with silicone caulk. Run the whole thing as a unit, cleaning the air conditioner filters and out vent screens twice a day (so get two sets of those filters).
Keep spares of each redundant part. Buy two of those whole units (including air conditioners), because one unit will die anyway.
Run them on an ethernet switch, one powered down except once a day or so to sync their RAIDs.
Or rent a server at some global datacenter, and get WiFi/pringles antenna to an ISP somewhere.
--
make install -not war
I managed the IT for a couple of organisations in Cambodia and then Lao for a couple of years. Environments not so different from Bangladesh I suspect.
My experience was that it was best to buy standard mid range kit (IBM, or Dell Poweredge servers in tower cases worked just fine) and then invest in some physical infrastructure and climate control. It was generally straightforward enough to find a secure corner of an office and put install a small self contained rack with a UPS or two. Or even better get someone to wall up a corner of an office and put in an aircon. That kind of skill was in plentyful supply.
Lugging around some serious kit in that kind of environment would give me sleepless nights. The chance of it getting dropped, rained on or stolen is just too high. (We had a couple of laptops stolen while I was there, and you aint going to be happy chap if you come back to your hotel one night and find your server has gone walkies.) I advise you try and travel with what you need, preferrably a run of the mill inconspicuous laptop and find a secure base or two for your servers.
What would you suggest? Lesser hardware? Surely there must be a solution somewhere in the middle of "I want this" and "I can use this".
Yep, there is. But it's not always where you think.
Shameless (but hopefully useful) self-promotion:
I've been living and working in Least Developed Countries in the tropics for nearly 6 years now, and for the last 2, I've been writing a weekly IT-related column called Communications. There's a ton of advice in there. Go take a look. Check my tag cloud for relevant topics.
Here are a few fundamentals:
-1- The first thing to do is to adjust both hardware and - and this is important- software to the circumstances. Focus on the task first, then avoid confusing how that task is completed in a North American office environment with 'the right way' to do things.
-2- Scale everything down, in order to make the cost of failure of any single element as small as possible. This way, you get a solution that's replicable, affordable and - most importantly- easily replaced when (not if) it breaks.
-3- If you have unreliable power, then do two things first: Make your system tolerant to current fluctuations[*], and then plan for an intermittently available service. Forget about trying to keep it running at all times. Just minimise the cost of interruptions. A surge suppressing electrical switch on the wall where your main power source enters the building will cost you less and save you more than anything else.
[*] Bad (i.e. poor quality) power is the source of about 80% of hardware failure where I live. Every time the local power company hits us with brown-outs and spikes, I'd get a surge (heh!) of customer service calls.
To me, this situation screams 'require redundancy'. I understand this was not given as an option originally, but with the environment described I would certainly not want to rely on one single server.
Yes, redundancy is good. Cheap, small, easily replaced devices are good. Snap-shotted VMs are also good. The bottom line is that you need to keep the cost of failure low, because the system is certain to fail due to environmental factors. A good motto for working in the Developing World is: If you can't beat 'em, at least don't lose too much.
The best way to do this is to try to run on hardware that's about 3-5 years behind the curve, or to go straight to the bleeding edge of low-power tech.
To the submitter: I have a personal interest in Bangladesh, by the way. You can reach me by leaving a comment on my website. Good luck!
P.S. Unless money and space are no object, you'll never run full-time computing services on solar power. Especially in monsoon season. IMO, best not to try.
Crumb's Corollary: Never bring a knife to a bun fight.
Define Portable.
That's the first thing we need to know as it defines what type of system I'd recommend. Forgoing that information, what I would suggest is dropping Windows from the equation and switch to ARM processor base systems. This gives you the advantage of replacing all of the Linux VM's with standard Hardware, providing multiple redundancies. Another advantage is that the ARM systems can be spec'd to run on as little as 1watt of power (incl HD's) and since they're full linux boxes, you can easily administer them using SSH from a Windows box such as your current laptop.
The hardware redundancy offered by this is enormous and as the units they can easily be powered by a single Solar panel (sized correctly) combined with a large Deep Cycle 12 volt battery. Use Pico Power supplies (12volt input) and you've got your portable data center. The biggest thing you need to do is ensure that your Solar Panel has sufficient power to recharge the battery while powering the ARM systems. This means figure on at least 2x Solar Wattage over maximum demand to properly recharge the battery to handle a full run overnight. EG: 24 watts of demand means a single 55watt PV panel, though I'd look at using a pair of smaller 30 watt panels for redundancy in case you break a panel. It means you'd be able to run the system at reduced levels. One last item is provide an auxiliary power input for charging the battery bank from a vehicle or generator.
Mod me up/Mod me down: I wont frown as I've no crown
Forget air cooling entirely. Look up oil-immersed PCs. Immersing all but the HDDs in oil entirely removes the problems of dust and humidity. All you then need is to cool the oil, which can be done with either a pump and a passive radiator, or an immersed cooler which can be easily replaced when/if it breaks down. It would also provide a small amount of protection of shocks and vibration if the components were isolation mounted. It also has the benefit of being surprisingly when custom made (compared to commercial ultra-rugged servers). Unfortunately, it will still need some creative thinking to handle poorly regulated power supplies, and can be unpleasent to work on if a component fails. And it's nowhere near an elegant, low power solution, but as the article specifically mentions a quad-core with a terabyte drive array, I doubt power efficiency is high on the list of priorities.
I haven't done Bangladesh, but in Thailand if I had it to do over again I would go for four low-spec machines, and a sealed enclosure with compressed air cooling. It isn't the most energy efficient approach, but having a sealed (and slightly positively-pressurized) enclosure does wonders at keeping out moisture, dust, and ants.
The general idea is to have a couple small compressors (with check-valves) feed into a common reservoir that has adequate time to cool to ambient temperatures. Ideally, you would run at about 300 psi/20 bar, and have a pressure reducing valve inside the enclosure to drop the air to about 10 psi with a 1/16" orifice into the enclosure. (You might have to experiment on orifice and pressure.) Provide a pressure relief valve to keep the enclosure under 2 psig. (Another constrained orifice would work, but you will lose more air.)
Keep a spare machine in a pelican box with desiccant along with two or three spare hard drives. Keep a backup external USB hard drive in a separate pelican box with dessiccant and only open it up when you are doing a backup.
I'd also second comments about running everything in virtual machines and being willing to make compromises when one of them isn't working.
Back in my day, getting 12V power supplies wasn't nearly as easy as Google makes it sound. (You need to have a high enough float voltage to charge the batteries, and have a regulated output that will handle the end cell voltage of the batteries.) The logical alternative is to use 48VDC power supplies which are much more expensive. They are designed to operate within the float/ECV requirements of a VRLA. Don't forget your blocking diodes! Try to stay away from car batteries if you can and find some real deep-cycle batteries. Getting through monsoon season on battery isn't realistic without a huge battery plant. Our island's phone switch was pretty well equipped, but for two months a year the phones only worked when the sun was shining.
External connectivity was always my nemesis; when the phone switch was down, everything else crapped out. Satellite phones weren't viable from a cost perspective; the consumer satellite service was too unreliable to even be considered.
I don't know any off the shelf solution that meets all the requester's needs, but with a little backwoods / third world ingenuity, I think it can be done.
Seconded, start with a laptop. It's already met a whole bunch of your needs. The big missing piece is environmental hardening. Since it only needs to be moderately portable, there are a few things you can do.
#1, put the whole setup in a small refrigerator. Seriously. It's an airtight, watertight box. It'll keep out dust, rain and bugs, in addition to obviously keeping everything cool. Set the temperature as high as it'll go, there's no advantage to keeping the system any cooler than 20-25C, but the cooler you make it, the more problems you'll have with condensation on the evaporator coils. Deal with condensation by keeping down the number of times you open the fridge, and have some microfiber towels under the coils to catch the occasional drips. Loosely wrapping might work, but be careful not to over insulate the coils.
The downside: You'll take a hit on power efficiency, but I'm willing to bet that a decked out laptop and a small fridge pull less power than even the best servers will. Very small fridges based on peltier coolers are even less efficient, but their small size may make up for it. Also plan for what happens when the power goes out: The laptop keeps running, so make sure to have a UPS for the fridge, or that the laptop's battery will die before it gets too hot, or better yet, monitor a temperature sensor, and shut down gracefully when it starts getting too hot.
It's also a good solution because a fridge is relatively easily serviceable or replaceable compared to server parts. You'll need to carry spares for the laptop, of course.
Idea #2: Similar to above, but build your own peltier fridge. The main advantage is that you can make a custom enclosure to hold everything, which will keep the size and weight down.
Idea #3: Just build a hushbox for the laptop. Basically, you build a box around it, with some cooling fans, and furnace filters on the inlets. It'll take care of dust. You might have to shut down in extreme heat, and extreme humidity won't be handled.
Idea #4: Rugged laptops, like the ToughBook. They're ready to handle dust, moisture, physical abuse, etc, and with upgrades, can get near to the specs you're looking for. These might work well in combination with the suggestions above.