Slashdot Mirror
Low-Bandwidth, Truly Remote Management?
kaiser423 writes "I'm looking to integrate some highly critical solutions into what would essentially be a remote, moving datacenter. No operators will be allowed at the site, and we may be able to have a high-speed INMARSAT data link. As a backup, we're planning to have multiple redundant low-speed Iridium data links. Essentially, we're looking to be able to power up/down and reboot some computers, and be able to start/stop some programs. We're willing to write the terminal interfaces necessary for our programs, and possibly do the remote desktop thing with some of our 3rd-party programs. But what is out there that would give us this type of access, work robustly over a high-latency, low-bandwidth stream, and would be tolerant to intermittent network outages? Please hold the pick 2 of the 3 jokes, I know they're contradictory goals; I'm looking for a compromise here! These boxes would regrettably nearly all be running Windows (with some VxWorks). Does anyone out there remember those days, and have any solutions that they preferred?" Read on for a few more details of this reader's requirements.
We've been looking at remote in-band and out-of-band management solutions, and really have found a ton of products. However, the "low-bandwidth" solutions still exceed our potential Iridium bandwidth (~10Kbps). Even if we have the INMARSAT link (192Kbps sustained, higher burst), a number of these solutions would hit that limit. We're starting to look at going old-school with some terminal-style applications, but haven't found much of a market for it; it seems to be a market that died with 56k modems. PC Weasel looks kind of like it might work, but the demo doesn't work for Windows.
We've been looking at remote in-band and out-of-band management solutions, and really have found a ton of products. However, the "low-bandwidth" solutions still exceed our potential Iridium bandwidth (~10Kbps). Even if we have the INMARSAT link (192Kbps sustained, higher burst), a number of these solutions would hit that limit. We're starting to look at going old-school with some terminal-style applications, but haven't found much of a market for it; it seems to be a market that died with 56k modems. PC Weasel looks kind of like it might work, but the demo doesn't work for Windows.
you can do an aweful lot with IPMI, i.e. power cycle, and a remote access card that supports ssh can do the rest, alternatively a TTY terminal and pipe your consoles to serial ports
oh, windows? nevermind
There's a surprising amount you can do from the command line within Windows these days. For UI access, RDP beats the common alternatives hands down, even if you log in just to use a command prompt remotely and thus have console state stored between sessions if the connection goes down. Have you actually tried this?
I wonder if anybody can put some numbers on the latency and bandwidth? I spent four months in China maintaining Windows servers in California via RDP. With latency often around 600-750ms and packet loss, it was painful but still usable. I was even contending with nested RDP sessions (RDP over the VPN to a machine in an office in CA, and then RDPed from there to a colocation facility).
It's no joke. Embedded remote access cards like iLO from HP or DRAC from Dell are the only real solutions.
It would be painful to use their console viewing features over a low speed satellite link but, it would be no problem to power cycle, collect statistics and more. You can even forgo the web interfaces and use ssh on many of them.
"Essentially, we're looking to be able to power up/down and reboot some computers, and be able to start/stop some programs."
Dial in using the telephone system and use a sequence of DTMF tones on your telephone keypad to carry out a task. This will be low bandwidth (about 2,700 Hz) and low cost.
Ganty
Good ol' RS-232 let's you do a lot. Run one very low power board that can sit there listening to RS-232 input and act on commands. It can then toggle the power of other equipment plus route messages from them however you choose.
The world is made by those who show up for the job.
Tivoli Management Framework had configurations designed to work with satellite links as slow as 16k. That solution was for monitoring and configuration management though -- not what you want.
Your big problem here is your expectations. Remote Desktop over a slow-speed, high latency link just isn't viable. Anyone paying the megabucks required to support a field-deployed solution will not be happy with the crappy service you'll ultimately provide.
You need to extensively model how your application works and develop appropriate procedures, runbooks for your remote operators and a toolset of programs or script to provide support for this "critical" solution.
Conformity is the jailer of freedom and enemy of growth. -JFK
Two words will get you far in this situation, Command Line. Low bandwidth, latency tolerant, and generally asynchronous. If you can get any tools with a command line option, embrace them.
GUIs suck, and they suck more over the conditions you describe. Avoid them like the plague. Also, think about mirroring the files you need to manage and editing them locally, then uploading them when you are done. Not always possible, but if it is doable, it can make your life a lot easier.
Scripting is your friend here.
-Charlie
Perhaps I am not understanding what the issue is with using the standard console servers and PDUs out there? All serial access is pretty darn low bandwidth.
http://www.avocent.com/products/serial-consolemanagement.aspx
http://www.raritan.com/products/serial-console-switches/
Plug one of these in, then connect a serial cable to your servers. Many include a modem if you have a pair of copper wires for a phone line so you can keep it out of band.
a: Remote management cards often have command line interfaces for resetting, system health, etc, through SSH. True, SSH with 800ms RTT times is a pain-in-the-ass, but if scripted, should work fine.
b: Once you can power cycle/machine health remotely, now you use SSH to connect to a command line shell on the system itself (yes, even windows) and do all further tasks from the command line.
Test your net with Netalyzr
I think it is important that you have all the equipment you can on IP addressable Ethernet Power strips so you can physically cycle the power remotely independent of higher level computer control. Something like this: Power Strip
There is no substitute for the ability to toggle the most significant bit--for sure.
It sound like this is for Science in Antarctica.
http://www.bitvise.com/winsshd It does the job connecting all kinds of platforms/client implementations. It does PKI too.
HP's Compaq line of servers has **excellent** remote admin capabilities.
Push the whole thing over an stunnel and you are good to go.
Implementation is another issue. Publish an email if your budget supports consultants. Errmm. Well, it looks like slashdot is taking the place of a qualified expert, so good luck with that.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
You may be able to run RDP across even the low speed links - choose the bare minimum screen resolution and color depth possible. It will still be pretty frustrating and slow, but you could use it with enough patience. Or you can run VNC, though I believe its performance will be less than RDP.
For CLI access, install an SSH server on the Windows box. If your code runs as a service, you can interface to it through a CLI client. It's some development work, but possible.
For power, I like APC's smart power strips. They support HTTP and SSH access.
Whatever solution I used, it would have to be run over an encrypted satellite link.
We use VNC and NetOp with our satellite sites. It works decent. It is slow (maybe 2 minutes) to authenticate with AD when you initially log on (if you log in locally, it is faster), but once you are authenticated, things work pretty good. There is definate latency between when you click the mouse and action happens, but it is definately usable.
As for the iridum setup, you might want to check those speeds again. When we looked into it, they were only able to offer a 2400 baud serial connection over the Iridium system. That is REALLY slow, and with high latency. We decided against going with the backup.
I'm not sure where you are planning on deploying these setups, but maybe a cellular modem would fit your needs more. They are fast, low latency, and comparable in price. The only hitch with them is that there is no SLA; just best effort. If you are out of coverage area, try to hook up a high gain antenna to the cell modem and try again.
I would just use remote desktop tools to manage your servers. VNC especially works quite well with low bandwidth, high latency connections.
One thing to let you know though. Make sure that you have someone that can go to the site and has access to the network equipment. We have almost 20 sites like this, and about 2 go down a month. The sat-modems or sat-controllers sometimes need to be rebooted, and having someone near the site to do that can save you guys A LOT of money. When we have to send out a tech, it costs around $2,000.00 due to driving charges.
Copyright 2010. All rights reserved. This comment may not be copied in any way including, but not limited to caching.
Get an IP aware KVM.
Using these you are separated from the operating system completely. You can see the machine boot, get into its BIOS, do a power reset (with a compatible power strip).
They have encryption and use a Web interface. Some have a fat client.
And try to run things from a command line as much as possible. Have the machine start a full screen command session upon boot, and hide the task bar. That should minimize the initial screen scrape.
Its the next best thing to being there...
- - - - - - - - - - -
I am a programmer. I am paid to produce syntax not grammar. Deal with it.
We've been extremely happy with the eRIC remote management board from Peppercon for all our lights-out remote-datacenter Windows machines. We use it over landline dialup modem (33.6) but it will work fine over any serial link you can throw at it, as long as you're patient. Full graphical remote console, remote IPMI, remote reboot, remote poweron, indepdent power supply (optional), it's great. They're a little hard to track down in the US, but I believe Raritan distributes them now. They're not cheap, but if you're paying for INMARSAT and Iridium, you don't care.
Without knowing what you're trying to accomplish, it's hard to give any real advice. Automated spy van? Unmanned laser plane? Knight Rider style service truck? Continually running train with nuke launcher? There are only so many things you can do with an intelligent roving unmanned platform.
I wouldn't fight the jokers defending ridiculous specs like low-latency low-bandwidth remote windows implementations. Use telnet, only call it secur-link 2008 in the specs. The joke is right back at them.
There are plenty of commercial ssh implementations for windows.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
The folks at RealWeasel have a cute little device. Plugs into a PCI slot and emulates a VGA card. It then outputs, over rs-232, a serial console approximation of whatever the system is displaying on the VGA device. Also has watchdog, manual reset, and keyboard functions. Those, plus a bog standard serial terminal server, and you are all set.
Why would people do this when powershell is available for Windows servers and has all the same advantages and then some? Why not use the software that is already there, it's low bandwidth and you can do whatever you need to from it. Of course you'll want SSH as your interface to it as VPNs would be too bandwidth intensive to maintain.
Start with your server hardware. Most Dell servers have a Dell Remote Access Card which allows you to get a full console (including BIOS and power on options) via web page. Performance of the full GUI over a slow link is marginal however. I'm sure other server vendors have similar options.
Also or alternatively, look for BMC controllers (http://en.wikipedia.org/wiki/Baseboard_management_controller) and IPMI interfaces (http://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface).
From there address further needs with RS232, Telnet, SSH, etc. Step up to RDP and VNC for GUI needs.
I'm going to go back in my box and will think within the limits of my box: MS Sucks Linux Good I read too much Slashdot.
Get a very long network cable. Plug one end in at your principle location. Send the other end by mail to your secondary location. Wait for it to arrive
This may take a while as threading the mail system tends to cause kinks that have to be sorted autonomously by remote postman protocol.
When it arrives, plug in and use normal LAN remote control technology. Thanks to XKCD for the idea
I have been a user for about 10 years. This ends Feb 2014. The site's been ruined. I'm off. Dice, FU
Well, I used to do sales/technical support for both Iridium and Inmarsat systems, so I guess I know a little about these. The BGAN/SwiftBroadband solution from Inmarsat can easily handle Remote Desktop sessions to these units. 256Kbps is the top sustained QoS you can get, but the units can get speeds as high as 492Kbps. The network is nowhere near saturation, so speeds of ~350Kbps are not uncommon. Latency is (and will always be) around 1.2s, which sucks for remote desktop, but is workable.
I've seen people watch a slingbox stream over these things with no skipping, in a dense area. In a sparse area like what you've described, no one else will be on your spot beam, so the entire bandwidth of the beam is basically yours to use. It's really not an issue.
Also, if this doesn't work for some reason, easy solution for the CLI that no one wants:
1. Write CLI
2. Write Client-side GUI for CLI so that customers think it's new technology
3. Profit!
...only I used BGAN instead of standard INMARSAT (which may actually be what you're referring to here). In instances where I needed GUI access on Windows boxes I found a very workable solution: Installed 1 Linux box with FreeNX server and put RDP client on that box. I'd NX into the Linux box and then RDP into each Windows box from there. Absolutely workable over even a crappy connection.