Slashdot Mirror
Ask Slashdot: Tools For Managing Multiple Serial Console Servers?
An anonymous reader writes "I've recently been charged with updating our existing serial console access tools. We have 12 racks of servers each with a console server in it (OpenGear, ACS, and a few others). Several of these systems host virtual machines which are also configured to have 'serial' management (KVM, virt serial). In total it comes to about 600 'systems.' All the systems also have remote power management (various vendors). Right now our team has a set of home grown scripts and a cobbled together database for keeping this all together. Today any admin can simply ssh into the master, run 'manage hostname console' and automatically get a serial console or run 'manage hostname power off' to cut the power to a system. I'd rather use some tools with more of a community than just the 4 of us. What tool(s) should I move my group onto for remote serial/power management?"
If you published your tools, you might find you're not the only ones who need the flexibility you've written into your toolset.
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
You haven't identified any missing features or existing anti-features in your current toolset.
The only hint of anything wrong with your setup is "I'd rather use some tools with more of a community than just the 4 of us."
Q: What tool(s) should I move my group onto for remote serial/power management?
A: Yours. Clean the tools up, open source them, and market them. Your community will grow.
You might look at Digi.
I've used their stuff and it works well.
HPC scale: https://computing.llnl.gov/linux/downloads.html
I would recommend a look at http://conserver.com/
At a previous shop i worked we ran conserver against about 1k machines, some virtual but the big bulk of them was physical.
I also implemented a standardized way to control power on/off/reset on those machines as a patch to conserver. Those patches can be found at https://github.com/glance-/conserver/
My company builds a replacement (http://uplogix.com), smarter version of a serial console. They can all be managed by web UI and you can term directly into each device, keep configuration on them, and keep each device mapped to its outlet on the power controller. We even have a virtual version that runs on vSphere. You can hook up all the ports via telnet and keep your existing term server, but getting the benefits of the rich CLI and web UI.
Sounds like a perfect use case for you.
So... let me get this straight, you have a system which is easy to use and works just fine, and is written in house. Obviously, you want to change it... because? Jeez.
Are you willing to monoculture your vendors?
If the answer is "no", then you are stuck with your home-grown stuff. Vendors intentionally introduce incompatibilities to lock you into using only them, so you aren't going to find some project that provides a HAL, or at least not one that will live through the next software update from one of your vendors.
You should also be aware (I'm sure you are, if you understand the dynamics of your scripts, but some reading this probably aren't) that some systems won't negotiate a KVM style console unless they are selected active in the KVM prior to boot, so there's an interaction between your power management sequencing and the virtual serial and real serial, and that varies from vendor to vendor and software update to software update.
If you are also using Real KVM(tm) style virtual video consoles, you're probably already aware that Linux and most other Open Source OS's fail to negotiate EDID information correctly, unless you use the closed source video drivers, unless they are selected as the active input on the virtual/real video display device, since those implementations are usually not multithreaded, and so if you have 4 HDMI inputs, and #2 is selected, and #4 is where the device is that comes up and does it's one-time negotiation (this is what's broken about the OS drivers: they should retry periodically until they get a response, then echo up the response to the video driver, which if it's in X/Wayland in user space, it's not going to happen, since it only happens at startup) you are SOL. So your scripts probably have that knowledge, too. Not that it'll do you any good if you have a Linux box on the second input on a physical Samsung monitor, mind you, as they automatically switch away from inactive inputs, and default tho the first input if there are none active.
So good luck with your ask, unless you are willing to start your own project, and are willing to push to get UEFI, u-boot, Linux/BSD video drivers, and other things fixed as part of the project overhead.
There's a little known, but very useful program called rtty. You can find it at ftp://ftp.isc.org/isc/rtty/rtty-4.0.shar.gz. Yes, it was last updated in 2003. Yes, there are package for major open source distributions.
Here's serial consoles on the cheap:
Buy multiport USB to Serial devices. They are a USB hub with a bunch of USB to Serial adapters hung off of them. Here's a 16-porter for an example: http://www.startech.com/Cards-Adapters/Serial-Cards-Adapters/~ICUSB23216F
Hang them off a low end box, I like half-depth Intel Atom servers with lots of USB ports.
Run rtty. It records each console to a log file 24x7, and allows multiple people to connect at the same time (including typing).
The most important thing is that it works reasonably well and doesn't require excessive administrative overhead. You've described a reasonably well managed configuration, so it's tough to justify changing everything "just because".
Exactly what would you accomplish by switching to an "open" technology? Answer that, and then you can make the best decision.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Use the current host hardware.
now we need to go OSS in diesel cars
A web UI means I can handle an issue from any device - a computer in a hotel, grandma's cell phone, whatever.
Also, with many systems, a UI to navigate groups of systems can be handy.
to manage their servers including console access via a special NIC port and the ability to push the power button remotely through a web browser
basic features are free. remote console is a $400 per server option called iLO Advanced on HP servers. the management software is free as well with some features licensed per server
"We have something that works and I want to break it"
Open the source of your tools and make them available to the public. You will be able to grow your own community. I would LOVE to have a look at your home-grown stuff. We are an Avocent shop here when it comes to remote console/KVM/power access and so far the only tool we are currently looking into is DSView which a) is sucky enterprise-y software, b) extremely pricey, and c) does way more than we need it to do. Thanks to your posting, though, I found interesting links to tools like conman and powerman in the comments.
At $WORK we have used Livingston PortMasters to deal with our serial console issues. Network connected and logins can be setup to only have access to certain ports. Have worked well for us. Usually found pretty cheap on eBay.
--JLockard - "Some mornings, it's just not worth chewing through the leather straps." - Emo Phillips
Not sure if this is what you mean, but we use an MRV server that has a bunch of serial ports and provides an SSH port corresponding to each serial port.
If the current system works well then don't try to fix it. " I'd rather use some tools with more of a community than just the 4 of us." is a very bad reason to try to install and learn a completely different system.
Unless more information is presented, I'll place this in the "idle hands are the devil's playground" category.
These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
Serial consoles allows you to fix boot issues remotely.
These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
I see you didn't bother to read one word of my post before replying.
Does the computer in the hotel business center have ssh installed? No. Does a borrowed phone have ssh? No.
Ssh is great, until I run out of battery.
IPMI can certain help at the hardware level. But if you need frequent hardware level access, or access to it by more than a couple people, something is wrong. The cloud infrastructure allows appropriate virtual level access (terminate an instance and start a new one). Maybe his "community access" was meant to do OS installs, which an infrastructure cloud stack would do. If not, maybe we need to know what the OP is really wanting to do.
now we need to go OSS in diesel cars
Outside the realm of windows, for boot and boot issues you'll be in command line land anyway. Most mainframe and supercomputer and midrange and micros in my 30+ years of professional life other than windows, OS/2, later novell netware...all needed the command line skills.
Try running some random executable from a thumb drive on a hotel computer and tell me how will that works.
Then the next person you see, ask them if it's okay for you to install weird "hacker" apps on their phone
Im a little lost.. why not using ILO/ILOM/DRAC etc.. its all the same thing. Just give each ILO IP a DNS address and ssh to it, log in and there you have all the tools you need, a way to access the system console, power on/off remotely, check hardware events, hell it can even send SNMP traps to a NMS so you get alerts even if the OS dies! Using serial is so old hat, and clunky. Why on earth would anyone be using it at all these days? Serial! LMAO *scoffs*
If it is not broken, do not fix it. Your installation seems to work, why do you want to waste time "fixing" it?
That's a good point and I retract my comment in the context of console servers.
The point I had in mind is that although I use CLI for almost everything, sometimes a GUI is much nicer. The CLI for LSI RAID cards comes to mind.
and I tend to use /usr/bin/minicom. /usr/bin/screen.
Some of the guys at work prefer
Both have huge followings and have been around a long time.
Screen sounds like it may do what you want easiest.
But neither of these are going to work for the power management aspect.
I think you may find you will need to use your custom scripts for that with most solutions.
Consoleworks is what we used at one of my previous employers. It had software that would run on Windows or Linux.
"We're gonna need a bigger boat"
Locally, sure. Heck, dropping a UI in front of the scripts that the article was asking about might make the management of the variety of devices easier; Just toss together a python program, or other easy language at hand, with the ability to call bash scrips and the ability to throw up a UI and mess with the database that tracks all the devices and their login details; that would make their scripts more usable should they all get fired tomorrow and the new team has access to an easier work flow.
But remote access from a physically unsecured device to what are probably secured systems without rolling one-time-use passwords? I'm just a CS drop out, and I can spot some major security problems there. And yes, I see even remote access to the switches as a problem, since havok could be caused. Imagine copying credentials, then (if it's a smart switch or other smart device) using that to tunnel to the companies boarder router (I recall some Cisco hardware had the ability to go from it's terminal to the terminal of other Cisco devices it was connected to...could be a faulty memory, my CCNA and other stuff expired 10 years ago) and then telling your gateway (from the inside at this point) to ignore the IDS and firewall. Or terminal jump to the IDS/firewall or database and see what other credentials could be had. Heck, I'd demand a VPN from a secure(ish) computer before allowing access to a GUI or CLI...preferably one with an onscreen, randomized keyboard for credential input or crypto signature credentials and a one-time-password (SecurID or something like it). But a public computer with gods know what running on it? Even copying crypto signatures from a thumb drive would be dangerous if the right malware was paying attention; or the malware could just infect the thumb drive. And if BadBIOS turns out to be real and is infecting any usb storage device then that thumb drive with a signature would have to be a single use device, never allowed to touch a secure machine again. A bootable write only (cd or dvd) with a on-screen randomized keyboard, and the public cryto sig burned with it, and a OTP device might be away to use an unsecured computer. Might, I haven't thought, yet, on ways I could still get the info if said machine was in my physical possession and I had hardware logging devices; maybe just recording the screen and network feed would get close, but the OTP device would be a stumbling block. If it used a weak chain, and you used the key at predictable intervals (so I could get multiple "key @ X time") to feed into a rainbow table...tricky, but maybe, I'll have to ponder more.
And don't take my security concerns as a personal thing. I'm just brainstorming so if the article author reads these posts, they might get to thinking about further issues and why those local CLI scripts might not be so bad. Or someone else thinking about remote access might question their security protocol.
Hey there, We are in the process of migrating away from proprietary serial console servers to one that's based on simple hardware and open source software. See http://cmrg.fifthhorseman.net/wiki/cereal it's also packaged in debian already.