Slashdot Mirror
Ask Slashdot: Tools For Managing Multiple Serial Console Servers?
An anonymous reader writes "I've recently been charged with updating our existing serial console access tools. We have 12 racks of servers each with a console server in it (OpenGear, ACS, and a few others). Several of these systems host virtual machines which are also configured to have 'serial' management (KVM, virt serial). In total it comes to about 600 'systems.' All the systems also have remote power management (various vendors). Right now our team has a set of home grown scripts and a cobbled together database for keeping this all together. Today any admin can simply ssh into the master, run 'manage hostname console' and automatically get a serial console or run 'manage hostname power off' to cut the power to a system. I'd rather use some tools with more of a community than just the 4 of us. What tool(s) should I move my group onto for remote serial/power management?"
If you published your tools, you might find you're not the only ones who need the flexibility you've written into your toolset.
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
... like OpenStack. Then you have access to everything.
now we need to go OSS in diesel cars
Raritan motherfucker! Do ya Speak it!?
You haven't identified any missing features or existing anti-features in your current toolset.
The only hint of anything wrong with your setup is "I'd rather use some tools with more of a community than just the 4 of us."
Q: What tool(s) should I move my group onto for remote serial/power management?
A: Yours. Clean the tools up, open source them, and market them. Your community will grow.
Step 1: Murder the idiot who thought saving 50$ up front while costing 5000$ in the mid term was a good idea
Step 2: Sell his organs on the black market
Step 3: Buy enough of whatever you have the most of to have one system
Good luck.
You might look at Digi.
I've used their stuff and it works well.
HPC scale: https://computing.llnl.gov/linux/downloads.html
I would recommend a look at http://conserver.com/
At a previous shop i worked we ran conserver against about 1k machines, some virtual but the big bulk of them was physical.
I also implemented a standardized way to control power on/off/reset on those machines as a patch to conserver. Those patches can be found at https://github.com/glance-/conserver/
if it ain't broke don't fix it!
My company builds a replacement (http://uplogix.com), smarter version of a serial console. They can all be managed by web UI and you can term directly into each device, keep configuration on them, and keep each device mapped to its outlet on the power controller. We even have a virtual version that runs on vSphere. You can hook up all the ports via telnet and keep your existing term server, but getting the benefits of the rich CLI and web UI.
Sounds like a perfect use case for you.
So... let me get this straight, you have a system which is easy to use and works just fine, and is written in house. Obviously, you want to change it... because? Jeez.
Are you willing to monoculture your vendors?
If the answer is "no", then you are stuck with your home-grown stuff. Vendors intentionally introduce incompatibilities to lock you into using only them, so you aren't going to find some project that provides a HAL, or at least not one that will live through the next software update from one of your vendors.
You should also be aware (I'm sure you are, if you understand the dynamics of your scripts, but some reading this probably aren't) that some systems won't negotiate a KVM style console unless they are selected active in the KVM prior to boot, so there's an interaction between your power management sequencing and the virtual serial and real serial, and that varies from vendor to vendor and software update to software update.
If you are also using Real KVM(tm) style virtual video consoles, you're probably already aware that Linux and most other Open Source OS's fail to negotiate EDID information correctly, unless you use the closed source video drivers, unless they are selected as the active input on the virtual/real video display device, since those implementations are usually not multithreaded, and so if you have 4 HDMI inputs, and #2 is selected, and #4 is where the device is that comes up and does it's one-time negotiation (this is what's broken about the OS drivers: they should retry periodically until they get a response, then echo up the response to the video driver, which if it's in X/Wayland in user space, it's not going to happen, since it only happens at startup) you are SOL. So your scripts probably have that knowledge, too. Not that it'll do you any good if you have a Linux box on the second input on a physical Samsung monitor, mind you, as they automatically switch away from inactive inputs, and default tho the first input if there are none active.
So good luck with your ask, unless you are willing to start your own project, and are willing to push to get UEFI, u-boot, Linux/BSD video drivers, and other things fixed as part of the project overhead.
Why are you looking for work to do? Your system is proprietary; your vendor mix and use cases are unique to your "community" and it works, so why the fuck do you want to reinvent it with a system that is not specific to your needs?
If you want to re-do it the "right" way you will pick one vendor, replace all of your non-compliant stuff with their hardware, and then fully adopt their toolset. Of course this is only "right" in the eyes of that vendor. Since you have your own right way, stop looking for a different one and go fix something else.
Want to do it cheap?
A raspberry pi has a serial input and enough GPIO's to externally switch it to many different devices.
the Beaglebone Black probably has much more potential in this regard as it has many serial ports and the switching could switch all of them at once to a new set of ports giving a much higher number of potential devices.
There's a little known, but very useful program called rtty. You can find it at ftp://ftp.isc.org/isc/rtty/rtty-4.0.shar.gz. Yes, it was last updated in 2003. Yes, there are package for major open source distributions.
Here's serial consoles on the cheap:
Buy multiport USB to Serial devices. They are a USB hub with a bunch of USB to Serial adapters hung off of them. Here's a 16-porter for an example: http://www.startech.com/Cards-Adapters/Serial-Cards-Adapters/~ICUSB23216F
Hang them off a low end box, I like half-depth Intel Atom servers with lots of USB ports.
Run rtty. It records each console to a log file 24x7, and allows multiple people to connect at the same time (including typing).
The most important thing is that it works reasonably well and doesn't require excessive administrative overhead. You've described a reasonably well managed configuration, so it's tough to justify changing everything "just because".
Exactly what would you accomplish by switching to an "open" technology? Answer that, and then you can make the best decision.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
A web UI means I can handle an issue from any device - a computer in a hotel, grandma's cell phone, whatever.
Also, with many systems, a UI to navigate groups of systems can be handy.
to manage their servers including console access via a special NIC port and the ability to push the power button remotely through a web browser
basic features are free. remote console is a $400 per server option called iLO Advanced on HP servers. the management software is free as well with some features licensed per server
"We have something that works and I want to break it"
Open the source of your tools and make them available to the public. You will be able to grow your own community. I would LOVE to have a look at your home-grown stuff. We are an Avocent shop here when it comes to remote console/KVM/power access and so far the only tool we are currently looking into is DSView which a) is sucky enterprise-y software, b) extremely pricey, and c) does way more than we need it to do. Thanks to your posting, though, I found interesting links to tools like conman and powerman in the comments.
At $WORK we have used Livingston PortMasters to deal with our serial console issues. Network connected and logins can be setup to only have access to certain ports. Have worked well for us. Usually found pretty cheap on eBay.
--JLockard - "Some mornings, it's just not worth chewing through the leather straps." - Emo Phillips
Not sure if this is what you mean, but we use an MRV server that has a bunch of serial ports and provides an SSH port corresponding to each serial port.
Can someone help me understand why you need to manage servers over serial connections?
I can see how serial management of routers and stuff would be necessary, but why servers. I feel dumb for having to ask, so thanks in advance!
Our company got excellent results by using http://www.skyline.be/Products/AboutDataMiner.htm . Company has always been extremely friendly implementing "drivers" to take care of vendor specifics.
If the current system works well then don't try to fix it. " I'd rather use some tools with more of a community than just the 4 of us." is a very bad reason to try to install and learn a completely different system.
Unless more information is presented, I'll place this in the "idle hands are the devil's playground" category.
These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
Consoleworks is designed to connect and manage multiple serial consoles, and provides logging, alerts, etc.
http://www.tditechnologies.com/
(full disclosure: I'm one of the patent inventors, but have no current financial interest)
I see you didn't bother to read one word of my post before replying.
Does the computer in the hotel business center have ssh installed? No. Does a borrowed phone have ssh? No.
Ssh is great, until I run out of battery.
Disclaimer: I work for these guys: http://www.ovirt.org/Features/DRBD
As somebody said before, this shop sounds like a fragile thing if some of those people leave. If customers depend on it, it might be advisable to switch to standardized tools for managing KVM environments. oVirt is the upstream project to Red Hat Enterprise Virtualization, i.e. those guys who really know KVM.
http://www.ovirt.org/OVirt_3.0_Feature_Guide
oVirt has pretty much everything he could ever dream of - and it is well documented, so any successor will immediately be able to handle the environment. Of course Open Source, it has a very active community with real experts:
http://www.ovirt.org/Documentation
Can't think of any reason no to use oVirt. It the exact feature set the OP is looking for, addressing his specific needs:
"Having a minimal CLI console available can make the product more attractive to users who use the command line and prefer to avoid using the GUI. It can also provide a simple and fast shell that requires no special client besides an ssh client, without having to connect to the actual VM. Serial console access can also be used for VM troubleshooting at the lower level."
Here you are:
http://www.ovirt.org/Features/Serial_Console_in_CLI
Also, oVirt has a very active community:
http://lists.ovirt.org/pipermail/users/
Take a look, it's free...
Since you haven't specified what the problem is, I'm guessing you're hung up on how to "mux a bunch of muxes." The key is to write two simple scripts: one to lookup which mux a given server lives on, and another to dispatch the mux command to the correct mux. Here's the latter:
Usage:
mux-muxer.sh HOST [COMMAND]
Now you just have to write lookup-server-mux.sh using your favorite database, and make sure all the server names are unique. Personally I'd use grep and sed on a plaintext file (literally just two piped commands), but you might prefer to use NOSQL or something else entirely (regardless of language, try to stay under 20 lines, or you're provably over-engineering it).
You're welcome.
p.s. I don't actually advocate using two 3-4 line scripts in production. You should add some error checking and syntax help if more than four of you will be using it.
Try running some random executable from a thumb drive on a hotel computer and tell me how will that works.
Then the next person you see, ask them if it's okay for you to install weird "hacker" apps on their phone
Im a little lost.. why not using ILO/ILOM/DRAC etc.. its all the same thing. Just give each ILO IP a DNS address and ssh to it, log in and there you have all the tools you need, a way to access the system console, power on/off remotely, check hardware events, hell it can even send SNMP traps to a NMS so you get alerts even if the OS dies! Using serial is so old hat, and clunky. Why on earth would anyone be using it at all these days? Serial! LMAO *scoffs*
If it is not broken, do not fix it. Your installation seems to work, why do you want to waste time "fixing" it?
That's a good point and I retract my comment in the context of console servers.
The point I had in mind is that although I use CLI for almost everything, sometimes a GUI is much nicer. The CLI for LSI RAID cards comes to mind.
and I tend to use /usr/bin/minicom. /usr/bin/screen.
Some of the guys at work prefer
Both have huge followings and have been around a long time.
Screen sounds like it may do what you want easiest.
But neither of these are going to work for the power management aspect.
I think you may find you will need to use your custom scripts for that with most solutions.
http://www.conserver.com/consoles/
Conserver is an application that allows multiple users to watch a serial console at the same time. It can log the data, allows users to take write-access of a console (one at a time), and has a variety of bells and whistles to accentuate that basic functionality. The idea is that conserver will log all your serial traffic so you can go back and review why something crashed, look at changes (if done on the console), or tie the console logs into a monitoring system (just watch the logfiles it creates). With multi-user capabilities you can work on equipment with others, mentor, train, etc. It also does all that client-server stuff so that, assuming you have a network connection, you can interact with any of the equipment from home or wherever.
Locally, sure. Heck, dropping a UI in front of the scripts that the article was asking about might make the management of the variety of devices easier; Just toss together a python program, or other easy language at hand, with the ability to call bash scrips and the ability to throw up a UI and mess with the database that tracks all the devices and their login details; that would make their scripts more usable should they all get fired tomorrow and the new team has access to an easier work flow.
But remote access from a physically unsecured device to what are probably secured systems without rolling one-time-use passwords? I'm just a CS drop out, and I can spot some major security problems there. And yes, I see even remote access to the switches as a problem, since havok could be caused. Imagine copying credentials, then (if it's a smart switch or other smart device) using that to tunnel to the companies boarder router (I recall some Cisco hardware had the ability to go from it's terminal to the terminal of other Cisco devices it was connected to...could be a faulty memory, my CCNA and other stuff expired 10 years ago) and then telling your gateway (from the inside at this point) to ignore the IDS and firewall. Or terminal jump to the IDS/firewall or database and see what other credentials could be had. Heck, I'd demand a VPN from a secure(ish) computer before allowing access to a GUI or CLI...preferably one with an onscreen, randomized keyboard for credential input or crypto signature credentials and a one-time-password (SecurID or something like it). But a public computer with gods know what running on it? Even copying crypto signatures from a thumb drive would be dangerous if the right malware was paying attention; or the malware could just infect the thumb drive. And if BadBIOS turns out to be real and is infecting any usb storage device then that thumb drive with a signature would have to be a single use device, never allowed to touch a secure machine again. A bootable write only (cd or dvd) with a on-screen randomized keyboard, and the public cryto sig burned with it, and a OTP device might be away to use an unsecured computer. Might, I haven't thought, yet, on ways I could still get the info if said machine was in my physical possession and I had hardware logging devices; maybe just recording the screen and network feed would get close, but the OTP device would be a stumbling block. If it used a weak chain, and you used the key at predictable intervals (so I could get multiple "key @ X time") to feed into a rainbow table...tricky, but maybe, I'll have to ponder more.
And don't take my security concerns as a personal thing. I'm just brainstorming so if the article author reads these posts, they might get to thinking about further issues and why those local CLI scripts might not be so bad. Or someone else thinking about remote access might question their security protocol.
Hey there, We are in the process of migrating away from proprietary serial console servers to one that's based on simple hardware and open source software. See http://cmrg.fifthhorseman.net/wiki/cereal it's also packaged in debian already.
We use it to manage 2000+ consoles across 3 servers. Latest version supports external commands e.g. to power on/off. A puppet class automatically dispatches consoles across servers.