Slashdot Mirror
Server Room Temp Monitoring and Notifications?
Supp0rtLinux asks: "Like many businesses, my server room is a standalone environment from the rest of my building. It has its own UPS, its own survelliance system, and its own AC system powered by its own generator. These are separate from the global building UPS, survelliance, AC, and generator systems and are designed to operate even when the rest of the building is down. However, in my current server room and in others I work with, I find that the AC systems generally lack a network-based notification system. As such, while my server room AC failed at 2am last night and temperature climbed to over 98 degrees, no one was aware until after 8am this morning when the audible alarms were heard. How do other Slashdot readers handle this?"
"I've thought about using some server motherboard with thermal monitoring, but they typically: a) only allow for shutdown at a certain temp, not for warning/email; and b) a well cooled server may not necessarily become excessively hot even if the room heats up. I know some newer AC systems *do* support SMTP notifications, but older ones either do not or are cost-prohibitive add-ons. The very popular Lieberts that are found in the ceilings of many server rooms are a good example of this. Do you know of devices that are network/SMTP capable that can be set with thresholds and alarms and generate emails, pages, or SMS messages when said alarms go off?"
Hot Little Therm but see the warning about no longer selling them ... great thermo probes, wish they were still selling them. I am glad I have a few extra.
Weather Duck and Power Egg
These ought to do the trick just fine. A bit of configuring or shell scripting, send email to a cell phone or pager or whatever, you should be happy as a clam at high tide.
There are probably others as well. There may even be source code on sourceforge. Hot Little Therm has software. Weather Duck may also.
Infuriate left and right
I found a possible low-cost solution to your problem here. It's not exactly what you're looking for, but it could be tailored to help out in your situation, particularly if you choose to monitor a HDD with low utilization, and set the warning temp at just a few degrees above the highest temp it normally hits when the AC is operational.
Hope this helps....I'm thinking I might hit up my supervisor for this software as well...there's certainly no harm in being careful.
____
~ |rip/\/\aster /\/\onkey
1) Get a digital multimeter with RS232 and a temperature probe.
:)
2) Connect it to a small, well-cooled, Linux box (Mini/ITX would be good)
3) Write a quick perl script to poll the DMM every 10 seconds (or something).
4) If current_temp > max_temp { send_mail(); }
It's cheap, AND you get to play with Mini ITX linux boxes at work
My other car is first.
http://www.nagios.org/products/environmental/esens ors/em01b.php
Doesn't get any easier.
Google it. They make a device that will monitor temperature, power, humidity, flooding, etc, and call up to 4 different phone numbers in a loop until someone acknowledges the alert.
Build yourself a couple of Thermal Cubes ($3.50 - $5.00 each), and connect them to a box running Nagios (which you should be running anyway). Hey presto, temperature monitoring. And you get to play with soldering irons at work, which can be great fun if you act secretive and mutter about overclocking.
Another one bites the dust
I did this in our server room after the A/C's kept "randomly" shutting off. We use nagios and the Esensor. It's kinda pricey but it's way worth it. There are scripts that will make it integrate directly with Nagios so there's a TON of ways it can alert you via email/SMS/etc...
--Ajay
Cron job. mbmon. 5 lines of Perl.
I bought a black box that detects power, humidity, and temperature changes and calls me. It's programmable by DTMF, it's cheap, and you can tweak the threshold parameters to your heart's content. The ones I use were purchased from Microtechnologies, Inc. in CT. It woke me up after a bunch of power outages last night. Some might prefer something more exotic or flexible, but this is quick, dirty, and it works (4+ years).
http://www.netbotz.com/.
Configure your servers to use SNMP to broadcast their current temperature to (two) monitoring systems. Configure the SNMP traps on the monitoring systems to email you when the temperature is out of acceptable range.
Not only will this catch AC failures, it will also catch problems with the server's cooling fans and so on.
Our Server room has a small box on the wall that mainly monitors our UPS. It hooks up with alarm wire contacts (normally closed/normally open) And also has a temprature sensor. When an alarm triggers it calls pre-programmed phone numbers in sequence, and tells you which alarms are going off (and the room temprature). It also has a noise level monitoring system too, if the noise level gets too high, it will call you and let you listen to the room via it's built in microphone. All you need is a power outlet, and an analog phone line.
This box is the series we use. I don't work for them, it's just worked well for us, so I figured I'd share.
We use a Sensaphone 1104 (my boss is a real geek). It will dial phone numbers, send pages if certain conditions are met regarding the server room. Sensaphone offers many more products that do similiar tasks.
YOU'RE WINNER !
Another lame blog
Alarm companies will sell you a monitored service to do just what you want.
That's what we use.
We bought a WeatherGoose for one of our server rooms. They are fairly cheap ($399 for the unit) and have many accessories that can be purchased as well as leads for "homebrew"/specialty sensors.
The WeatherGoose has a builtin web server that has (among other things) an xml feed...this makes writing scripts to interpret the data a breeze...not to mention it has a builtin "pager" alarm that will email and address whenever an alarm is tripped (high temp for instance)...
IT WatchDogs
Have whomever is in charge of building maintenance install a temperature sensor and remote monitor connection to the building alarm system in your computer room. I did that at the last four facilities I managed. It is a hassle free solution, the monitoring company can be given a list of people to contact -- with a specific order for contacting them.
That way, you can go on vacation! Too many homebrew solutions end up being dependent on the sysadm being available. Vacations are nice! Try to remove anything that prevent you from taking one.
For every problem there is a solution that is simple, obvious and wrong.
Check http://www.sensatronics.com/ They sell very neat network-attached environmental sensors which are very easy to work with.
(Friend works for them)
http://www.digitemp.com/ for the software,
http://www.ibuttonlink.com/ to get the hardware.
Serial interface, and you can run sensors hundreds of feet away over cat5. Just remember to test the alerts if you roll your own system.
If you want to know more, let me know I'll see what I can do.
Larger routers have thermal sensors... Long ago in a job far far away, I wrote a simple script to SNMP poll the router and then send an E-mail if the temp got too high. -Adam
We just use Nagios along with a temperature sensor and a custom-written Nagios plugin. It cost us about $200 in parts and about an hour's worth of labor to write the plugin.
Of course, there is always the esensor, which happens to go on sale tomorrow.
Just hire a security guard and post him at the door. Give him orders to shoot to kill if this guy tries to enter the server room.
Don't blame Durga. I voted for Centauri.
Out of the box monitoring for everything including an equipment cabinet door being openned.
http://www.netbotz.com/
My company (Disclaimer: I am an engineer, not a sales guy) makes products designed jsut for this! Ethernet-based (or serial-port based) and work with MANY software options including Nagios, or we have a perl script for data collection that you could hack to send you email if you like do-it-yourself. Here's the device I recommend. There are some homebrew-ish solutions out there, but we sell to thousands of very happy customers, and provide everything from the basics to lots of bells and whistles. SMS messages are the most commonly used notification method, though a few prefer pagers. Hey, you can even contact one of the engineers directly for tech support. ;-)
At my work we have a remote with delicate equipment, including an atomic clock, which needs constant temperature monitoring. The problem is that this is all in a mobile lab (in a modified truck, basically). So, our solution is simple enough:
- have the AC kick in automatically when it's too hot,
- have the heater kick in when it's too cold
- if for some reason these systems don't kick in, a temperature sensor installed and monitored by the security systems sets off an alarms.
So, for us, the same security company that monitors movement and access also monitors temperature, and calls us at home (or on our cells) if for some reason the room temperature is out of control.
Though this may be annoying, it works well.
I just pooped your party.
APC sells a monitoring board that goes in some of their UPSses and will do SNMP. Even if you don't need a UPS get one, hook it to your network, and you have your monitor cheaply. Marc
You can get a lot more than room temperature monitoring, if you want. I use a Davis Instruments weather station to monitor server room temperature as well as outside temperature, wind speed+direction, humidity, barometer, etc.
I use the Davis Weather Monitor II station, which can be picked up for about $350. Controlling the station from *NIX couldn't be more simple. I use the Device::WxM2 Perl module to pull data from my station. I wrote a small daemon that stores the data in RRDtool data files, as well as Perl Storables (a representation of a Perl data structure stored in a file). Another daemon monitors the current readings from the storables and sends e-mail to my pager if things go awry.
I even wrote an Asterisk AGI that speaks the weather to the caller. Call 1-866-859-7359 for a demo.
Why reinvent the wheel? --kash
switch them off on your way out
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
I use these products from AKCP. The Sensor Probe line is inexpensive and does the job. Web interface. Customizable thresholds. Also can monitor things like humidity too.
I've used this before. jeez. learn to use google.
http://www.sensorsoft.com/st6105c.html
I've thought about using some server motherboard with thermal monitoring, but they typically: a) only allow for shutdown at a certain temp, not for warning/email
:). The UPS software can probably send pages and/or email.
All my UPSes that have ethernet cards monitor temperature, and can send email when the temperature exceeds a specified value.
Even if your UPS doesn't have ethernet, you must have a computer connected to the UPS by serial or usb cable - how else do you shut down the servers in an orderly fashion
All of my dell rackmount servers monitor temperature at many points in the case, and can send email when the temperature exceeds a specified value.
One nice thing about host monitoring is that if you lose a fan or have some localized cooling issue, you will know about it this way, rather than via a whole-room temperature monitor. There are certainly scalability issues with doing per-host monitoring though.
Hulk SMASH Celiac Disease
"I've thought about using some server motherboard with thermal monitoring, but they typically: a) only allow for shutdown at a certain temp, not for warning/email;
What operating system are you using that can read the temperature itself but won't allow programs to read the temperature? With Linux you can just # cat /sys/devices/platform/i2c... to get the temperature from a shell script that can then go on to email, IM, page, etc.
and b) a well cooled server may not necessarily become excessively hot even if the room heats up.
Then what's the problem? Surely it's not a problem until the servers become excessively hot? Keep it in perspective here.
This can vary depending on your disks and other hardware, but to check the temperature of a hard disk I can type "smartctl -a /dev/hda | grep Temperature". If I were to parse that, I could make a cron job to notify me when a disk gets too hot. Disk temperature is your greatest concern, right? A broken AC is not the only way to fry a disk.
APC makes an inexpensive but very effective monitoring device:
x .cfm?base_sku=AP9319
http://www.apc.com/resource/include/techspec_inde
I have installed these at multiple sites with great sucess. They do email or SNMP notifications and are manageable through a web interface.
I use an internal APC environmental monitoring card, which costs about $150. There is an external version for about $250 USD. http://www.apc.com/products/category.cfm?id=7&subi d=29
I am using the internal card in an APC MasterSwitch, which allows me to control the power, and SNMP/web monitor the temperature (and create MRTG graphs). I also have a script which watches the temp and pages me at x degrees.
-=Down Syndrome in Maine
I second the IT Watchdogs products. We have a WeatherGoose hooked up to Cacti. Works like a charm.
Gabriel Ricard
We use a couple of simple serial thermometers http://www.google.co.uk/search?num=20&hl=en&safe=o ff&c2coff=1&biw=1152&q=rs-232+thermometer&btnG=Sea rch&meta= and a custom build app to read the serial port and feed the temp into our alerting system which handles our thresholds. This works well for us.
I also recommend two aircons with one set to kick in when the temperature gets too high (i.e. when the other fails)
http://www.ambientweather.com/sethandensep.html
This plug and go package has everything you need to protect rooms and equipment from temperature problems using a network enabled Windows NT/2000/XP/2003 machine. All software runs as Windows services.
It can alert you via email, pager and run commands when temperature goes outside your set limits.
The graphing tool allows you to graph temperature data from a web browser.
This package includes the ST6105C Sensorsoft Thermometer, RWME Remote Watchman Enterprise software (three monitored item limit), RWMS Remote Watchman Device Server software, F1000 Velcro fastener, printed Quick Start Guide and User Manuals on CD.
Product may not be as it appears in the image.
Shipping weight: 0.68 kg (1.5 lbs).
Inventory Status...
Sku: ST6105C-ESPAKpad$399.00
III.IIVIVIXIIVIVIIIVVIIIIXVIIIXIIIIIIIIVIIIIVVIII
APC makes some environmental monitoring units. Here's a list of them from their site:
9
http://apc.com/products/category.cfm?id=7&subid=2
They monitor temp and humidity among other things and include customisable relays, threasholds, remote access, and email monitoring.
IPSentry is cheap and easy to set up, and it can monitor a lot more than temperature and humidity. It has a ton of ways to notify you, and you can write your own notification modules in VB if you want.
And a note about contact lists: If you have more than two people getting the notifications, you should have a 'war room' voice bridge set up, so all the involved people can call in and figure out which one's going to get out of bed, etc.
If you're really slick, you'll give this number to the alarm company, so they can "remind" each groggy responder of the number, and so someone at the alarm company can join the bridge and provide updates while you're driving in to the office. (Ask your local Asterisk geek to set up such a thing, or try a service such as Mr. Conference.
The company I started working for this year bought a Sensatronics temp monitor that supports up to four probes. The manufacturer didn't ship software with it that would allow alerting, so I wrote an app to do so in about 150 lines of Perl. Since the device is accessible via SNMP, this was pretty simple using the Perl SNMP module. Notification is done via Net::SMTP and scheduling is done via Windows Scheduler (Windows shop, Windows solution). We're now up to about ten of these devices in various sites, all using my app for alerting.
My local nature center has a very accurate environmental monitoring system. There's a rock outside a window, and a guide which reads:
This is the weather rock.
If you can't see the rock, it's night.
If the rock is wet, it's raining.
If the rock is white, there's been snow.
If the rock is moving, there's a tornado.
If the rock is gone, there was a tornado.
Now, pair one of those babies with a nice Axis network camera....
At work we use the sh2+ from http://www.uptimedevices.com/. I coded a custom plugin for BigBrother, and we now have temperature and relative humidity with paging and trending.
The one thing I'd tell you to do is the trending. It's great to be able to look at the RRD plots and say, "I see the temperature goes up 3 degrees at night, I'll have to talk to the facilities people".
KMC makes a small digital control board with a built in web server and the ability to talk to an SMTP server. Their Weblite product can easily be set up to control any AC system. It's got 8 inputs so you can monitor 8 temp/humidity/airprove swithes, really anything you want and any alarm to SMS,Email,etc. Built in graphing and logging. Their product line is perfectly capable of running an entire building, very modular. Easy to program. Anybody that can hack together some Perl can program this stuff. Just be careful, short cycling and AC system gets stupid expensive very quickly http://www.kmc-controls.com/ Disclaimer: I work for a contractor that distributes and installs KMC systems.
Most of the larger, APC ups's have temperature sensors that you can query with snmp. Surely this can also tie into nagios...
http://www.digitemp.com/index.shtml
I had a live cam site, where they'd turn up the heat to crash the machines they were using, so they didn't have to work. They were Windows machines. All it took was putting a thermistor into a little housing, so it plugged into the game port. I wrote a little QuickBasic program to read the value, and report it back. It took a little math to figure out the value to temperature conversions.
I'm sure it could be done in Linux too, but the machines I was working with were Win98 workstations. If you can read the value of a game port axis, you can read the thermistor. It was just a couple bucks for parts, and a few minutes of programming. You could wire the termistor to anywhere you'd like. Mine sat right on the game port, but if you want the ambient temperature of the room, you'd probably want it a bit farther away from the machine(s).
You'll need to find the pinout for a game port, to figure out where to put it. It wasn't very hard to do. Pinouts are available all over the Internet.
Serious? Seriousness is well above my pay grade.
Check the front page of Nagios for network based environmental sensors that plug straight into your network.
We sell clusters, many of them I include an SP2 temperature sensing device. http://www.jacarta.co.uk/products/prod_sp.html This can detect temperature (duh) and other things, like water or contact sensors. The sp2 has 2 sensors, you can get more others. The sp8 may be more your thing. The sp2 can email, snmp trap sms etc. etc. etc. on configurable limits, a warning and an alarm. You can also query it via snmp (what I do) and web pages etc. where you get a history of the temperature. Personally, I use one temperature sensor in my server room. I also connect one to the clusters, I use email warnings if the room goes too high, and can shut down the cluster nodes if it gets serious.
Yeah - this may be all low-tech and homebrewish, but it is essentially the exact same thing that a UPS does (albeit with temperature sensing and not voltage level). Also, you can't sue a company if it fails, but considering nothing is being currently done, and you have already had a failure - this would be the easiest and cheapest way to go (a few hours worth of time and a few dollars for parts, tops)...
Reason is the Path to God - Anon
We picked up an 8 sensor model of the AKCP sensorProbe a while back. I don't recall it being particularly expensive and it seems to work very well. It's got 8 ports on it, into which you can plug a variety of different sensors. Sensors can be things like temperature, water, AC voltage, smoke, motion, etc. The sensors all terminate in normal RJ45 connectors, so it's pretty easy to reuse existing wiring infrastructure to get your sensors wherever you need them.
It has a web interface for configuration with all the pretty graphs you might want. Notifications are handled via email. We've got the emails going to a box running sendpage to send us SMS messages to our cellphones as well. SNMP is also an option.
I'm reasonably happy with it and would say it's at least worth a look. The specific model we have is here.
We use these NETIC Sensors [ http://www.netic.com/SP2.htm ]
They are excellent. Tiny, LINUX based, including TCP/IP stack, Web server, email, SMS and full SNMP functionality.
We have one for ambient temperature/ humidity, and one in every server rack.
We've written some self-rolled C code and Perl scripts to pull data from the Weatherduck, stuff it into a Postgres database, and trigger an alarm if the temperature or humidity get outside a certain range. Here's a link to our CVSWeb.
The basic procedure is that you pipe output from monitor into db_interface, and then run alert as a separate process. You can use the CGI script to generate GNUPlot graphs from a web form, or you can invoke it directly with graph.
The APC environmental monitor AP9319 does a nice job of monitoring temp and humidity. I added the second sensor to monitor both under floor temp and room temp. In addition the unit has dry contact inputs which I connected to a waterbug leak detector. I like the apc unit since it is self contained and has a built in webserver and supports SNMP. It also will email alerts directly from the monitor. I have been sucessful in integrating the SNMP output using cacti go graph temperature trends in our data centers and telcom room.
I'll show you how to build a PC (Linux/Windows/Mac) temperature sensor for less than $20 in parts.
-marc
We use Netbotz http://www.netbotz.com/ monitoring appliances for our server rooms, both the main datacenter and the alternate. We get SMTP mail, pagers, and cells...and the system has SNMP MIBs, so you can get MRTG graphs of the temperature, humidity and all enviromental variables.
You can get models with cameras, motion detectors, etc.
Minix en español! http://www.es-minix.org
We are using one of them, together with a simple Perl script that runs on a Linux PC. The script reads out the data from the Ethernetbox, sends Email alerts, makes nice graphs with RRDTool, and even will run an automatic shutdown if the temperature gets critical.
http://temperaturebug.com/
Works great and cheap at $129.