Slashdot Mirror
Know What Time It Is? Your Medical Device Doesn't
An anonymous reader writes "A man with one clock knows what time it is, goes the old saw, a man with two is never sure. Imagine the confusion, then, experienced by a doctor with dozens. Julian Goldman is an anaesthetist at Massachusetts General Hospital in Boston. After beginning to administer blood-thinning medication during an urgent neurological procedure in 2005, Mr Goldman noticed that the EMR had recorded him checking the level of clotting 22 minutes earlier. As a result, four hospitals in the northeast had their medical devices checked, and found that on average they were off by 24 minutes. The easy solution that devices could have used since 1985? NTP."
Why can't the medical devices be hardcoded to use an NTP server on the hospital's LAN?
First, they will use Windows Active Directory for NTP because someone will say "it's authoritative for the whole network". And their clocks will be off.
Then they will run into config hell, and blaming that for clocks being off - they will load balance the domain controllers. Which is precisely what you're not supposed to do with NTP. And their clocks will be off.
Then, some small but relevant IT subgroup will secede, claiming that they need "real" NTP. "Network Security" folks are typical suspects here. So their clocks won't match the rest of the gear (which is still off, remember?)
If you have poor enough technology discipline that your clocks are 24 minutes off already, you're probably screwed.
I want to delete my account but Slashdot doesn't allow it.
NTP does not require access to public networks. Private time servers, usually GPS sourced via rooftop antennas, are very common.
Can You Say Linux? I Knew That You Could.
You don't need to connect to an outside server. You can easily run your own time source (GPS is really easy these days), or have the devices talk to a single internal server which then securely contacts outward. If they're off, at least they're all on the same time. It's really dangerous if everything is reporting different incorrect times.
The easy solution that devices could have used since 1985? NTP.
Holy hell, what about no? There's a huge reason why hospitals try to keep off networks, especially public ones. Do you really want to connect all the timing devices in a hospital to an outside public server? Because running it yourself does no good, it can just fuck up all the devices in the hospital.
Sometimes the ideas non-thinking geeks come up truly scare me.
No one said the NTP server had to be external. NTP is just a method to keep a bunch of clocks synchronized to an arbitrary master.
Sometimes the stupidity of non-technical and non-thoughtful people scare me.
You could do an implementation of NTP on a closed network, with a local time source(compared to the rest of a hospital, an OK atomic clock doesn't cost that much, and a GPS timebase could be lost in a rounding error) with devices flagging anomalies in the NTP source and falling back on local quartz oscillators if needed.
It'd be more expensive than just having IT bring a patch cable; but there isn't anything about "NTP" that requires putting gramps' pacemaker on the internet...
Medical devices should not have Internet access.
Receiving time from a GPS receiver is much safer. That's a broadcast signal with a fixed message format.
NTP is just a protocol that you can implement. There are solutions that you can install internally that don't require internet access. Just stand up your own internal NTP server and have your own internal official time (possibly synced to something more authoritative). I agree with your sentiment about keeping sensitive medical equipment disconnected from the internet but with hospitals becoming more and more interconnected and not having their own physical infrastructure to do so, the internet looks like it's probably the best option. Yes, there are way to protect your traffic and all that but I must be pedantic and point out that NTP does not mean you must use the common servers available on the internet.
My work here is dung.
Oh noes!!! My android phone will make me 15 seconds early to any appointment!!!! I must therefore dump it and become and apple fanboy.
Supplies!
I don't get why people (like the parent poster) are blowing this up, out of proportion?
You don't have to expose all of the hospital devices/systems to the Internet, just to ensure they all have the same, accurate clock time!!
All you need is ONE device permitted to access the proper port for NTP protocol through a firewall, to set its own clock as the master, and then have it redistribute the date/time info to the remaining devices on the hospital's LAN!
It's not like the hospital doesn't already have Internet access and a firewall in place. They probably offer free wi-fi in the waiting areas, if they're half way modernized -- and even if not? They surely have Internet connectivity in place for at least a few needs.
There's a huge reason why hospitals try to keep off networks, especially public ones.
And yet device makers create equipment that requires their time to be set...
I can't help but wonder why they don't use a different report style. If you want to look at the log to see when the last event occurred, why not report in elapsed time? Like:
20 min ago: 10cc of supermed dispensed
40 min ago: 5cc of supermed dispensed
60 min ago: unit reset
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
hospitals try to keep off networks, especially public ones
The sysadmins try, but fail. Most hospital networks and devices ARE connected to the internet in some way. Doctors want their access from the various desktop machines, of course, but many of the diagnostic machines offer things like "click this button to email the ultrasound pictures". So they do.
I was appalled to learn this a few years ago from a hospital sysadmin here on /. The thing he pointed out is that Doctors are Gods. If they say "This thing can email pictures? Well yeah, hook it up!" then the sysadmin has zero choice. Holes get punched in firewalls that should never have been punched, and the gear gets hooked up.
And because medical devices are certified only to work with a particular operating system at a particular patch level, they don't get upgraded unless the vendor comes out with a new certified patched OS. That means the ultrasound machine sitting on that cart might still be running Windows XP SP 1. It's crazy.
NTP would actually be the least of their worries. That's something they could more easily house internally.
John
" Because running it yourself does no good, it can just fuck up all the devices in the hospital."
IF your devices are set to "fuck up" when time changes, you are buying really low grade crap.
NTP server on closed network for hospital devices, it's source is a GPS timing signal. Zero internet access, provides NTP and a accurate time signal.
This is SOP in any secure environment.
Do not look at laser with remaining good eye.
The easy solution that devices could have used since 1985? NTP.
Holy hell, what about no? There's a huge reason why hospitals try to keep off networks, especially public ones. Do you really want to connect all the timing devices in a hospital to an outside public server? Because running it yourself does no good, it can just fuck up all the devices in the hospital.
Sometimes the ideas non-thinking geeks come up truly scare me.
So the GPS synced clocks used to generate microsecond-accurate timing distributed via NTP for utility billing, high speed trading, and a million other mission critical (read: *beyond* life critical) things aren't good enough for your EMR tablet to update on once a day? Sure thing. Keep hiding under that table.
Actually a linux server on it's own you can calibrate the system clock to be incredibly accurate. you can calculate the drift of the cmos clock and adjust to get pretty darn accurate.
http://tldp.org/HOWTO/Clock-2.html
Do not look at laser with remaining good eye.
It's better than being a rabid android fanboy you are now.
P.S. Nokia phones also did the 15 second adjustment. Google just figures it is not important and also get's its time from the cellular towers that just happen to get their signal from GPS.
Do not look at laser with remaining good eye.
NTP have the problem of discontinuing his UTC timestamp while a leap second occur and NTP do not broadcast the actual UTC-TAI offset (historically because he broadcast UTC directly but this is now more a problem that an advantage). GPS and PTP broadcast (something very closely related to) TAI and a UTC-TAI offset, witch is the right thing to provides the precise actual time without discontinuity.
But all of them, NTP, GPS and PTP, have the problem of not broadcasting the historical leap second table, making the client of those protocols alone unable to safely compute a precise date in the past. I hope next NTP protocol will broadcast TAI, and that NTP, GPS and PTP will be able one day to broadcast the leap second table. I am certain that there is still some reserved bits somewhere in those protocol to make that working.
I thought of another good solution - when the electronic reporting system polls a device, it should ask the device what time the device thinks it is, and then adjust the report from the device accordingly.
Older devices without either the option to report in elapsed time or the ability to tell you the current time would be a problem. You'd have to either ignore the timestamps until someone manually verifies the time - which would be a joke since everyone would just click "Verify" without actually looking at the device - or come up with some more-clever-than-me solution.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
I am just playing with one said GPS unit. US$50, plus USB/Serial cable, plus conversion chips. 4h work, less than US$80 in hardware. There are ready made units for around $400, with a rubidium standard if the GPS is not available.
Not having a time standard is beyond not acceptable.
1. You don't know how GPS works
2. You don't know how network time is set
3. You don't know how atomic clocks work.
4. You don't know fucking anything about leap seconds and how they're handled by downstream users of GPS clock ticks.
Trollpost is troll.
--
BMO
We are talking about medical equipment that would have to be certified by the FDA. That would mean that every GPS receiver and every implementation of local NTP would have to go through a rigorous and costly certification process. The following issues would have to be certified;
1. Is the device accurate.
2. How does the device interact with the software.
3. How does the device interact with every device receiving data. This is the hard part.
Secondly, is it even necessary? The issue seems to be that there is an offset between the clock on the equipment and actual time. How about at the beginning of the operation the doctor writes down all the times as stated on the medical equipment. If necessary these offsets can be applied later to normalize the times. This reminds me of the time when the US spent tens of thousands of dollars to build a pen that would work in zero gravity(it was pressurized with gas). When a cosmonaut was asked how they coped he said "In Russia we use pencil". Sometimes high tech just complicates the issue.
This is often a case of poor administration, perhaps more frequently than poor design.
For example, I was recently tasked with reviewing the performance of several hospitals in the diagnosis and treatment of stroke. Under national guidelines (UK) a patient with suspected stroke must have had a CT scan within 30 minutes of arrival at hospital, with blood-thinning treatment administered within 60 minutes (if appropriate).
The problem was that the times on the CT scanners were discrepant by +/- 45 minutes from true time - so the images were tagged with the incorrect time. Further, the CT viewing workstations had times up to 2 hours discrepant. The CT scanners were Windows or Gentoo depending on the manufacturer's preference. Similarly, the CT workstations were windows, and were all bound to the hospital domain.
The time discrepancies made my assessment very difficult - and I had to correct for each individual scanner, and assume that the clocks hadn't drifted over the 6 month period of the audit.
I also found several safety issues because of this - e.g. if it was 1am, and a patient had a CT scan, some workstations would be 2 hours slow, so would read 11 pm on the previous day. These workstations would refuse to load the CT scan because the files were filtered by "WHERE [StudyTime] NOW".
I raised a support issue with the workstation vendor who simply said "These are windows workstations. You should ensure that they are appropriately bound to your domain, and configured to sync with your time server or domain controller". So I called IT to configure this, "No way. These are medical devices, we can't change the configuration - and anyway, what will happen if the clock is fast, and the sync pushes the clock back, so that there are 2 occurrences on the same time. That would cause chaos. Even if the manufacturer supports it, there's no way we'll set it up". Of course, their concern doesn't actually exist, because most time sync algorithms (even on Windows) are clever enough to avoid "double time".
There was similar obstruction with the CT scanners. The vendors simply said - we support and encourage synchronisation with a time server. IT or the radiology administrators simply stonewalled the ideas. They refused even to correct the clocks on teh scanners - so the clocks are still wrong to this day (even more so, due to accumulated drift).
Of course, even if the time can be set right - there is disagreement as to how daylight-saving is managed. Some equipment, esp. older embedded kit isn't daylight-saving aware. Do you set it to Summer time or winter time? In most hospitals I've been in, it's been an inconsistent mixture - often with lots of clock drift added, so you can't actually be sure.
When I had a hospital gig, we knew back in 1995 that time would need to be syncronized amongst all the servers etc. We ran a local time server synched to a Tier 1 NTP server which was fortuitiously about 180 miles away. It has since gone to restricted access, but it's nailed to the USNO, and is still a Stratum One server. I bet they still use it as reference.
But even in 1995, NetWare servers were well behaved and accept NTP, and we set workstation time on login. As other servers came on, we went through the inevitable 'my server is more accurate' and blew them off until the Sun server showed up,and they refused to use our NTP. Fine. Took two weeks to resolve a 300ms difference, and then I watched as they re-fixed the error and synched with the NTP server they initially refused to use. In fariness, it was not SUN engineers involved, but they were arrogant enough to qualify.
Time is important to networks.
We did not, however, have any way to manage time on 'devices', such as infusion appliances etc. I do NOT think of an EMR as a 'medical device'. Nor do I think of the EMR sytem that way either. But if time isn't being synched on your network, you got some other problems, I suspect, that are not making your work easier or efficient as a network admin.
deleting the extra space after periods so i can stay relevant, yeah.
our society is further behind then I've calculated.
Your calculations are right. The problem is your clock is way off.
Why does clock drift happen? It doesn't need to happen. It can totally be avoided. It only happens because the equipment manufacturers design inaccurate clocks to save money.
My quartz LCD watch from 1985 was accurate to within 1 second per year. That would WAY outlast the usefulness of the medical device. There should be no way in the world that device was off by 24 minutes.
Right now, I'm dealing with the same problem in my brand new car. It has a fancy on-board computer with a screen that tells me gas mileage, service info, mp3 and radio interface, etc.... The clock is ridiculously fast (gains 3 minutes a week). My new $20,000 car should have a clock in it at LEAST as accurate as the watch I can get from a happy meal.
The issue isn't accuracy. As you mentioned, even being off by a minute or two is rarely important. The problem is that all the little medical gizmos are not and will not be on any sort of network. The security ramifications of putting every last IV pump on the hospital network are simply too great to deal with, at least at present. Nobody is going to set up yet another network for time signals.
For larger, already networked machines - at least modern ones already are talking to a time server. Just looked at our Vitros analyzer manual and that's how it's set up. But a lot of portable machines are going to float alomg in their own time space for a while.
Faster! Faster! Faster would be better!
Well, there is the Radio Data Service that can synchronize time using actual FM broadcasts, but I guess I should have been more explicit.
What you're talking about is using a WWVB receiver. Also a good idea, especially since it comes directly from NIST at Boulder, Colorado.
Point remains, you don't need GPS, and you don't need Internet.
:(){
I would never trust the reported time from any device. Use the time that the recording system receives the data as the time of record (and poll data often.) Who cares what time the device thinks it is, along as it tells you the current data "now" and YOU know when "now" is?
Those who fail to understand communication protocols, are doomed to repeat them over port 80.
Is there a difference between leaking and transmitting? In practice, I don't think so. Otherwise, how would Britain have been able to use TV detector vans to find households that had not paid the TV licence?