Slashdot Mirror
Know What Time It Is? Your Medical Device Doesn't
An anonymous reader writes "A man with one clock knows what time it is, goes the old saw, a man with two is never sure. Imagine the confusion, then, experienced by a doctor with dozens. Julian Goldman is an anaesthetist at Massachusetts General Hospital in Boston. After beginning to administer blood-thinning medication during an urgent neurological procedure in 2005, Mr Goldman noticed that the EMR had recorded him checking the level of clotting 22 minutes earlier. As a result, four hospitals in the northeast had their medical devices checked, and found that on average they were off by 24 minutes. The easy solution that devices could have used since 1985? NTP."
The easy solution that devices could have used since 1985? NTP.
Holy hell, what about no? There's a huge reason why hospitals try to keep off networks, especially public ones. Do you really want to connect all the timing devices in a hospital to an outside public server? Because running it yourself does no good, it can just fuck up all the devices in the hospital.
Sometimes the ideas non-thinking geeks come up truly scare me.
Few people realize this, but Android phones don't keep time correctly. They use GPS satellites for timekeeping, which was last updated in 1982, and since then there have been 15 leap seconds added. As a result, nearly all Android devices are 15 seconds too fast. Note that iOS compensates for this and shows the correct time.
I recall in the mid 90s wondering why cell phones had local time rather than network time. My phone rarely showed "correct time" - now it does. For whatever reason, syncing time to "reality" has never been high on the list - this seems to be changing..
Why can't the medical devices be hardcoded to use an NTP server on the hospital's LAN?
First, they will use Windows Active Directory for NTP because someone will say "it's authoritative for the whole network". And their clocks will be off.
Then they will run into config hell, and blaming that for clocks being off - they will load balance the domain controllers. Which is precisely what you're not supposed to do with NTP. And their clocks will be off.
Then, some small but relevant IT subgroup will secede, claiming that they need "real" NTP. "Network Security" folks are typical suspects here. So their clocks won't match the rest of the gear (which is still off, remember?)
If you have poor enough technology discipline that your clocks are 24 minutes off already, you're probably screwed.
I want to delete my account but Slashdot doesn't allow it.
Medical devices should not have Internet access.
Receiving time from a GPS receiver is much safer. That's a broadcast signal with a fixed message format.
NTP is just a protocol that you can implement. There are solutions that you can install internally that don't require internet access. Just stand up your own internal NTP server and have your own internal official time (possibly synced to something more authoritative). I agree with your sentiment about keeping sensitive medical equipment disconnected from the internet but with hospitals becoming more and more interconnected and not having their own physical infrastructure to do so, the internet looks like it's probably the best option. Yes, there are way to protect your traffic and all that but I must be pedantic and point out that NTP does not mean you must use the common servers available on the internet.
My work here is dung.
You're right that you wouldn't want to use untrusted time sources. But that doesn't mean NTP is useless here.
If the various devices need to work together, they also need to have similar time. It doesn't even need to conform to universal time; it only really needs to be the same in the same hospital, and especially in sync with the wall clocks doctors and nurses look at. I'd also want an indication on each clock that it still runs, that it's not standing still, and if mains-powered needs to be on the emergency power-enabled grid, too.
A well-tuned central clock or three (not two, thank you) should help quite a lot already. What that is, well, it could be GPS or an atomic clock or a receiver to the local government radio time service or what-have-you. But it really doesn't matter much. Keeping time among the devices inside the hospital is quite valuable in and of itself already.
Most devices dont. In fact it's highly rare for a device to NTP sync.
Most building lighting control systems do not do a NTP sync. Security systems, etc... all rarely do this.
It sounds like that someone finally realized that clock drift happens and is shocked about it.
Do not look at laser with remaining good eye.
Don't they have checklists to make sure their equipment is working as expected? Even brain dead pilots like me check every instrument on the panel, every time I fly. If I'm betting somebody else's life on it, I check twice. I'm living proof that an idiot can set a clock within a few seconds. Every nurse I've ever met keeps a watch with a sweep second hand (for observing pulse rate easily). How hard can it be to compare it to a GPS receiver,, networked computer or short wave receiver from time to time? I keep my watch within a second or two of UTC by looking at my computer, listening to WWV on the short wave radio, or looking at a GPS receiver, it's good enough for celestial navigation, it's surely good enough for medicine ..... how hard can it be to check life supporting equipment from time to time?
Every device has its own way of supporting various protocols. Microsoft doesn't even properly support it. Sure, the domain controller can go get time from a reliable source (best if it's Microsoft's time servers, thanks), but the damn things won't SERVE NTP. We just had a big brew over NTP support here because the damned clocking servers were off by 90 seconds. 1 minute a day by 2500 employees supposedly works out to about $1Million a year in overtime. I didn't bother with the math.
NTP have the problem of discontinuing his UTC timestamp while a leap second occur and NTP do not broadcast the actual UTC-TAI offset (historically because he broadcast UTC directly but this is now more a problem that an advantage). GPS and PTP broadcast (something very closely related to) TAI and a UTC-TAI offset, witch is the right thing to provides the precise actual time without discontinuity.
But all of them, NTP, GPS and PTP, have the problem of not broadcasting the historical leap second table, making the client of those protocols alone unable to safely compute a precise date in the past. I hope next NTP protocol will broadcast TAI, and that NTP, GPS and PTP will be able one day to broadcast the leap second table. I am certain that there is still some reserved bits somewhere in those protocol to make that working.
In my experience, NTP is the easy solution that nobody uses. Because hey, it's only a clock, right? How hard can it be?
:/
Do you see what I did there?
I think it could be done with a small device supplied by the medical device manufacturers. It would plug in at home or the office, or wherever the medical device user spends a lot of his or her daily time. It would then receive the time broadcasts either from GPS, or, even better, from the WWV radio time signal in Colorado. The small device could then push a standardized time format to the medical device via it's own short-range radio transmitter, perhaps triggered by a daily sync request from the implanted medical device. A larger-scale version of this could be available for hospitals and other medical offices to allow for easy time syncing of all medical devices. No need to plug into an internet connection, making it pretty secure, and it would be easy to test reception of the timebase signal, too: just add one of those self-setting atomic clocks, which also receive the WWV signal. If it syncs up, put the medical device time server near it to ensure timebase reception.
"A man with one clock knows what time it is, goes the old saw, a man with two is never sure."
That's why the sailor's adage is to take one clock or three to sea. With one, well, that's all you've got to go on. With two, you never know which one is right. With three, usually at least two agree.
NTP is secure, right? Nobody can jump on the hospital network and impersonate the NTP server(s) and set times incorrectly on devices, causing them to do bad things like produce false readings or overadminister drugs, yes?
We are talking about medical equipment that would have to be certified by the FDA. That would mean that every GPS receiver and every implementation of local NTP would have to go through a rigorous and costly certification process. The following issues would have to be certified;
1. Is the device accurate.
2. How does the device interact with the software.
3. How does the device interact with every device receiving data. This is the hard part.
Secondly, is it even necessary? The issue seems to be that there is an offset between the clock on the equipment and actual time. How about at the beginning of the operation the doctor writes down all the times as stated on the medical equipment. If necessary these offsets can be applied later to normalize the times. This reminds me of the time when the US spent tens of thousands of dollars to build a pen that would work in zero gravity(it was pressurized with gas). When a cosmonaut was asked how they coped he said "In Russia we use pencil". Sometimes high tech just complicates the issue.
This is often a case of poor administration, perhaps more frequently than poor design.
For example, I was recently tasked with reviewing the performance of several hospitals in the diagnosis and treatment of stroke. Under national guidelines (UK) a patient with suspected stroke must have had a CT scan within 30 minutes of arrival at hospital, with blood-thinning treatment administered within 60 minutes (if appropriate).
The problem was that the times on the CT scanners were discrepant by +/- 45 minutes from true time - so the images were tagged with the incorrect time. Further, the CT viewing workstations had times up to 2 hours discrepant. The CT scanners were Windows or Gentoo depending on the manufacturer's preference. Similarly, the CT workstations were windows, and were all bound to the hospital domain.
The time discrepancies made my assessment very difficult - and I had to correct for each individual scanner, and assume that the clocks hadn't drifted over the 6 month period of the audit.
I also found several safety issues because of this - e.g. if it was 1am, and a patient had a CT scan, some workstations would be 2 hours slow, so would read 11 pm on the previous day. These workstations would refuse to load the CT scan because the files were filtered by "WHERE [StudyTime] NOW".
I raised a support issue with the workstation vendor who simply said "These are windows workstations. You should ensure that they are appropriately bound to your domain, and configured to sync with your time server or domain controller". So I called IT to configure this, "No way. These are medical devices, we can't change the configuration - and anyway, what will happen if the clock is fast, and the sync pushes the clock back, so that there are 2 occurrences on the same time. That would cause chaos. Even if the manufacturer supports it, there's no way we'll set it up". Of course, their concern doesn't actually exist, because most time sync algorithms (even on Windows) are clever enough to avoid "double time".
There was similar obstruction with the CT scanners. The vendors simply said - we support and encourage synchronisation with a time server. IT or the radiology administrators simply stonewalled the ideas. They refused even to correct the clocks on teh scanners - so the clocks are still wrong to this day (even more so, due to accumulated drift).
Of course, even if the time can be set right - there is disagreement as to how daylight-saving is managed. Some equipment, esp. older embedded kit isn't daylight-saving aware. Do you set it to Summer time or winter time? In most hospitals I've been in, it's been an inconsistent mixture - often with lots of clock drift added, so you can't actually be sure.
When I had a hospital gig, we knew back in 1995 that time would need to be syncronized amongst all the servers etc. We ran a local time server synched to a Tier 1 NTP server which was fortuitiously about 180 miles away. It has since gone to restricted access, but it's nailed to the USNO, and is still a Stratum One server. I bet they still use it as reference.
But even in 1995, NetWare servers were well behaved and accept NTP, and we set workstation time on login. As other servers came on, we went through the inevitable 'my server is more accurate' and blew them off until the Sun server showed up,and they refused to use our NTP. Fine. Took two weeks to resolve a 300ms difference, and then I watched as they re-fixed the error and synched with the NTP server they initially refused to use. In fariness, it was not SUN engineers involved, but they were arrogant enough to qualify.
Time is important to networks.
We did not, however, have any way to manage time on 'devices', such as infusion appliances etc. I do NOT think of an EMR as a 'medical device'. Nor do I think of the EMR sytem that way either. But if time isn't being synched on your network, you got some other problems, I suspect, that are not making your work easier or efficient as a network admin.
deleting the extra space after periods so i can stay relevant, yeah.
Medical devices should not have Internet access.
That depends entirely on the purpose of the medical device. "Medical devices" covers a lot of products that serve a lot of different purposes. For some devices internet access would be pointless, for others internet access would be dangerous, and for still others internet access makes perfect sense. It depends entirely on what you are doing with it and what the risks are. There is nothing inherently wrong with hooking up a medical device to the internet, provided that the risks of doing so have been adequately addressed.
Receiving time from a GPS receiver is much safer.
Use of NTP does not require internet access. How the time is (securely) transmitted to devices is a separate issue than what protocol is used. You can use NTP on a network that is completely segregated from any other network and it works just fine.
I have two devices in my house that I wish had some form of clock syncing: the range, and the Ademco VISTA-20P burglar alarm.
Slashdot seems to be able to read my mind. I propose one of two solutions, and I openly invite device manufactures to join in on this:
- NTP. The device would have an Ethernet port on it. I'd read the MAC address off the label, make a DHCP reservation on the router, then go into a very simple Web interface where I would select the time zone and DST rules.
- GPS: The device would have an F connector on it. There would be a (commercially available) GPS antenna on the roof with RG-6 running into the structured wiring cabinet. From there, a (commercially available) F splitter splits the signal out to the range and burglar alarm, which would use GPS coordinates to determine the DST rules in effect and set the time. Some simple programming at the pushbutton oven controls would let you set the DST rules for the next time Congress decides to screw with them.
Notice that I am leaving WWVB completely out of this; it's not exactly reliable. My Casio Waveceptor watch hasn't picked up a signal since 2011-07-26 (I live in Florida), and I have to take my WWVB clock off the wall and put it on a windowsill for several days to get a sync during the DST changes. In a local public building, they round up all the wall clocks and put them in a van parked outside overnight to sync them.
The burglar alarm is a particular pet peeve of mine, which seems to have its clock drift measured in hours. The control panel is wired into a communicator that has a SIM chip and uses AT&T's GSM network to send monitoring signals...the damn thing already has the hardware to sync its clock, so why can't it?
In the drawer of the nurses station on 2 south we have manual blood pressure cuff and stethoscopes - they don't need to know what century it is even.
We also have residents who don't know what time it either and some of them are over a century too.
.. does everyone have to massively over-complicate a simple job that was a solved problem in my school in the early 1960s? They used mechanical counters that were sent a nice slow switching drive pulse every second, and a reset pulse every 12 hours, all down one three core cable. Every room in the school (and it was a big one) was synchonised to a single reasonably accurate master clock, and that is exactly all you need in a hospital. You don't need atomic clock accuracy, just reliability. Also, you got used to the one second click and stopped noticing it, however you instantly noticed if it stopped.
I think people are missing part of the point that's brought up in the article: NTP should have been used from the beginning, but since they don't use NTP now, how much will it cost to make that change?
Most places have an FM broadcast that is synchronized to an atomic clock. Ever seen one of those "radio clocks" that never needs the time set? It's using FM to know what time it is.
Put an FM antenna on the roof, hook it up to an NTP server, problem solved.
:(){
They could use clocks that sync to WWVB. That will at least reach the continental US and southern Canada. As for Hawaii and Alaska (aka, the Freak States), build transmitters there.
Why does clock drift happen? It doesn't need to happen. It can totally be avoided. It only happens because the equipment manufacturers design inaccurate clocks to save money.
My quartz LCD watch from 1985 was accurate to within 1 second per year. That would WAY outlast the usefulness of the medical device. There should be no way in the world that device was off by 24 minutes.
Right now, I'm dealing with the same problem in my brand new car. It has a fancy on-board computer with a screen that tells me gas mileage, service info, mp3 and radio interface, etc.... The clock is ridiculously fast (gains 3 minutes a week). My new $20,000 car should have a clock in it at LEAST as accurate as the watch I can get from a happy meal.
My post is too long at 55 characters? Is /. using Twitter technology? If 80 column cards were good enough in 1965,. they are good enough now (Lawn:Off)
Sent from my ASR33 using ASCII
Create a standard that all time is measured but a wall clock in the room. Hospitals have been dealing with this long before computers. So while a clock is a single task tool, it (can be) very good at what it does.
A bunch of thousand dollar solutions to a one dollar problem. People already maintain these machines, probably daily or at least weekly. Follow steps:
1. Set the time
I would never trust the reported time from any device. Use the time that the recording system receives the data as the time of record (and poll data often.) Who cares what time the device thinks it is, along as it tells you the current data "now" and YOU know when "now" is?
Those who fail to understand communication protocols, are doomed to repeat them over port 80.
Bet you your on dash readings are also off by 10-30%. Hooking in ODBII scanner to read injector pulses on a BMW and found the dash reading was 25% optimistic... trying to make the owner feel better.
Do not look at laser with remaining good eye.
If the device is reporting events as it happens (I don't know anything about how these systems work, but I would assume it did?) then why would the EMR be accepting external information regarding time, especially if they mention some EMR systems reject "wonky time". Wouldn't it make more sense to use the time the EMR receives the update as the time?
Is there a difference between leaking and transmitting? In practice, I don't think so. Otherwise, how would Britain have been able to use TV detector vans to find households that had not paid the TV licence?
The GPS-to-NTP server doesn't have to be in the same room as the existing devices. It just has to be on the same network.
Arizona [...] You build that 20 foot fence yet?
Again, China is way ahead of the United States on this one.
"ntp over the Internet - it doesn't support authentication, for one thing"
This says it had, in 2007
AccountKiller
My quartz LCD watch from 1985 was accurate to within 1 second per year. That would WAY outlast the usefulness of the medical device. There should be no way in the world that device was off by 24 minutes.
QFT. However, even as lame as this drift is, in a surgical context it's still simpler/better to add a procedural step to check the time on the appropriate devices before starting a procedure.
I mean, all the NTP advocates in this thread seem to miss the following facts:
Certainly, if a given device is already on a network and is NTP-capable, then by all means use NTP. Otherwise, just check the damn time on the device before starting the procedure and correct it once a week/whenever.
I wish they will use some sanity for this. Healthcare already costs too much without spending additional millions to avoid a five second pre-flight checklist item.
"Actually, the solution is obvious .. Highly sensitive GPS receivers .. I've been using one for years now, syncing my desktop machine every minute.
I have this small circular crystal-controlled device on my wrist that I've been using to tell the time for years now.
AccountKiller
In my country, it's not over power lines, but it's close enough: WWVB
Sorry, embedded lifes-at-stake systems don't use NTP. Or even network.
My quartz LCD watch from 1985 was accurate to within 1 second per year. That would WAY outlast the usefulness of the medical device. There should be no way in the world that device was off by 24 minutes.
Its doubtful that drift accounted for the entire 24 minutes, the problem is that the initial set was probably horribly done in the first place ("oh its 9 oclock [clock actually says 9:13]".) Add to that the tendency for makers of connected devices to basically take for granted that you have some means of clock checking, and they really don't care about how accurate the onboard method is (its probably not even a dedicated crystal on an isolated voltage controlled circuit, like you would need to keep time accurately.)
You confuse "GPS time" with GPS as a source for precision time. The difference between GPS time and UTC is broadcast every 12 minutes in the data stream, and includes the accumulated whole second difference, as well as drift value to correct for sub-microsecond. GPS receivers have access to this information and generally output UTC as soon as they read the offset value. Since the offset does not change often, it can be stored and used on subsequent startups, though this is a vendor specific decision. Though I've heard that they exist, every receiver and time source I've used outputs UTC by default by the time it downloads the full almanac and starts reporting position.
I worked at an emergency response centre. They liked windows (too much). Time was quite important because timestamps are logged by the database when calls come in, when emergency people arrive on scene, etc. Its important, as in subpoenas from a lawyer kind of important. Pity they used windblows. NTP was broken for such a long time. And their advice: use a 3rd party product or something. Remember: this software is licensed, not sold, and comes with ABSOLUTELY NO WARRANTY! All that rubbish I've heard people yelp about 'I want to be able to choke someone and with Mickeysoft I can do that" BULLSHIT! I worked for a freakin 911 call center and *I* couldn't get help or support from those bastards. Use Unix NTP, (basically a derivative of BSD NTP), and you will be fine. Stay the hell away from microsoft.
If they can't be accurate clocks, don't put a clock display on the damned device in the first place.
An internal tick-tock so they get dose rates right, fine, still need that, but TOD when you can't
guarantee accuracy is a PITA. The nurse's watch would be more reliable.
I HATE Microwaves and stoves that have a clock that you have to set before they'll even run -
for the same reasons.
All I want to do is nuke a pot of noodles for 90 seconds, I don't care if the microwave knows it's
after sunset or not.
GPS timing should be fine...the clocks (rubidiums) onboard the satellites get timing corrections regularly so should work and not be susceptible to hacking, spoofing, or any other major issues.
Yes, noise levels. Open this page and search for "easter".
I'm reading lots of nit-picking posts arguing about which system would handle leap seconds properly etc etc. Don't lose sight of the real problem: right now many of the clocks in use are not synchronized at all. They can be off by a very very wide margin. It would be good to pick one system such as NTP and use it on a closed network that is connected to GPS. Once you pick a system, especially if the software it open source, it would be much less of a problem to modify that system at a later date to handle leap seconds differently. You also need to think about the purposes of all these clocks. What is important is that they all be syncronized.
If your quartz clock was accurate to 1 second per year I think you should submit it to the guiness book of records. Quartz watches with proper temperature compensation and inhibition compensation (effectively a calibration for an oscillator running slightly fast) are accurate to about +/15 seconds in any given month. While I agree that it shouldn't be that far off it's actually quite difficult to design an accurate oscillator. More so that most medical instruments don't really use an oscillator divisible by 8 so there's some error even in the basic math before even talking about accuracy of the oscillator.
Typical quartz oscillators that are not temperature compensated or calibrated will typically gain or lose 1 minute in a month out of the box and that's if you keep them at a perfectly steady temperature.
More importantly why spend $10 on accurate temperature compensated components when a $1 component will still give the user a number, which is what really matters. I mean who needs their car to keep perfect time? We have wrist watches and mobile phones for that. If you only spent $20000 on your car I'd be more worried about the plastic starting to rattle, the paint fading off, the carpets wearing out, and all those other things that are actually visible and which matter to the value of a car.
Computers use the same quartz crystals so have the same accuracy. You see will see drift on the execution speed level (usually as a result of the scheduler not the actual clock), but in terms of seconds you'll rarely see it on a new computer.
Generally the clock drift you see occurs when the clock battery is low or dead. In a wristwatch you'd notice it quickly and replace the battery. In a computer, most people don't replace that battery in the entire lifetime of the computer. A medical device could well be 5+ years old which probably means its battery would need replacing, but since it doesn't interfere with its operation and isn't very obvious it wouldn't be as soon as you'd replace the one in your watch.
Actually it is far more difficult for a car to have accurate timekeeping than a wristwatch, as the various sources of noise, the CPUs, the injectors, the spark ignition, all contribute to clock drift. In a watch there is almost nothing to interfere with the clock, everything is encased in metal, and the only thing that is attached to the clock is the IC that drives the display and keeps time.
On the other hand, 3 minutes a week is fucking lousy for any clock, and any car with GPS should be setting the clock from the GPS signal. Any other car in the US SHOULD have a radio time receiver (which they could include for under a dollar), but I've NEVER came across one that does.