Ask Slashdot: Developer Responsibility When Apps Might Risk Lives?
First time accepted submitter bashaw writes "What ethical responsibilities do software developers have in determining the role that mobile devices take in our lives? As performance increases, size decreases, and the only limitation is the software available, mobile devices have expanded into new areas of our lives for which they were not designed. This raises the ethical question of who decides what software is available, and therefore what role these devices should take. I am a software developer at the Canadian Avalanche Centre. We recently issued a warning about mobile avalanche search applications that are marketed as avalanche rescue systems. Three smartphone applications are presenting themselves as economical alternatives to avalanche transceivers, the electronic device used by backcountry users to find buried companions in case of an avalanche. The applications are not an adequate replacement for an avalanche transceiver for many reasons, and we are concerned about the use of this software in lieu of a specifically-designed avalanche transceiver. When it is a question of public safety, does the onus fall on the developers, a government agency or the users themselves?"
Unless someone shows otherwise, the apps mentioned seem to do what the software developers who created them made them do. But the publisher of these apps tries to sell them for uses that they are not fit for. That's the publisher's problem, not the developers'.
App did not warn me about tornado.
Seriously, people have to take responsibility for their own choices.
We're too litigious nowadays; we ought to set the standard that grownups are required to think.
does the onus fall on the developers, a government agency or the users themselves?
Yes.
As soon as executives are financially responsible for the money they lose or swindle from customers.
The developer of the software has no say in release schedules, etc. You're way off dude...C level execs are the ones responsible...and the ones making all the money.
On the one hand, we can crack down hard on anyone who tries to even hint at some medical or safety purpose for a particular app. On the other we can be wild and free-booting and allow people into precisely the sort of trap that the poster outlines.
These apps may well be better than nothing (though they are not tested in any meaningful sense, nor are they compliant in any meaningful sense), but to the extent that they give a false sense of security, they are dangerous.
Personally, I lean towards crystal clear disclosure, and, in Canada, and restrictions on marketing. I do not favour an outright ban, since I could see that as having unpleasant consequences.
Look forward ten years. Suppose my smartphone has a ~90% reliable software and sensor package to tell me if I'm suffering from a heart attack. Suppose also that I'm part of a demographic group that by gender, age, fitness, weight, diet is highly unlikely to be suffering one. (There have been cases before where software has successfully diagnosed heart attacks in situations where physicians didn't believe it -- consider the case of psychologist Helen Smith a fit 37 year old woman who came close to dying since humans didn't believe she could be having a heart attack).
It would not make rational sense in that case for me to purchase a $1000 bespoke medical device to monitor me, but a $5 app might make sense even if it wasn't as reliable.
Similarly if I ski only occasionally and in areas highly unlikely to suffer an avalanche, it might make sense for me to not purchase a transceiver. (For those who say they'd spend anything to protect their lives, even on extraordinary low probability, I suspect you may have some irrational optimizations in your life.)
Offering consumers informed choice seems key; if they are marketing their apps as the equivalent of Avalanche transceivers, that clearly is not informed choice.
Similarly, I'd pressure Google and Apple and Blackberry to come up with a common standard for fine grid device location that these apps could use.
The OP raises some interesting points; I still come down somewhat on the libertarian side of things.
People themselves should be responsible for choices they make. In this scenario a smartphone app could act as a supplement but not a replacement for an avalanche tracker.
but let them go through otherwise.
As long as they're claiming "better than nothing" and not "as good as an avalanche beacon!" then I have no real problem with it.
We recently issued a warning about mobile avalanche search applications that are marketed as avalanche rescue systems. Three smartphone applications are presenting themselves as economical alternatives to avalanche transceivers, the electronic device used by backcountry users to find buried companions in case of an avalanche. The applications are not an adequate replacement for an avalanche transceiver for many reasons, and we are concerned about the use of this software in lieu of a specifically-designed avalanche transceiver.
Assuming they work to some degree its probably a matter of whether the buyer was accurately informed about their performance, not what the level of performance actually is. To be honest I expect that the users of such apps are those who would otherwise go into the backcountry with no device at all, are the apps better than nothing at all?
I would say the onus falls upon the user. In the absence of accepted standards or regulating entities I think it's buyer beware man. On the other hand, If you have standards dictating required operational parameters, then it's quite obviously the developer or designer I think. My post is over simplified and ignores the hardware/software integration question, but I think I'm on the right track. At least by slashdot standards. I welcome argument and rectification.
The headline reminded me of a story in a book of mine:
When Brunel's Ship the SS Great Britain was launched into the River Thames, it made such a splash that several spectators on the opposite bank were drowned. Nowadays, engineers reduce the force of entry into the water by rope tethers which are designed to break at carefully calculated intervals.
When the first computer came into operation in the Mathematisch Centrum in Amsterdam, one of the first tasks was to calculate the appropriate intervals and breaking strains of these tethers. In order to ensure the correctness of the program which did the calculations, the programmers were invited to watch the launching from the first row of the ceremonial viewing stand set up on the opposite bank. They accepted and they survived.
This post contains no rudeness or derision of any kind. All arguments are friendly. Terms and exclusions may apply.
This is the best story/discussion topic in a long while here on slashdot. I hope many chime in with their serious thoughts.
Fuck systemd. Fuck Redhat. Fuck Soylent, too. Wait, scratch the last one.
"13. Note on Java support. The software may contain support for
programs written in Java. Java technology is not fault tolerant and is
not designed, manufactured, or intended for use or resale as online
control equipment in hazardous environments requiring fail-safe
performance, such as in the operation of nuclear facilities, aircraft
navigation or communication systems, air traffic control, direct life
support machines, or weapons systems, in which the failure of java
technology could lead directly to death, personal injury, or severe
physical or environmental damage.
Is a parachute with a giant hole in it better than no parachute?
Interestingly, for round parachutes a parachute with a hole in the apex is better than a parachute with no hole at all.
As for your question, yes a parachute with a giant hole is better than nothing. It still slows you down and depending upon what you land upon that slowing may make a difference. There are occasional incidents where parachutists survive parachute failures.
If the developer is the owner of the software and has provided warranty for the software and/or as a selling points, uses the accuracy and reliability of the software and, assuming this, that the users of the product are following all the rules and guidelines and the software fails.. then, there could be a point towards this. So, this becomes a question of what are you selling, how much are you willing to back it up and finally, if there are any disclaimers, are they easily understood? Nothing is infallible. Considering the platforms being used, etc.. It's hard to make anything fool proof, so, how much QA was done? and are people using the product as it is intended on the devices it is meant to work on? And let's talk a professional software with proper SDLC. That means there is QA, UAT, PROD, etc.. It's not just the developer. So, this can easily be a grayish area.
Why do you hate the free market?
Everybody can make their own standard and just let the consumers sort it out (or not).
I don't read your sig. Why are you reading mine?
Ok, I am not a lawyer, and I am not a Canadian, but as far as I can see there are a few different questions:
Where I am (the UK) criminal liability is likely to lie with the company that developed the product (or that sold it, if it was misrepresented). If they are selling safety-significant software, they should have appropriate processes in place to ensure the software is of sufficient quality and to ensure that developers are working to those processes,
For "Who can get sued", the answer is "almost anybody", and it can be financially ruining, but as far as I can see a customer who suffers harm is in the first instance likely to go after the company that sold the app and the company that developed the app -- the company would have trouble passing the responsibility down to the developer if they did not have such processes and if they did not make them sufficiently known to the developer. That might not stop them trying, though, if they're desperate to pass the buck.
For "Who can get successfully sued, that depends on local legislation and how the product is represented. It's probably not you, but as I said, I am not a lawyer.
For "moral obligation" I would say that you have an obligation to raise your concerns with your management, and whatever their response to be aware of, and as far as is in your power apply, the appropriate measures that should be applied to safety significant software -- if you can get hold of a copy of IEC61508 or local standards and guidelines it would be a big help.
Quidnam Latine loqui modo coepi?
That depends.
How big of a hole in relation to the surface area of a parachute do you consider giant?
There are always (and more so in modern times) people that cannot adequately estimate risk and will do stupid things without adequate protection. If some app-makers bank on that and claim properties which are not true or gross exaggerations, by all means charge them with involuntary manslaughter when their trash kills somebody. But the app-makers are only opportunistic parasites here, the real problem is people grossly overestimating their own skills. Just let them do it to themselves, its evolution at work.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I think the developers are ethically responsible for being crystal clear about the capabilities, reliability, and limitations of what is produced. They are responsible for reasonably thorough testing of any claimed capabilities on the hardware/OS combos it is stated to run on.
I think it is also a question of reasonable liability. In the case of liability, I would say it depends. As stated earlier consumers need to be aware of the reliability of the application, its capabilities, and limitations. Developers should be liable for harm caused by major flaws (unlike most current EULAs) or false claims.
Obligatory XKCD:
http://xkcd.com/937/
In the case of this tornado guard, if the app doesn't actually have any functionality to warn the user of nearby tornadoes, then the consumer should be able to hold the developer liable. If it does, and the situation in question was just beyond the stated capabilities of the app, then no, the developer should not be liable.
Government involvement should consist of enforcing accurate marketing and enforcing penalties and civil liabilities when things don't work as advertised. If a government is going to develop or use a safety critical app, they should thoroughly test it, and demand source code access in the case of commercial applications.
It's alarming how there's a certain group of people who are scared of the "danger" of radiation from mobile phones, when in fact mobile phones can save lives -- 911 calls, emergency alerts, and now apps that may help folks out of an avalanche.
For now the focus needn't be on which apps are best for various purposes, but on providing cellular service globally, anti-radiation crackpots be damned.
There's no -1 for "I don't get it."
Avalanche beacons definitely help. And the technology has dramatically improved in the last twenty or thirty years. They are definitely easier to use and much more reliable nowadays.
Having said that, there are some fairly serious limitations on how well they can work. From what I remember of avalanche school, about fifty percent of avalanche victims die of traumatic injuries during the event, so obviously beacons can't help in those situations (my own personal experience in an avalanche included the tail of my ski hitting me in the forehead while still attached to my boot). But for the remaining fifty percent of victims who are likely to suffocate within minutes unless found avalanche beacons are an invaluable tool.
There is, of course, much that can be done to improve them. But that is another story.
I have my own company, with 1 business partner. We write software for 911 dispatch systems. Some of our clients require us to carry "Errors and Omissions" liability insurance, which costs us upwards of $15k a year. Along with with the insurance, we have a pretty detailed EULA agreement covering bugs, etc.
If you're writing any kind of software that could directly affect the safety of others, insurance is a must.
Frog
This is from May of 2012...
http://avyupdates.blogspot.com/2012/05/iphone-avy-beacon-app-jokes-on-you.html
Facebook, foursquare, and just plain old texting, all impact people's safety because of their improper use, mainly by drivers. Are we going to go after texting app developers when people get hurt while using such software?
It falls on you, assuming your license sucks, but just make sure your Professional Liability coverage handles that sort of thing. I make search-and-rescue alerting software for the aerospace industry, my premiums are about 33% more.
...does the onus fall on the developers, a government agency or the users themselves...
If by "the developer" you mean the company that sells the product, then the answer is clear: it is the developer that is responsible. The government can at best issue guidelines for how safe and fit for purpose a product should be, and it is not realistic to expect all users to understand the full ramifications of their choices, when it comes to things that are likely to be well outside their general competencies.
On the other hand, if by "the developer" you mean the engineer who designed and coded the app, then I would say not so much, but it depends on the circumstances. In general, software is not sold directly by the developers; there is a business organisation between, that decides what to produce and how to sell it. They are far more likely to be the real culprits.
So what happens when you write a marco polo game using GPS and then a month later you see that some jackass in marketing has claimed it is the perfect way to find someone buried in an avalanche?
I think you are confusing me with someone else. I'm not necessarily a libertarian and I have already stated in another thread you have replied to that I think government should regulate it.
However, the begger in Soviet Russia doesn't need to know anything about the US. Soviet Russia should have the laws in place too. This system is internationally recognized the world around and is in use in Russia too. But a lawsuit over international persons or shell companies only gets complicated, not impossible.
You are right, people can game the system, fake bankruptcies and all that. But that would only embolden government to take action for criminal charges and the app stores would likely end up banning the apps before then to avoid vicarious liability.
That's the way the legal system works. There is liability in EVERYTHING. You carry E&O insurance so you don't lose your house.
I hate being bipolar; it's awesome!
Stalled in the Mojave desert 106F in the heat of the day, a 2011 VW decided all three keys were ' not in range'. Two hours later I reached water, tow truck and ride 67 miles to civilization. It remains to this day whether the memory was wiped clean from extra-terrestial sunspot activity, fault ECU or programmed to lock-out the car at specified date/mileage.
Lexus' judgement renders the ECU behavior to be the vehicle ' owner' responsibility in California. So there's definitely a need to get tools, connections and programs on our cellphones to view logs at the very least. Programmer's are paid to build functionality. The liability unfortunately seems to rest with the end users.
Snotty, over-priced stock software corporation + millions of product copies in the field + deaths = lawyers = guess what, your stock becomes like a car company's.
There are ways to deal with this, but it involves massive process and redundancy and code reviews and design reviews and detailed checks of Lint, QAC, Polyspace, a dozen other checkers, software watchdogs that, by the way, damned well never actually be needed, etc.
Ya better put down the keyboard and get some training for a few months.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
I. Fundamental Canons Engineers, in the fulfillment of their professional duties, shall:
Hold paramount the safety, health, and welfare of the public.
Perform services only in areas of their competence.
Issue public statements only in an objective and truthful manner.
Act for each employer or client as faithful agents or trustees.
Avoid deceptive acts.
Conduct themselves honorably, responsibly, ethically, and lawfully so as to enhance the honor, reputation, and usefulness of the profession. http://www.onlineethics.org/Resources/ethcodes/EnglishCodes/9972.aspx
what about where the is a chain of Contracts / subs where they all just pass the responsibility on.
In Canada, under the various provincial acts (and a National act that keeps them largely consistent), professional engineers (note, the word "engineer" is legally protected in Canada, like Medical Doctor or Lawyer, unlike in the US.) must do any work that involves human safety. That INCLUDES computer/technical related work. The classic example is software for air traffic control systems or software on space shuttle modules.
One of the problems for the engineering regulatory bodies (Professional Engineers Ontario - PEO - in the case of Ontario) is that many companies don't employ computer/software engineers even when their software involves human safety. They use computer science majors, or people with 1 year technical diplomas from the local college, or people with Microsoft or Cisco courses, or whoever happens to know whatever programming language they are using. The companies are legally required to have the work reviewed and signed off on by licensed engineers, but they just assume "oh, it's not like software is like a bridge or a building or something", so don't realise that the engineering priciples are no different than those used in structural engineering. Where it becomes even more fuzy is that the laws also state that licensed engineers must be used when "financial welfare" is on the line. Very few banking systems are properly designed by licensed computer/software engineers...
Source: I'm a professional engineer (P.Eng) registered in Ontario. Related legislation in Ontario:http://www.e-laws.gov.on.ca/html/statutes/english/elaws_statutes_90p28_e.htm - Professional regulatory body in Ontario: www.peo.on.ca
Do you really know so little about your hero Ayn Rand that you totally missed the reference?
Yes, I do know so little about Ayn Rand. What is your point? I already said I'm not the libertarian you think I am.