Slashdot Mirror
Will Unpredictable 'Franken-Algorithms' Have Deadly Consequences and Make Programmers Obsolete? (theguardian.com)
Zorro (Slashdot reader #15,797) summarizes a new article in the Guardian:
The death of a woman hit by a self-driving car highlights an unfolding technological crisis, as code piled on code creates "a universe no one fully understands."
"In some ways we've lost agency. When programs pass into code and code passes into algorithms and then algorithms start to create new algorithms, it gets farther and farther from human agency. Software is released into a code universe which no one can fully understand."
The author dubs these man-made monsters "franken-algos," since "After a time in the wild, we no longer know what they are: they have the potential to become erratic." Self-learning algorithms are already part of the "new all-machine phase" of Wall Street trading, leading to what science historian George Dyson believes are rules "where nobody knows what the rules are: the algorithms create their own rules -- you let them evolve the same way nature evolves organisms."
Where does it end? There's already a robotic sharpshooter policing the demilitarized zone between North and South Korea, and "swarms of coordinated, weaponized drones" already being developed by three different countries. The article suggests re-thinking our legal system to assign blame for any badly malfunctioning algorithms, noting that the Association for Computing Machinery recently updated its code of ethics "along the lines of medicine's Hippocratic oath, to instruct computing professionals to do no harm and consider the wider impacts of their work.... Solutions exist or can be found for most of the problems described here, but not without incentivizing big tech to place the health of society on a par with their bottom lines.
"More serious in the long term is growing conjecture that current programming methods are no longer fit for purpose given the size, complexity and interdependency of the algorithmic systems we increasingly rely on." Toby Walsh, a professor of artificial intelligence at the University of New South Wales, even says "We will eventually give up writing algorithms altogether... "because the machines will be able to do it far better than we ever could. Software engineering is in that sense perhaps a dying profession."
"In some ways we've lost agency. When programs pass into code and code passes into algorithms and then algorithms start to create new algorithms, it gets farther and farther from human agency. Software is released into a code universe which no one can fully understand."
The author dubs these man-made monsters "franken-algos," since "After a time in the wild, we no longer know what they are: they have the potential to become erratic." Self-learning algorithms are already part of the "new all-machine phase" of Wall Street trading, leading to what science historian George Dyson believes are rules "where nobody knows what the rules are: the algorithms create their own rules -- you let them evolve the same way nature evolves organisms."
Where does it end? There's already a robotic sharpshooter policing the demilitarized zone between North and South Korea, and "swarms of coordinated, weaponized drones" already being developed by three different countries. The article suggests re-thinking our legal system to assign blame for any badly malfunctioning algorithms, noting that the Association for Computing Machinery recently updated its code of ethics "along the lines of medicine's Hippocratic oath, to instruct computing professionals to do no harm and consider the wider impacts of their work.... Solutions exist or can be found for most of the problems described here, but not without incentivizing big tech to place the health of society on a par with their bottom lines.
"More serious in the long term is growing conjecture that current programming methods are no longer fit for purpose given the size, complexity and interdependency of the algorithmic systems we increasingly rely on." Toby Walsh, a professor of artificial intelligence at the University of New South Wales, even says "We will eventually give up writing algorithms altogether... "because the machines will be able to do it far better than we ever could. Software engineering is in that sense perhaps a dying profession."
Someone's got to fix it when it goes wrong.
Typical click bait.
What do you call the person who gives the franken-algorithm a specific list of instructions to accomplish your goal? A programmer.
Translation: "My work is shoddy, but potentially profitable. Exonerate me for cutting corners on research because I have cheap Tensorflow CPU cycles."
Why not. Both prefix Franken and suffix gate are so heavily overused by idiot Americans.
Probably unavoidable. Our IRL universe is unpredictable, chaos and no-one fully understands it. Our own body mutates in undesirable ways, we suddenly get cancer. This is the way of the universe. As we get closer and closer to some sort of AI indistinguishable from real life I suspect this is an organic consequence.
Are we talking about a program, ... ...or the legal system?
I swear -- I read this, and I can't help but think about the legal matrix we live in.
For sure!
That's exactly what I think about our legal system. Who understands how the city works? It's a byzantine maze of interests, legislators, boards, bureaus, ... I think I could easily spend 8 years trying to understand How Seattle Works, and by the time I was done, it'd already have moved.
Can I just call them "cities?"
Self-learning algorithms... ...= people?
I don't see how that's all that different from, say, having the US military. What's the US military up to right now? I wouldn't be surprised if we're involved in some war or another war right now, and it just doesn't hit the headline news.
Now there's irony. "This horribly confused not-understandable block of code, shall be made accountable to this other confused not-understandable block of code." The ACM will putt to ... whatever legal associations exist. I suppose that they'll putt to the Constitution or something. And the Constitution will putt to God, or something.
Good luck with that...
"We will eventually give up on self-governance, because the government machine is able to do it far better than we ever could."
I don't know if this is good or bad for software engineering.
Might want to look up the facts of this case. Uber had disengaged the "stop if something's in front of you" aspects of the self-driving car's code, and the woman who was responsible for taking over in such a situation was watching a reality show on her phone at the time of the incident. Nobody even tried to stop this car.
The code won't be debuggable, so need humans to blame and fire.
There's a real threat these days of big companies and naive developers pushing machine learning as a panacea. Really it's more like a potent mutagen escaped from lab containment into your other work. People trying to obsolete expertise with poorly thought out and even more poorly tested applications of machine learning are definitely going to continue to cause deadly consequences in and around self-driving vehicles.
Someone has to program the initial version.
who better than Java programmers?
forget all those slow, bug-laden, inefficient, crappy languages like C/C++ or ASSembly. Java is where it's at.
Java is fast, and each iteration of a function or program makes it faster!
Forget stupid stuff like IEEE-754 compliance, ditch the hard stuff and focus on performance!
Java is where it's at.
This pretty much describes me trying to decipher my code two weeks after I wrote it. And I don't even program in Perl.
We just need to hook up those monkey brains in the jars with some jumper cables and we'll never need programmers again.
Sig. Sig. Sputnik
Well then, it sounds like there WAS a person who knew exactly what the "Franken-Algo" was doing. GUILTY!
Will Unpredictable 'Franken-Algorithms' Have Deadly Consequences
Probably yes.
and Make Programmers Obsolete?
Almost certainly not.
We are creating algorithms where the result can not be explained in human terms. Nobody can truly understand why AlphaGo thinks a move is good, it's a neural network of weights we don't understand. It's about as useless as trying to get a chess grandmaster to articulate why a particular move is good, it's subtleties you can't record and put in a rule book. Which is fine for AlphaGo since the worst it'll do is lose a game. If it's Watson totally misdiagnosing your cancer or Waymo's car T-boning a school bus it matters a lot.
That is why I think developers will always be busy implementing guard rails. Like if you're trying to minimize humanity's environmental impact then the divide by zero solution is obviously superior. It's not a practically feasible solution in the real world though.
Live today, because you never know what tomorrow brings
We've been a long time now running code whose actions are not knowable to the person trying to run it. How long has it been since it really made sense to ask "did the person at the computer authorize this action?" with all the bits of code from All Over The Place doing things nobody is told about.
Now we are hearing about self modifying code (hmm...where've I heard of that before?) that is less predictable.
If you want to hold anyone responsible, best figure how to tell who is starting some code, and how you propose to enforce that the poor sod at the keyboard might be able to tell what his actions will do. We have ZERO insistence on getting that information, and have had none for decades now.
Doesn't insisting on assigning blame, and looking at legal liability, sound like magical thinking then, or like not thinking at all?
"Do you try to implement the code contained in the first link google gives you for your search?"
I'm guessing about 80% of the code tossed into franken-vehicles is directly downloaded by shoddy/H1-B/lazy programmers, who haven't ever written a line of code or documentation for a production envuronment (no marter what their CV says.)
*cringe*
The answer is, "Yes, there will be deadly consequences and programmers will be obsolete." There's nothing you can do about it. Now make yourself a nice cold drink with some rum and fruit juice or something and go grill a piece of meat.
You can't change things, so you might as well enjoy your holiday weekend. Maybe things will look different on Tuesday, but for now, don't sweat it and go outside. It's nice outside, and summer's almost over.
You are welcome on my lawn.
Toll the Great Bell Once!
Pull the Lever forward to engage the Piston and Pump...
Toll the Great Bell Twice!
With push of Button fire the Engine And spark Turbine into life...
Toll the Great Bell Thrice!
Sing Praise to the God of All Machines
It use to be that we used requirements and design to create algorithms and well defined and comprehensive tests to evaluate them. That changed when companies began hiring cheaper programmers to replace more expensive engineers, and now algorithms are created by the cerebal equivalent of vomit which is swallowed and revomitted until it works well enough to fool the agile team leader. You get what you pay for.
Software has generally come with "not suitable for any specific use" clauses in the warranty for 40+ yrs.
When human lives and puppies are involved, that just doesn't cut it. Passing the liability buck is NOT acceptable.
So, which idiot insurance company will insure these software products against all faults?
Because that would be awesome.
Pretty sure monkeys all prefer tabs. Are you really going to attach that hot mess to crucial systems?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
We're doomed as doomed can be. - Ed Grimley
It's the best of all possible worlds. - Leibniz/Pangloss.
"National Security is the chief cause of national insecurity." - Celine's First Law
The referenced strategies only come into play to accomplish tasks that were pretty much out of reach of traditional programming. Essentially a last resort. For the problem set that has been feasible for programmers to tackle, it almost always remains the better way.
Further, it's really about bringing taming complex, chaotic, unstructured data into a structure so that programmers can address it. Generating these routines never speaks to how to apply the approach to solve a problem. Human's are required. As far as I have seen, once the image recognition or similar trained system is ready, it needs a programmer to actually do stuff with the resultant model.
XML is like violence. If it doesn't solve the problem, use more.
So-called 'AI' has been so over-hyped by development companies and their marketers (because they need to show ROI or get their heads chopped off), the news media doesn't have a clue how anything actually works and they're amplifying the hype, then entertainment media (TV, movies, even books) present these fantasy images of 'AI' technology that doesn't exist (and might never exist), and shockingly enough, people believe what they see hear and read. The long-term result of this, left unchecked, will be people actually believing that these 'algorithms' masquerading as Artificial Intelligence are capable of far more than they actually are, resulting in financial disasters, property damage, and loss of human life. Meanwhile the programmers that create these half-assed machines can't even tell you what's going on 'under the hood' when the thing's running, and can't really explain why it does what it does when it screws up. I for one will be glad when the current crop of so-called 'AI' they keep trotting out is shit-canned.
—employs a rarely seen strategy of “code reuse”.
“Don’t fly on payday.” — Wally
Which would free us up to vacation in our flying car and date Rosy the Robot.
Table-ized A.I.
The hallmark of a crap article.
We will eventually give up writing algorithms altogether...
Automated code generation. Automated test generation. Automated test coverage. Been there, done that. Twenty years ago. The people who say 'eventually' are the ones with a vested interest in selling meat sack coding labor to customers.
You think it's unpredictable and not to be trusted? Better not fly in a modern airplane.
Have gnu, will travel.
I work for an AI startup. We are hiring more programmers. All models operate under assumptions, requiring programmers to give them accurate data. Garbage in - garbage out. All that's becoming obsolete are programmers who don't want to learn new things.
It's the engineers that seem to have a hard time wrapping their mushroom heads around reality. The aspersions they cast on others and lame-assed, 'It's the tech's fault!', is getting pretty old, too. This is ludicrous, they are ludicrous, and I can't help but notice that these whiny ass trends perfectly coincided with the introduction of millennials into Silly Valley. Give us all a break, already.
The "get it out the door and monetize it" business model of software is a problem.
You are literally paying for software that is in development, until it's not. Patches, updates, big and small.
But hey, the development manager needs to get his bonus, and so does his boss, and the CEO, and the shareholders.
I still can't believe after decades of this shit we still put up with it.
The result is perfectly predictable. The algorithm will eventually determine that the correct answer is to hit the problem with a stick. All the answers will be "The Stick" TM. Why, because humans still use "The Stick" and what do you think the algorithm will be seeded with? What Would A Human Do? You already know, hit it with a stick.
And no, the stick does not have to be an actual stick. Sometimes the carrot is really the stick. Sometimes words are the stick. Sometimes the stick is a stick.
This is really just a continuation of the "software crisis", the discovery of human error when programming machines. Human error occurs on all levels of the software process, there is no "silver bullet" for fixing conflicting requirements.
We should not "incentivizing" big tech to pass the risks on to the public. QA is the boring stuff that involves regulation, redundancy and statistical modeling, of course the fancy Internet companies want to pass the responsibility for QA to end users with eternal Beta versions and EULAs filled with legalese.
Only if by Franken algorithm they mean some doomsday device with autonomous killing robots that drive humans into extinction. Programmers may some day become obsolete, but the death of programmers will be nothing so vague as the article.
It was actually a man lol
Ridiculous statements delivered by an evidently programming-ignorant person aside, I have an anecdote which might help sensible people without a too deep understanding about all this to get the real implications of ideas on these lines (= absolute impossibility).
Although I have lots of experience in automated data understanding, numerical modelling, even in coming up with relatively complex algorithms taking care of a wide range of situations, my expertise on the image side of things is quite limited. More specifically, I haven't worked too much with the approaches which are more widely used to deal with these problems: neural networks. I mean that I haven't gone too depth into all these methodologies like properly understanding how (the complex versions of) their algorithms work or spending a relevant amount of time on tuning them up/coming up with good models. In fact, I have always tried to avoid this kind of trendy, theoretically-easy-to-use approaches which usually require you to spend a relevant amount of effort to understand everything properly (basically, new names and concepts to perform pretty much the same kind of calculations that many other methodologies can do) and rarely are comprehensive and adaptable enough, not more than older and more solid methodologies together with a good deal of personal experience.
Despite the previous paragraph, I am a very reasonable person always ready to update my assumptions when required. It is quite clear that all the work done in NN during the last years have allowed these systems to deliver a quite good performance under conditions like image analysis. So, I have been spending some time during the last weeks to gain a better insight into this sub-world, from both the algorithm development and parameter tuning sides. One of the first things I observed was that efficiency wasn't precisely a main concern for some of the most popular software packages; this is a relatively common situation nowadays, but here might become particularly problematic by bearing in mind the fact that some of these simulations might grow really complex/slow. Precisely improving efficiency is one of my strongest suits and that's why I decided to focus this first deep contact on that aspect.
It is still work in progress, so I will not talk too much about the specific details. The basic idea is that it is a popular, open-source package dealing with NN. It relies on well-documented theories and its algorithm, although pretty complex, is reasonably well commented too. All this together with my extensive experience on (efficiency) algorithm improvement (not too much on that specific programming language, although this issue isn't too relevant on account of my expertise and the myriad of available resources) seems to indicate that this should be a relatively easy endeavor. Logically, I had done a pre-analysis concluding that a relevant performance improvement was possible by even locating the parts to be modified. But the reality has been way different than expected...
Even after perfectly understanding the underlying theory, the main parts of the algorithms to be modified and having done a relevant amount of debugging/tests, the final solution isn't still quite there. The reason? Too complex, highly-abstracted code and (also complex) external dependencies. Similarly to what happens with many modern pieces of software, it was built to be used under certain conditions and together with certain resources; theoretically scalable and easily modifiable, but practically linked to a set of specific conditions. The whole approach didn't take efficiency into consideration since the start and, consequently, any relevant improvement on this front implies a big effort and essential modifications. Or, in other words, optimising that code by following its defining guidelines might be relatively easy, even automated in some ways; any other change can be very complex. Applying a set of more or less defined instructions is easy/automatisable, but creating those instructions or performing re
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
Stepped out a number of seconds before, such that the SDC should have been able avoid her, given it was well lit.
Can you imagine even the smartest speculative fiction AI dealing the typical "I don't know exactly what I want, but it's something like this: and I need the demo ready in a couple of days for a sales presentation."
If anything sets off the AI revolt, it would be dealing with today's mangers and marketing morons.
Question more what you read
News for nerds? No, I guess it's opinion for the scared.
"If you design a system that doesn't log out what it does in a matter that can be understandable in it's most core or complex of events, you have failed design 101."
- Neruos 2001
The article basically states that humans are too stupid to understand complex algorithms... and robots probably say the same about human behaviour! Interestingly, most crashes of self-driven cars are down to old fashioned human error.
The answer is and was always 'NO'!
The number of seconds was less than 2. But more to the point, Uber's programmers knew exactly what they were doing. They changed the default action when the car detected a possible obstacle from "stop" to "return to manual human driver control." Probably because they couldn't get the car to quit constantly stopping over a huge number of false alarms, and lacked the skills to make an algorithm which could reliably sort them out.
So no matter how you slice it, that incident actually contradicts the thesis of the article, which is that the accident happened because nobody knows what the code does.
When a journalist writes about stuff he doesn't understand.
It was actually a man lol
Nobody knows what it really was, it was unusual for sure.
No, not unpredictable, unless you don't understand how AI and neural networks work!
Are there edge cases where input generates unexpected results? Absolutely! And that's no different from any kind of programming since the computer was invented.
For that matter, when we used to ride horses everywhere, there were edge cases that caused horse brains to freak out, such as gun shots nearby, or the sight of a wild animal. That didn't stop people from making safe use of horses for transportation, with accidents occurring only once in a great while.
The key was to learn to understand and anticipate the kinds of things horses would react to. Likewise, with time and testing, we will learn to anticipate and compensate for things that make AI "freak out." It's certainly not unpredictable.
For most significant consumer electronics in the U.S. we can clearly see how the complexity and layers of software cause systems to exhibit unexpected behaviors all the time.
Most of the software we use for business seems to change functionality at some point, or stop doing something we used for no apparent reason.
Distributed systems which handle mission-critical data can run for months or years and then suddenly start doing something odd.
Your mobile phone and most newer cars will do something odd or unexpected at least once a week.
Adults who have seen this over the last 20 years know that you do not use/touch/install/configure anything on a computer unless you have no other choice.
That horse left the barn years ago. Its sad all the sheep are just realizing technology is messed up and not really helping We The People.
Make America Great Again: Eliminate all politicians with blunt force trauma!
Franken-Algorithms? Just wait until they are used by law enforcement and the court system which is already starting.