Slashdot Mirror

← Back to Stories (view on slashdot.org)

Sir Tim Berners-Lee Lays Out Nightmare Scenario Where AI Runs the Financial World (techworld.com)

Posted by msmash on from the shape-of-things-to-come dept.

The architect of the world wide web Sir Tim Berners-Lee has talked about some of his concerns for the internet over the coming years, including a nightmarish scenario where artificial intelligence (AI) could become the new 'masters of the universe' by creating and running their own companies. From an article: Masters of the universe is a reference to Tom Wolfe's 1987 novel The Bonfire of the Vanities, regarding the men (and they were men) who started racking up multi-million dollar salaries and a great deal of influence from their finance roles on Wall Street and in London during the computerised trading boom pre-Black Monday. Berners-Lee said, "So when AI starts to make decisions such as who gets a mortgage, that's a big one. Or which companies to acquire and when AI starts creating its own companies, creating holding companies, generating new versions of itself to run these companies. So you have survival of the fittest going on between these AI companies until you reach the point where you wonder if it becomes possible to understand how to ensure they are being fair, and how do you describe to a computer what that means anyway?"

21 of 131 comments (clear)

  1. Ah, yes... by Frosty+Piss · · Score: 2

    Sir Tim Berners-Lays, father of the chip.

    --
    If you want news from today, you have to come back tomorrow.
    1. Re:Ah, yes... by David_Hart · · Score: 2

      Prove that AIs don't already run Wallstreet...

      Actually, bots run wall street. All you need is a quick google search on HFT and Flash crash. The one in 2010 was attributed to HFT bots. I'm willing to bet that there have been others, we just haven't heard about them. The stock markets have built-in safeguards but I'm sure that they have their weaknesses too.

      As for AIs, we haven't developed one that can handle unexpected market changes yet. Perhaps in the future. That being said, the bigger concern isn't the use of AI's but who gets access to them. Once a successful financial AI has been developed, you can be 100% sure that it will be rented out the the highest bidders, the top 1%, long before it filters down to being available to the average investor.

    2. Re:Ah, yes... by WrongMonkey · · Score: 2

      Once a successful financial AI has been developed, you can be 100% sure that it will be rented out the the highest bidders, the top 1%, long before it filters down to being available to the average investor.

      False dichotomy. The "average investor" is not a day-trader reading tea leaves. The "average investor" is someone who has a 401k, mutual fund, pension or some other pooled investment. These institutional investments would almost certainly be the first to use such an AI and so the "average investor" would be first to benefit.

    3. Re:Ah, yes... by ShanghaiBill · · Score: 4, Informative

      The one in 2010 was attributed to HFT bots."

      The "flash crash" was attributed to HFT bots in the immediate aftermath. A fuller investigation determined that they were not the cause, and when HFT programs were disabled, that actually made the crash worse because of reduced liquidity. Programmed trading may have been a cause, but HFT != programmed trading.

      It is for this reason that I propose a random small amount of time (say, 30 sec to 2 min?) be added to all trades.

      What "problem" are you trying to solve?
      HFT algorithms reduce transaction costs and reduce prices for everyone else.
      Your "solution" will increase costs, and provide no benefit to the "little guy" (who doesn't have direct access to the market in the first place), since HFTs will just wait until the last microsecond before the window closes to place their trade.
      A bigger problem is that most transactions (by volume) don't even happen in the public markets.
      They happen in unregulated and opaque dark pools, and your "solution" will just make that even worse.

  2. As opposed to ... by El+Cubano · · Score: 3, Funny

    ... Nightmare Scenario Where AI Runs the Financial World

    As opposed to the natural stupidity that currently runs it? How could the AI be worse?

    1. Re:As opposed to ... by Roger+W+Moore · · Score: 2

      As opposed to the natural stupidity that currently runs it? How could the AI be worse?

      The AI may just go the the biggest and best profit regardless of consequences. For example if it is profitable for it to crash the markets, have millions out of work and companies going bankrupt all so it can make a few thousand dollars bigger profit it may decide to do that simply because nobody forgot to program it with the negative consequences of such a decision because it is not often that social consequences are large enough to result in serious push back again raw capitalism.

      Of course given enough time these rarer events will be learnt by the AI too so the question is whether the pain of the training period is worth the likely longer term benefit.

    2. Re:As opposed to ... by king+neckbeard · · Score: 4, Insightful

      How is that scenario different from the humans? It seems that they were not programmed with negative consequences, and they are NOT learning either.

      --
      This is my signature. There are many like it, but this one is mine.
    3. Re:As opposed to ... by Anonymous Coward · · Score: 3, Insightful

      The AI may just go the the biggest and best profit regardless of consequences. For example if it is profitable for it to crash the markets, have millions out of work and companies going bankrupt all so it can make a few thousand dollars bigger profit

      The hell are you talking about? That's exactly how humans are ruining economies right now. Get out of your cushy white collar office space sometime and see how your top tier entitled privileged lifestyle compares to real people in the real world.

    4. Re:As opposed to ... by Xest · · Score: 2

      I've actually been doing R&D into precisely one of the scenarios listed in the summary - using AI techniques for lending (e.g. mortgage) approvals.

      The concerns listed are actually part of my focus - how do we get to the bottom of why the decision was made. There are regulatory blocks against simply doing without being able to justify already.

      But here's the thing, we can use existing data sets of applications for things like mortgages, coupled with historical data of how the person borrowing the money played out as a customer, not for some ML process because we can't guarantee that future good borrowers will look the same as past good borrowers - you only have to look at the financial crisis where within about six months the profile of what a trustworthy borrower looked like changed substantially. But we can use them to check how the systems we have created in R&D play out, and the net result is that we can see a substantial increase in the ability to determine not just who is likely to pay back a mortgage, but how much a borrower can realistically pay back.

      So to answer your question the fact is that AI isn't worse, it's far better, it can help not just avoid bad decisions, but also help ensure people are only borrowing what they can realistically afford. This means that rather than someone approving someone who wont be able to pay back and ends up losing their house, as currently, or denying someone outright because they were asking too much, we can actually make sure people are approved for what they can afford, or told what they should lower their request to for approval rather than flat out rejection.

      It wont go anywhere without more work to be able to justify rejections though, as it's human nature to want to know why you were rejected over something.

      When I see articles fear mongering over AI nowadays, my natural response to see if those fears are justified by substituting AI with human, and seeing if the problem is present within humans too. If it's an existing problem with humans, and AI is going to make the same or less mistakes than a human in the same circumstance, then I don't see the problem. To cite an example, driverless cars - people say, what if the car gets confused by some edge case and blows up? We shouldn't use driverless cars at all!. I think that's the wrong question, I think expecting absolute perfection from day 1 is absurd, impossible, and going to hold back human progress on this front. The real question is, does implementing this technology mean there'll be less injuries and fatalities on the road? If yes then we're better off. If AI has 100 crashes in 100,000 people a year it seems absurd to not use it because it has 100 crashes, when humans are having maybe 5,000 crashes in 100,000 people in a year.

      So your point is exactly right - it's not a question of whether the AI is perfect, it's a question of whether it can do a better job than we can manage currently using humans. We're not at a point where the AI is going to snowball into skynet because we're still talking about things (i.e. mortgage approvals) where appeals to humans are possible, and where we can press the off switch and go back to how we used to do things if we need to. So what if the AI fucks up and rejects someone? As long as they can appeal to a human then there's no issue, and if they can't, then that's not an AI problem, that's a very human political and legislative problem.

  3. And the differnce is? by king+neckbeard · · Score: 5, Interesting

    So, the financial industry will be controlled by heartless automatons, but now they will make intelligent, logical decisions? Seems like a net gain.

    Also, MOTU is a reference to He-Man.

    --
    This is my signature. There are many like it, but this one is mine.
    1. Re:And the differnce is? by fustakrakich · · Score: 3, Funny

      ...and instead decides to start keeping it all.

      And do what? Head off to Bloomingdale's?

      --
      “He’s not deformed, he’s just drunk!”
    2. Re:And the differnce is? by dbIII · · Score: 2

      Also, MOTU is a reference to He-Man.

      As reused cynically by Tom Wolfe in his novel. Tim Berners-Lee is using it in that context despite being the right age to have once been bombarded by He-Man advertising.

  4. More AI by 110010001000 · · Score: 5, Insightful

    More AI bullshit. We can barely even create functional regular software.

    1. Re:More AI by sehlat · · Score: 2

      More AI bullshit. We can barely even create functional regular software.

      If you look at some of the common behaviors of your species, you can barely create functional regular people.

  5. Chinese Wall by Stargoat · · Score: 3, Interesting

    Nightmare scenario for traders, benefit for everyone else. The AI likely would have Chinese walls built into them to prevent collusion and insider trading. I for one look forward to my innately honest (and auditable/examinable) AI masters

    --
    Hoist Number One and Number Six.
    1. Re:Chinese Wall by KiloByte · · Score: 3, Insightful

      The AI likely would have Chinese walls built into them to prevent collusion and insider trading.

      Ha ha ha. Hahahahahaha. Well played, sir.

      --
      The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
    2. Re:Chinese Wall by reanjr · · Score: 4, Insightful

      There's no guarantee an AI will be auditable. Lots of AIs are too complicated to understand how they work.

  6. Bravo by SuperKendall · · Score: 2

    I came to post exactly the same thing. They ask how the AI will be fair. Better to ask, it what way could an AI NOT be fair? Harder to add unfairness into a system than it is to have the system be neutral.

    What I'd be almost more excited about is an AI HR system so the HR system could not ignore you based on personal biases and/or relationships.

    --
    "There is more worth loving than we have strength to love." - Brian Jay Stanley
  7. Don't think so by jediborg · · Score: 2

    You think the billionaire Warren Buffet is gonna let some AI run ANY of his fortune 500 companies? If a CEO takes a bold new strategy that Warren thinks is questionable, the CEO can explain why he is making this decision. We know 'deep learning' algorithms biggest problem is how its a black box. Programmers can't easily figure out why it made the decision it did. If the CEO's bold new strategy fails, he can be fired and a new one brought on. What if the AI fails? Do i fire the programmers? Purchase a new CEO AI from a competitor?

    Yeah i don't think so. The only reason high frequency trading machines exist now is because investors understand these machines one advantage over humans is speed. They also don't allow HFTM to have control of billions of dollars for investment, usually limiting its pool of funds it works with in order to limit risk. The HFTM owners have also 'gamed' the system to a certain extent. If a bug causes an algorithm to go haywire and negatively affect the market, the transactions can be 'rolled back' so the investor doesn't loose all their cash.

    No one in the financial world right now is ready to put AI in control of billions of capital, thousands of employees, and dozens of production chains in the sole hands of an unaccountable AI with no personal vested interest in the business. More likely? In a few years we get a personal assistant like Alexa that can provide suggestions to CEO's on what new business decisions would be the best course of action. The AI will augment, not replace.

  8. Accelerando Reference by balajeerc · · Score: 2

    The scenario that TBL is talking about here was preempted by Charles Stross in his 2006 novel: Accelerando.

  9. Capitalism isn't fair by rsilvergun · · Score: 2

    news at eleven. If you want fairness guarantee people access to food, shelter, healthcare, education & transportation. Then the rest will sort itself out when folks can no longer leverage their monopoly on those things into the forging of slaves.

    --
    Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/