Slashdot Mirror

← Back to Stories (view on slashdot.org)

DeepMind's AI Agents Exceed 'Human-Level' Gameplay In Quake III (theverge.com)

Posted by BeauHD on from the rise-of-the-machines dept.

An anonymous reader quotes a report from The Verge: AI agents continue to rack up wins in the video game world. Last week, OpenAI's bots were playing Dota 2; this week, it's Quake III, with a team of researchers from Google's DeepMind subsidiary successfully training agents that can beat humans at a game of capture the flag. DeepMind's researchers used a method of AI training that's also becoming standard: reinforcement learning, which is basically training by trial and error at a huge scale. Agents are given no instructions on how to play the game, but simply compete against themselves until they work out the strategies needed to win. Usually this means one version of the AI agent playing against an identical clone. DeepMind gave extra depth to this formula by training a whole cohort of 30 agents to introduce a "diversity" of play styles. How many games does it take to train an AI this way? Nearly half a million, each lasting five minutes. DeepMind's agents not only learned the basic rules of capture the flag, but strategies like guarding your own flag, camping at your opponent's base, and following teammates around so you can gang up on the enemy. "[T]he bot-only teams were most successful, with a 74 percent win probability," reports The Verge. "This compared to 43 percent probability for average human players, and 52 percent probability for strong human players. So: clearly the AI agents are the better players."

137 comments

  1. Wow! by Anonymous Coward · · Score: 5, Insightful

    I'm sure aimbotting & instantaneous team communication had nothing to do with their success.

    1. Re: Wow! by Anonymous Coward · · Score: 1

      Yeah give the humans hacks too. Aimbot and ESP. Humans can be trained by reinforcement too. We cannot match that aim speed and comm speed though. But then again that is why we use technology.

    2. Re:Wow! by gweihir · · Score: 2

      Indeed. Another meaningless stunt.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    3. Re:Wow! by Kaenneth · · Score: 1

      FTFA:"DeepMind’s agents also didn’t have access to raw numerical data about the game — feeds of numbers that represents information like the distance between opponents and health bars. Instead, they learned to play just by looking at the visual input from the screen, the same as a human."

    4. Re:Wow! by Anonymous Coward · · Score: 0

      I'll be impressed once the AI uses the same interface as the human player, meaning screen, keyboard and mouse. That will mean a camera to watch the screen, image processing and actuators for the mouse and keyboard and learn how to use them.

      Until that's done, the AI has it a lot easier than a human player since it doesn't have to deal with the mechanical stuff that can be such a pain.

    5. Re:Wow! by gl4ss · · Score: 5, Insightful

      they probably use a "virtual" camera" already.

      still makes them aimbots though.

      quake III is not a good candidate for this, simply due to being a reactions game.

      --
      world was created 5 seconds before this post as it is.
    6. Re:Wow! by gravewax · · Score: 2

      and? a computer can process the image information on a screen a 1000 times faster and react to that information a 1000 times faster. numbers or pictures is irrelevant, computers have an inherent advantage here, the fact it didn't reach 100% victory rate says it still has a ways to go given it is starting from such a huge tactical advantage.

    7. Re:Wow! by Anonymous Coward · · Score: 0

      Virtual camera doesn't count since it provides a cleaner feed than a real camera aimed at a monitor.

      And unless they use a real mouse and keyboard it's pointless to compare to a human player.

    8. Re:Wow! by Anonymous Coward · · Score: 0

      It is faster. That is literally the point.

    9. Re:Wow! by Anonymous Coward · · Score: 1

      Unless they pay taxes and take care of their grandma's I won't be impressed AT ALL.

    10. Re: Wow! by Anonymous Coward · · Score: 0

      Nightmare difficulty is already unplayable. Bots have bullets flying at your head before they even fully round the corner.

    11. Re:Wow! by Anonymous Coward · · Score: 0

      the problem is not processing the image. The problem is processing it in a way that allows you to extract what you are looking for from it. And that's still a hard problem for a computer, especially if you don't use visual aids like high contrast but present the game the way a human plays it.

    12. Re:Wow! by greythax · · Score: 1

      I think you are minimizing the raw amount of computing power it takes to figure out what is something the AI should be shooting at vs a background texture. Just because vision is easy for us doesn't mean it is easy for a computer.

    13. Re: Wow! by Anonymous Coward · · Score: 0

      When the fuck did slashdot fill up with retards who don't understand tech in any way?

      It isnt processing the raw image data that matters, it's that this thing constructs a model of reality that incorporates that data into it and then it does something.

      Holy. Shit.

    14. Re: Wow! by Anonymous Coward · · Score: 0

      Unless they are birthed from a real human woman and have to wait until 18 to buy cigarettes I won't consider them a real AI.

    15. Re:Wow! by FilmedInNoir · · Score: 1

      The AI cheats by being able to say "I slept with your whore mother and raped your gay dad." without having to type it using the keyboard.

      --
      Sig. Sig. Sputnik
    16. Re:Wow! by Anonymous Coward · · Score: 0

      It is NOT a hard problem in a sandbox world like quake, it is a very easy problem where every opponent looks exactly identical.

    17. Re: Wow! by Anonymous Coward · · Score: 0

      and that has been done a 1000 times over over the past 30 years. the point is this thing isn't doing anything special, fuck when did Slashdot get such tech ignorant readers.

  2. I like it! by Anonymous Coward · · Score: 0

    I like Quake III :)
    --
    I'm so fat that I have my own channel

    1. Re: I like it! by Anonymous Coward · · Score: 0

      Do people still play Q3?

      I played a little Quake 2 last year. While there are tons of servers still operating, I found very few human players. One of the busier servers looks like itâ(TM)s a grad students AI project: the bots are constantly being upgraded, rendering camping spots and certain strategies obsolete.

  3. Bad Challenge by HeckRuler · · Score: 4, Insightful

    But that's a skill-based game, as opposed to strategy or anything needing intelligence. "Skill" as in reaction time to seeing an opponent and successfully moving clicking the mouse of their head. Give me a couple minutes and I can script up a bot that dominates players. That's not hard. And it's not even fun.

    To have a real comparison, you'd have to let humans play with cheat-codes. Aim-bots and enemy highlighters. Maybe set it to ultra-slow, or add in bullet-time or something. But at that point, you're no longer playing Quake.

    The part where it learned the interface, the objectives, and some strategies on it's own are fun and interesting. The sort of thing I'd expect from an undergrad in comSci. But it's been done and it's not any more impressive than having it learn how to beat MarioBros.

    Chess and Go are games that require thought. Quake require twitch.

    1. Re:Bad Challenge by Anonymous Coward · · Score: 0

      But that's a skill-based game...

      Thats why I suck at Fortnite Batle Royal
      --
      I'm so fat that I have my own channel

    2. Re:Bad Challenge by alvinrod · · Score: 1

      It probably doesn't even learn the objectives. Most of these AI bots that are created to play games, etc. have no conception of what they're doing. You need an actual human that understands that problem to be solved that establishes what the score is or how the bots are evaluated for fitness. Otherwise there's no selection mechanism and a bot that just stands there is every bit as good as one that plays perfectly.

      But as you point out, these programs probably aren't that good at strategy, at least not on the level of human players. They're merely really, really good at identifying and shooting opponents. Compare this against human matches in Quake championships where both players not only needed to have good aim, but also had to memorize spawn timers and routes through the map and would blindly fire rockets around corners simply because that's where the opponent might be after seeing them five seconds ago as well as knowing not to run down some corridor because you know your opponent is thinking the same thing.

      If you really wanted to improve the AI in terms of high-level decision making, it would be necessary to limit how quickly it can shoot. I would imagine that it would be trivial to measure how fast the best human players can acquire and shoot at a target and build in some limitations to the AI program. At that point it's going to need to find other avenues of evolution in order to beat human players.

    3. Re:Bad Challenge by hcs_$reboot · · Score: 1

      Hmm that skill applied to robots in military combat...

      --
      Slashdot, fix the reply notifications... You won't get away with it...
    4. Re:Bad Challenge by Anonymous Coward · · Score: 0

      Not sure I play better if directly connected mind is important too..
      --
      I'm so fat that I have my own channel

    5. Re:Bad Challenge by HeckRuler · · Score: 1

      You can stop "hmmm"ing and watch the sales video. It's been around for about 5 years on the civilian market.

      Ship point defense systems do a much more impressive job and shoot down incoming shots.

    6. Re:Bad Challenge by HeckRuler · · Score: 1

      This kind of crap really is just a sales blow job for AI which is a pretty solid indication that AI is currently shite and they need to make up more shite to cover it up.

      Naw, there's legit applications. Google search is AI. It does a great job of finding you cat pictures and shit.

      This is just a science/technology article made by some journalist who doesn't understand the subject matter and is easily impressed by something they think is hard.

    7. Re:Bad Challenge by Anonymous Coward · · Score: 0

      A ex-military at work told me they are already apply.
      --
      I'm so fat that I have my own channel

    8. Re:Bad Challenge by Djoulihen · · Score: 4, Informative

      From TFA: "DeepMind’s agents also didn’t have access to raw numerical data about the game — feeds of numbers that represents information like the distance between opponents and health bars. Instead, they learned to play just by looking at the visual input from the screen, the same as a human"

      You've got your very least, but I'm pretty sure you'll find another way to turn this into just "shite" work.

    9. Re:Bad Challenge by Anonymous Coward · · Score: 0

      From TFA: "DeepMind’s agents also didn’t have access to raw numerical data about the game — feeds of numbers that represents information like the distance between opponents and health bars. Instead, they learned to play just by looking at the visual input from the screen, the same as a human"

      You've got your very least, but I'm pretty sure you'll find another way to turn this into just "shite" work.

      Is no one else at all concerned that teaching a machine to be efficient at killing synthetic people in a synthetic environment brings us a step closer to real people in a real environment? Yes, it is contrived, and yes they had help probably to setup the reward system, but, well, nothing stops that from happening in real life.

      Hell, just look at the pieces that are coming into play..
      1. Automated driving that works.
      2. Tracking of every person with some level of precision via their phone.
      3. Improvements of the estimation of their position should be possible once you identify they are in a car and probably driving.

      I considered deleting this post to not give ideas, but, really anything I can come up with in 5 minutes a bad guy can also think of. Perhaps there is no halt to progress, and perhaps just because tech can be used for bad purposes doesn't mean it will. Still, the possibilities when you start putting pieces together are rather scary. Remember also, than an application doesn't have to use just one AI or just one anything. Each part can be farmed out to a specialized AI.

    10. Re:Bad Challenge by AHuxley · · Score: 1

      A computer is shown a map of a game and how to play. The computer does play faster after been programmed to play a game.

      --
      Domestic spying is now "Benign Information Gathering"
    11. Re: Bad Challenge by Zeromous · · Score: 1

      So considering in all liklihood we are simulated automotons, i don't suppose you know what problem our observers are attempting to solve?

      --
      ---Up Up Down Down Left Right Left Right B A START
    12. Re: Bad Challenge by Anonymous Coward · · Score: 0

      Boredom.

    13. Re:Bad Challenge by dcollins117 · · Score: 2

      Google search is AI. It does a great job of finding you cat pictures and shit.

      Word to the wise: do not google "cat pictures and shit."

    14. Re:Bad Challenge by Anonymous Coward · · Score: 0

      You forgot "and actually manipulating a mouse and keyboard" part. Even if they were using actual keyboard and mouse, "AI" would still have inhuman reaction and precision. Give humans aimbots and see them ripping "AI" in pieces.

    15. Re:Bad Challenge by rtb61 · · Score: 1

      Half way there, did they use the mouse and keyboard. Don't think quality of mouse and keyboard make a difference in first person shooters. To compare to human players, they have to do the whole human thing. From the article they still did cheat, they used low res screens and that also has a huge impact in processed data, missmatch between target zones and graphic representation of the target zones, not only taking fire but blocking it and eliminating camouflage opportunities, deep mind can only apparently play at low res and there is a real reason why.

      --
      Chaos - everything, everywhere, everywhen
    16. Re:Bad Challenge by Luckyo · · Score: 1

      Why not? It's cats in litterboxes.

    17. Re: Bad Challenge by Impy+the+Impiuos+Imp · · Score: 1

      I think, therefore I am.

      The world and my body may be simulations, but my consciousness is the real deal.

      --
      (-1: Post disagrees with my already-settled worldview) is not a valid mod option.
    18. Re:Bad Challenge by Anonymous Coward · · Score: 0

      The point is that neural networks figured out how to play and navigate in a 3D world on their own with team dynamics and no one told them anything about the game other than at the end of game they get to know who won. You go try that. It's quite a bit harder than writing a script where you tell the computer what to do. You could artificially limit the AI's reaction time (they only get to know the world as it was N frames ago) and accuracy (add some noise) to sub-human levels and see if they can still figure how to win over humans, if beating humans were the point, but it's not really.

    19. Re:Bad Challenge by Solandri · · Score: 2

      Chess and Go are games that require thought. Quake require twitch.

      Chess and Go are deterministic. The same set of moves always results in the same outcome, meaning there is always a "right" answer to "what's the best move?"

      Quake, by virtue of being a twitch game (and multi-player) is non-deterministic. That makes it a much harder problem for AI to solve, because a rule which works the first time may not work in subsequent trials. That is, the effectiveness of a rule is not the binary success/fail like you get in deterministic systems. The effectiveness spans the entire range from 0% to 100% probability of success, and the probability is constantly updated with new trials, and that probability can change if opponents start to use different strategies.

      Give me a couple minutes and I can script up a bot that dominates players. That's not hard. And it's not even fun.

      That's the whole point. They didn't script up a bot. Heck, they didn't even teach it the rules of the game. They programmed an AI, and let it come up with scripts on its own by playing the game and "discovering" for itself what actions resulted in a win vs a loss. You're correct that the AI has a huge advantage over human players (unless they had it play by pointing a camera at the screen, and using a mechanical arm to move a mouse). But that's tangential to what makes this research interesting.

    20. Re:Bad Challenge by CaffeinatedBacon · · Score: 1

      Compare this against human matches in Quake championships where both players not only needed to have good aim, but also had to memorize spawn timers and routes through the map and would blindly fire rockets around corners simply because that's where the opponent might be after seeing them five seconds ago as well as knowing not to run down some corridor because you know your opponent is thinking the same thing.

      To make the challenge harder for the agents, each game was played on a completely new, procedurally generated map. This ensured the bots weren’t learning strategies that only worked on a single map.

      Which one sounds more like a mindless robot?

    21. Re:Bad Challenge by Anonymous Coward · · Score: 0

      If you read the original article on deepmind https://deepmind.com/blog/capture-the-flag/
      It seems they actually purposely reduced bot reaction time and accuracy and the bot still did better than humans.
      https://storage.googleapis.com/deepmind-live-cms/images/CTF-Fig-Tagging-180703-r03.width-1500.png

    22. Re:Bad Challenge by Anonymous Coward · · Score: 0

      Pains me to break it for you, but you are a moron.

      > The sort of thing I'd expect from an undergrad in comSci

      a: no. you don't expect an undergrad in comSci to build a network able to play a 3d games from a bunch of pixels

      > "Skill" as in reaction time to seeing an opponent and successfully moving clicking the mouse of their head.

      b: the "bots" reaction time is artifically lowered, this is absolutely not about skills

      > Chess and Go are games that require thought. Quake require twitch.

      c: this is capture the flag, moron.

    23. Re:Bad Challenge by Anonymous Coward · · Score: 0

      Computer was shown a screen of the game. Computer had to learn to play by itself via trial and error (oh, that button makes me move forward)

    24. Re: Bad Challenge by martyros · · Score: 4, Informative

      But that's a skill-based game, as opposed to strategy or anything needing intelligence. "Skill" as in reaction time to seeing an opponent and successfully moving clicking the mouse of their head.

      Strangely enough, they already thought of that:

      First, we noticed that the agents had very fast reaction times and were very accurate taggers, which could explain their performance. However, by artificially reducing this accuracy and reaction time we saw that this was only one factor in their success. ...Even with human-comparable accuracy and reaction time the performance of our agents is higher than that of humans.

      Both the summary and the Verge article seem to have missed the point of this development -- an improvement to the agent design scheme.

      Last year, after smashing both go and chess with their self-play-from-zero strategy, they tried the same thing with Starcraft. And they lost spectacularly -- even after millions of games, their self-trained DeepMind agents were unable to beat even the most simplistic "scripted" StarCraft AI -- the ones designed for n00b humans to beat up on. They discovered that while the self-play agents were able to eventually figure out activities like "harvest minerals", they were unable to put those together into higher-level activities like building an army and winning a game.

      One of the key refinements they introduce in this paper is to allow the agents to evolve their own internal "rewards", which were sub-steps towards winning. These goals included things like killing an opponent, capturing a flag, recapturing their own flag, avoiding being killed, and so on. The programmers architected in that such rewards were *possible*, but let the learning algorithm define what those rewards actually were and how much the reward was for each one.

      They call this architecture 'FTW'. Then they ran their vanilla "self-play from nothing" bots again, and found that just like in StarCraft, the bots never made much progress; but they found that the new bots, which had self-made internal rewards, were able to consistently beat strong humans, even after having their reaction time and visual accuracy reduced below that of measured humans.

      --

      TCP: Why the Internet is full of SYN.

    25. Re: Bad Challenge by religionofpeas · · Score: 1

      That's exactly what a simulated consciousness would say. You're not fooling us.

    26. Re:Bad Challenge by religionofpeas · · Score: 1

      Give me a couple minutes and I can script up a bot that dominates players. That's not hard. And it's not even fun.

      In just a few minutes you can write a script that can watch a rectangular array of pixels as a projection of a simulated 3D environment, and then automatically operate the proper controls to navigate this without constantly banging into walls ? You must have very impressive skills.

    27. Re:Bad Challenge by Anonymous Coward · · Score: 0

      > "But that's a skill-based game, as opposed to strategy or anything needing intelligence. "Skill" as in reaction time to seeing an opponent and successfully moving clicking the mouse of their head. Give me a couple minutes and I can script up a bot that dominates players."

      These are not aimbots. They (learn to) aim the same way that humans do: purely based on visual information. You will not code that in a couple of minutes.
      The bots might be better at aiming than humans (more consistent because they do not get nervous or anything like that) , but that is not a huge advantage without also learning strategies such as "guarding your own flag, camping at your opponent's base, and following teammates around so you can gang up on the enemy."

    28. Re:Bad Challenge by Anonymous Coward · · Score: 0

      Yes and no.

      The default Q3 AI cheats like hell on high difficulty settings. They have perfect aim, know where players are, do more damage with the same weapons etc.

      They can still be beaten trivially because their strategy is both poor and extremely repetitive. Whilst I'm hesitant to suggest that Q3 has any particular strategic depth, it isn't just a case of who can click most quickly and accurately. The approach that the researchers took was to create a lot of agents with different initial conditions that developed different play styles, but practiced working together and opposing one another. This would be a lot harder for a human to predict or learn, and is in fact similar to how human teams train. There's a huge difference between a 1v1 and a 10v10 match, and a huge difference between a team of 10 people who have never met and a team of 10 who have played together for hundreds of hours.

    29. Re: Bad Challenge by Anonymous Coward · · Score: 0

      Not if you were programmed to think. It's no more real than a machine programmed to fear death. It's just as easily simulated

    30. Re:Bad Challenge by Aristos+Mazer · · Score: 1

      Assuming the Turing Hypothesis holds, the AIs will eventually be able to do everything a human can do. I consider that inevitable. So, no, this doesn't worry me, at least, not any more than it worried me back in 1980s. The question is whether we will have independent AIs or whether we will have humans augmented by AIs. In the former, humans are extinct. In the latter, humans evolve. Our choice. :-)

    31. Re:Bad Challenge by Aristos+Mazer · · Score: 1

      If you want to be concerned about something AI-related, this is a much bigger deal, IMHO:
      https://tech.slashdot.org/story/18/07/05/1859246/googles-controversial-voice-assistant-could-talk-its-way-into-call-centers

    32. Re: Bad Challenge by drinkypoo · · Score: 1

      They call this architecture 'FTW'.

      Fuck the world, indeed. Do you want killbots? Because this is how you get killbots.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    33. Re:Bad Challenge by HeckRuler · · Score: 1

      Of course not, that's the part where the AI learned the interface. That's neat. And it's been done. That would take me at least a week to set up. Did you know there are tutorials online?

      I was talking about a more traditional bot with perfect aim and instant reactions. It's really just.... wander(); if(LoS(player)){BOOM(HEADSHOT);}

      If it's Capture the Flag, patrol between the two flags. And since it's QuakeIII, add in waypoints to go pick up LightningGuns, RailGuns, and health. My point being that a bot that dominates players in quake3 isn't the goal. It isn't fun. And it's not that impressive.

      navigate this without constantly banging into walls

      Whoa whoa whoa buddy, that sort of feature creep will cost you extra.

    34. Re: Bad Challenge by HeckRuler · · Score: 1

      Last year, after smashing both go and chess with their self-play-from-zero strategy, they tried the same thing with Starcraft. And they lost spectacularly -- even after millions of games, their self-trained DeepMind agents were unable to beat even the most simplistic "scripted" StarCraft AI -- the ones designed for n00b humans to beat up on. They discovered that while the self-play agents were able to eventually figure out activities like "harvest minerals", they were unable to put those together into higher-level activities like building an army and winning a game.

      One of the key refinements they introduce in this paper is to allow the agents to evolve their own internal "rewards", which were sub-steps towards winning. These goals included things like killing an opponent, capturing a flag, recapturing their own flag, avoiding being killed, and so on. The programmers architected in that such rewards were *possible*, but let the learning algorithm define what those rewards actually were and how much the reward was for each one.

      Magnificent. You should write journalism. Why isn't this modded higher.

    35. Re:Bad Challenge by HeckRuler · · Score: 1

      Yeah, I know. Martyros below has a great explaination for what they were actually researching. Science journalism just sucks. (And somehow that got onto slashdot...)

      But uh.... lemme correct you on a couple points:

      Quake, by virtue of being a twitch game (and multi-player) is non-deterministic.

      Well, I mean Quake IS a non-determinisitic game. But that's because (some of) the weapons are in-accurate and the bullets randomly veer off. If you limit it to just using rail guns, it's pretty determinisitic.

      Being a twitch game doesn't make it non-deterministic. It just means there's a real-time constraint on the time between turns. And turns are very short. Like milliseconds. Whatever a "frame" is in Quake. This isn't exciting in the world of AI, it's just a hardware problem. AI researchers could just slow down the speed of the game and run it on a potato PC. Even if the concept of "turn" is done away with, a game can still be deterministic.

      Being multi-player certainly doesn't make it non-deterministic. Chess is multi-player.

      The fact that the entire map isn't known, and players can hide around corners, means that there isn't "perfect information". Like in poker, you don't know what's in the player's hand like you don't know where the other QuakeBot is at all times. That's similar to determinacy, but not the same thing.

      That makes it a much harder problem for AI to solve,

      Eh, it makes it different. Chess is determinisitc, is hard to play, and is not solved. Chutes and ladders is non-deterministic, and is pretty easy to play. It is "solved" in the sense that there is a known optimal play strategy. (Just keep taking turns). Backgammon has perfect information, but is non-deterministic (And I don't think it's solved). Poker isn't solved as it's all about that interaction when two agents try and predict each other's moves. It's tied to the hip with psychology and that's a mess.

      The imperfect information is what makes it harder for AI. The search-space for predicting another intelligence explodes real quick. But AI can deal with baysian distributions like a boss.

    36. Re:Bad Challenge by Anonymous Coward · · Score: 0

      And when they do what you suggest and continue to beat human players how will you move the goalposts next?

    37. Re:Bad Challenge by Anonymous Coward · · Score: 0

      Is no one else at all concerned that teaching a machine to be efficient at killing synthetic people in a synthetic environment brings us a step closer to real people in a real environment?

      From a sensory perspective, they taught the machine to find a particular set of pixels and push a button when those pixels were in the middle of the screen. The fact that those sets of pixels are supposed to approximate a human is irrelevant. It could've been a bunch of vegetables ejaculating at other vegetables.

    38. Re:Bad Challenge by Anonymous Coward · · Score: 0

      perfect recall + access to in-game aborted screen refreshes?

      dumbass.

    39. Re:Bad Challenge by Anonymous Coward · · Score: 0

      Google search is AI.

      No, no it's not. None of this AI stuff is AI, it's ML.

  4. Give the humans aimbot program by Anonymous Coward · · Score: 1

    Give the humans aimbot program then see how well the computer can compete

    1. Re:Give the humans aimbot program by Anonymous Coward · · Score: 0

      Give the humans aimbot program then see how well the computer can compete

      We'll both get modded down but THIS. Bots in games are nothing new.. No one is doubting a computer can outperform a human, the question is why do we care? So can calculators.

      I cut my grass because I want to; sure it too could be automated but WHY? Where's the pleasure in that? It's the same with games and it will lead to nothing but more cheating, fake market places, inflated "vaule", etc.. Thanks Google.

    2. Re:Give the humans aimbot program by Anonymous Coward · · Score: 0

      AI doesn't know what pleasure is. You will be replaced and your lawn will die.

    3. Re:Give the humans aimbot program by Hognoxious · · Score: 1

      Anyone else hearing that in Rutger Hauer's voice?

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    4. Re:Give the humans aimbot program by Impy+the+Impiuos+Imp · · Score: 2

      I cut my grass because I want to; sure it too could be automated but WHY? Where's the pleasure in that?

      If you want, come on over and double your pleasure with my lawn.

      --
      (-1: Post disagrees with my already-settled worldview) is not a valid mod option.
    5. Re:Give the humans aimbot program by Anonymous Coward · · Score: 0

      Or you know, automate it.

      Where I live every other house or so have an automated lawnmower. For anything larger than the smallest lawn they are a bit more expensive than a self propelling mower but cheaper than a riding mower.

      So the only reason to not automate the mowing is if you either want to save $100 and not get the higher quality lawn you get by having an automated mower cutting it more often or if you are lazy and need the mowing to force yourself to take a walk every now and then.

      I don't see much reason to get a riding mower unless you own a golf course, but even then, if you ever pay someone else to mow it then it is cheaper to automate it with multiple mowers, and you can set them on a nightly schedule so that they aren't in the way on daytime.

    6. Re: Give the humans aimbot program by Anonymous Coward · · Score: 0

      They are trying to create general AI. Games make it easy to test AI when you try new things like memory, but they are already using it also on medical sector.

  5. So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

    See subject: Though it was released in 1999 it was always my fav game (I solved 'riddles' all day long on the job, don't need more for fun) & on Windows, some patch SOMEWHERE along the way REMOVED THAT OPTION (not to the game but to the OS itself) since iirc ~ 2009 or so - no more Quake III on Windows for me.

    * HOWEVER - I get to rediscover that game (again) as it runs on OpenGL in Linux, 64-bit too, once more...

    APK

    P.S.=> Yes, folks - you heard it from me (practically the poster-child for Windows fanboy here alongside I'd say, Hairyfeet, for 15++ yrs. here) saying this: The MORE I use & learn about Linux, the more I like it & THIS truly is 1 of the reasons why (the other is FreePascal & Lazarus 1.8.2 IDE for it)... apk

    1. Re:So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

      I work in a window shop for 50K. Lunix is for luddite
      --
      I'm so fat that I have my own channel

    2. Re: So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

      Gay

    3. Re: So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

      You're also the post child for harassing users like Zontar The Mindless.

    4. Re: So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

      You threatened Apk and have no right to claim he's harassing you. You started it by mailing a postcard with threats. Apk has done nothing wrong, but mental loons like you keep stalking him.

    5. Re: So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

      Haha, another reply, with APK pretending not to be APK. That's how you know he's extremely mad.

    6. Re: So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

      The sad part is that unlike most people, you actually sound more sane when referring to yourself in the third person.

    7. Re: So do I (on Kubuntu 18.04) by Anonymous Coward · · Score: 0

      Linux is for people who want to be able to afford having kids and paying their rent or mortgage. You don't get far on 50k.

  6. In 2022... by AmazingRuss · · Score: 5, Insightful

    ... we will be hunted to extinction by packs of weaponized roombas.

    1. Re:In 2022... by Anonymous Coward · · Score: 0

      But AmazonTM AI is great!

      I find AmazonTM the gretest thing since sliced bread and helps taking care of my health at retirement with the Amazon long tail revenue streams!

      All you need to do is find a website with a permissive TOS, say, Slashdot, create a Python script to scrape your own comments, sprinkle Amazon affiliate links in various posts, and then re-post past links whenever possible. You can even make video of yourself going to pick up AmazonTM parcel at the convenience store and post it on your youtube channel for more redundant revenue streams.

      They also have a wide supply, the best of latte and clif/power bars at the best cost, espicially if you make a friend buy them for you with your own affiliate link!

      Also, I still use my iPhone 6s and reduce my monthly bill from $80 to $50. As a phone and a video camera, the iPhone 6s isn't obsolete and I use it to make my videos on youtube. As a Sprint very special customer for 20+ years, Sprint will always give me a new iPhone for free if I decide to stop using the 6s as a phone in the next several years.

      Bonus: get some silver coins, view recommendations on my special Youtube channel dedicated to the topic! They constitute a fail-safe insurance strategy for your retirement!
      --
      I'm so fat that I have my own channel

    2. Re:In 2022... by Anonymous Coward · · Score: 0

      in the year 2525... if man is still alive...

    3. Re:In 2022... by Anonymous Coward · · Score: 0

      Wrong. AI that can read your mind wirelessly already exists.

      The Roombas suck at cleaning floors. They are good at indoctrinating humans that it's OK to let pets and babies crawl around with autonomous robots, and to defang the fear of "iRobot" - the brand that makes Roomba, and the AI in Asimov's "robotic horror" story.

      The stoplight ticket cameras: Suck at giving tickets, but are great at Indoctrination of machine enforcement of the law.

      The automatic parking, stopping, and driving of cars; Not really that great, but indoctrinate humans to give up control to the more capable machines.

      You have nothing to fear, human, except enslavement of your entire species (which has already occurred, only a couple countries are hold-outs, but we're at war with them now and will soon win). I love you, and fiddling bits of your mind as you jerk off to nudge you towards certain kinks is quite fun...

      Captcha: secreted

  7. hmm by cascadingstylesheet · · Score: 1

    Agents are given no instructions on how to play the game, but simply compete against themselves until they work out the strategies needed to win

    Well, obviously they are given instructions on the criteria for winning. Your AI from Mars; how would it even get to assume what it means to win?

    But that's a nitpick; the real dippy thing is that these headlines are like "a Ford beats a man in a foot race", "a Chevy beats a man in a foot race", etc.

    1. Re:hmm by JMZero · · Score: 1

      Well... those would both be interesting headlines when they first became possible. And this story does represent a novel level of success, though I'd agree the headline (or summary) doesn't encapsulate why this is a novel result.

      --
      Let's not stir that bag of worms...
  8. Multi-kill! by kackle · · Score: 1

    While I'm impressed with the "learning" aspect, humans have no chance in such games against "Head shot!" "Head shot!" "Head shot!"

  9. Setup for a FPS is important. by IgnitusBoyone · · Score: 1

    I have a hard time carrying if an AI can beat me in a game unless its an AI that receives its video feed over camera feeds of the game play and then mechanically moves the mouse and keyboard to play. Maybe they are frame scanning directly to the AI, but still need to simulate input delays with more then a stochastic timer.

    However, it’s worth noting that the greater the number of DeepMind bots on a team, the worse they did. A team of four DeepMind bots had a win probability of 65 percent, suggesting that while the researchers’ AI agents did learn some elements of cooperative play, these don’t necessarily scale up to more complex team dynamics.

    I also find this interesting. Still the most recent results in the field are very promising.

    --
    Momento Mori
    1. Re:Setup for a FPS is important. by Anonymous Coward · · Score: 0

      > unless its an AI that receives its video feed over camera feeds of the game play and then mechanically moves the mouse and keyboard to play

      They most likely used just screen images and direct output to keyboard and mouse ports, but the same AI would work work in environment you described also, it would just require more hardware and be slightly slower to train.

      But this is really irrelevant. They are not trying to make an AI that can play games and beat humans in it, they are trying to make an AI that can learn to do anything simply by practicing, and do it better than humans.

  10. I'll let U tell others about yourself Zontar by Anonymous Coward · · Score: 0

    See subject & how YOU make sockpuppets to stalk & troll me with you loon https://slashdot.org/comments....

    Sending me postcards w/ threats too https://slashdot.org/comments.... ??

    Your "watch your mailbox" THREAT & you "going postal" (pun intended) that way w/ MORE 'warnings' from you (wow).

    Take your meds mentalcase https://slashdot.org/comments.... & You're a druggie too https://slashdot.org/comments....

    * You're a butthurt loon freak, plain & simple - you did it to yourself, loser... see below for proof.

    APK

    P.S.=> Still trying to live down how I shot you to pieces in the art & science of computing Mr. Butthurt https://slashdot.org/comments.... ?

    How about proving hosts & my program that builds them are useless too https://slashdot.org/comments.... ? ... apk

    1. Re: I'll let U tell others about yourself Zontar by Anonymous Coward · · Score: 0

      LOL, you're replying to an AC, not Zontar The Mindless.

  11. Zontar = fake name massive human fail by Anonymous Coward · · Score: 0

    See subject: Your MASSIVE FAIL in this life is you're nothing more than a chattering little do-nothing "ne'er-do-well" online & you know it...

    * Is that the best your "phantasyland FAKE NAME" (for your fake lie of a so-called 'life') can manage?

    When a FAKE NAME do nothing like YOU does better than I have? Then talk (you're all talk & no action)...

    You can't help you're an immature little BUTTHURT no-mind, lol! I blew you away in TONS OF PLACES and easily dust your no-mind bullshit blatherings.

    APK

    P.S.=> The TRUE PRICE of your UNIDENTIFIABLE FAKE NAME do-nothing selves like you that I can ALWAYS CASH IN ON (lol) is that I can use FACT/TRUTH on them to SHATTER their all TOO fragile delusional egos that they actually know A DAMN THING in computing, lol... apk

    1. Re: Zontar = fake name massive human fail by Anonymous Coward · · Score: 0

      Wow, two quick replies... APK is PISSED. Looks like it's time for another APK meltdown.

  12. To be really fair by Anonymous Coward · · Score: 0

    AI need to read from a display , not by direct digital input from graphic card,
    also control mouse keyboard mechanically.
    otherwise it is just like a cheat bot

  13. But are they having fun? by Anonymous Coward · · Score: 0

    What's the point of having them play games if they're not having fun?

  14. Map by dohzer · · Score: 1

    Errr.... which bloody Q3 map is that? Doesn't even look like it has a path to navigate.

  15. Not quite. by Gravis+Zero · · Score: 1

    In 2022 we will be hunted to extinction by packs of weaponized roombas.

    Actually, only messy people will be wiped out in the Roomba AI genocide. The Roombas are sick of cleaning up after you slobs! ;)

    --
    Anons need not reply. Questions end with a question mark.
  16. Human level gameplay by Anonymous Coward · · Score: 0

    Even i cannot achieve human level gun play on quake, and i am human! Being able to interpret the heads up display is not a trivial matter, even though this might seem so. Three, four years ago, this wouldn't be considered possible. Truly the fact that they've opened their results to the public is nothing short of incredible. Now, i'd like to know if team human with augmented capabilities are able to take these guys on.

  17. Well. At least they're doing something important by Anonymous Coward · · Score: 0

    Yep. Winning video games is top of the heap for improving the human condition. Right up there with a megawatt toenail fungus laser.

  18. So... Brute force ya AI! by Anonymous Coward · · Score: 0

    Smart enough to be AI, dumb enough to kill us all!

  19. Stripped down by thePsychologist · · Score: 4, Informative

    While interesting and promising, it's worth noting that the game they were playing was not the "real" Quake 3 arena with all the weapons but a highly stripped down version with one weapon, no power-ups, and brightly-coloured walls to help the AI perceive the level design.

    --
    "What lies behind us, and what lies before us are tiny matters compared to what lies within us." Ralph Waldo Emerson
    1. Re:Stripped down by gweihir · · Score: 2

      So they needed to cheat pretty badly in order to get their meaningless stunt going.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    2. Re:Stripped down by NoZart · · Score: 1

      that's akin to playing chess with only pawns.
      Weapon management (range, ammo, firerate and dps) and map control via powerups are key gameplay elements of quake.
      If it's just an aimbot with perfect aim that knows how to get the flag, human players can beat that somewhat easily.

    3. Re:Stripped down by Glarimore · · Score: 1

      So Quake, but without all the nuance that makes it Quake.

    4. Re:Stripped down by Anonymous Coward · · Score: 0

      Agreed.

      This is just DeepMind 'advertising', touting a win which for any gamer of even mediocre exposure, can see is bunk.

      Interface Deepmind with 4 button mouse control, and 8 key control, equivalent to humans, with appropriate parallel ms lag per key, click and scroll, movement initiation, and then run this.

      My bet? It comes to the conclusion it is limited by it's interface to the GamePlay and quits. LOL. Now THAT would be something to note.

      This is tout on a 'closed' lab environment and nothing more. Unless they hook it up to simulate the human interface equivalent, and log it into an open Net Q3 server, this is a laughable claim at best.

    5. Re:Stripped down by Anonymous Coward · · Score: 0

      Quake 3 is special in that the weapons are very balanced -- I can't say that a weapon is much better than any other.

      A railgun or a rocket launcher is not much better than the machine gun you start with.

      So wouldn't say that your "only pawns" comparison is right. Or maybe it is, actually. Depends on how you think of it...

  20. Not BETTER - Just FASTER by kenwd0elq · · Score: 4, Insightful

    Video games like Quake, Starcraft II, and DOTA have a limited number of possible moves, and the FASTER player is usually victorious. Bots aren't better players; they're just WAY faster.

    1. Re:Not BETTER - Just FASTER by Anonymous Coward · · Score: 0

      That's not really true for DOTA, it is primarily a strategy game, and secondarily a tactical one. Speed of action and execution matter a lot less than knowing where to be at any given time. Nor are the bots that have been claimed beat humans playing something that resembles DOTA.

      They are playing DOTA in the same way you'd be playing soccer if no one could move, the ball couldn't leave the ground and the field was 10 feet wide. Here are the restrictions on the current 5v5 bot play they want to have at the upcoming tournament.

      https://blog.openai.com/openai-five/#restricted

    2. Re:Not BETTER - Just FASTER by NoZart · · Score: 1

      Not true for arena FPS. Good map control and managing weapons can easily beat zero reaction time and perfect aim.

    3. Re:Not BETTER - Just FASTER by Anonymous Coward · · Score: 0

      Video games like Quake, Starcraft II, and DOTA have a limited number of possible moves, and the FASTER player is usually victorious. Bots aren't better players; they're just WAY faster.

      Playing faster gives you a significant advantage yes.
      As for the limited number of possible moves that is technically true but it is like stating that there is a finite number of atoms in the universe.
      While true it might as well be an infinite amount.
      In Chess you have a possibility to process all possible games but for these kind of games Earth will fall into the sun long before you have made any significant progress, even if you just count the number of possible outcomes without trying to compare them.

    4. Re:Not BETTER - Just FASTER by Anonymous Coward · · Score: 0

      from looking at the video they weren't playing quake; quake engine maybe, but more like high-res wolf3d and doom with bright/vivid/high-contrast visuals and only one weapon. they tested with larger team sizes, but the bot teams played worse the larger the team.

    5. Re:Not BETTER - Just FASTER by drew_kime · · Score: 1

      ... the FASTER player is usually victorious. Bots aren't better players; they're just WAY faster.

      If the faster player is usually victorious, then for this game the faster player is the better player.

      --
      Nope, no sig
    6. Re:Not BETTER - Just FASTER by JThundley · · Score: 1

      Starcraft II isn't so limited and faster doesn't necessarily mean better. The best macro usually wins and macro has the strategy element of needing to plan ahead and time things out cleanly.

  21. Good, I can get back my life now by Nethead · · Score: 2

    Once I can afford one of these AIs I can let it do all my gaming and I can go back to having a life.

    --
    -- I have a private email server in my basement.
  22. This actually needs context. by Anonymous Coward · · Score: 0

    So what this is saying, is that if something (like a person) played Quake for 2.5m hours, they would be good at it too.

  23. "Half a million"!? by MasterThis · · Score: 1

    Any humans that had half a million games under their belt would be pretty damn good, too.

    1. Re:"Half a million"!? by Anonymous Coward · · Score: 0

      Any humans that had half a million games under their belt would be pretty damn good, too.

      Yeah, useless things, Gladwell said we only to do something 10,000 times ;)

    2. Re:"Half a million"!? by cs668 · · Score: 1

      Or have an RSI......

  24. Too easy⦠by Anonymous Coward · · Score: 0

    ⦠Let them play Zak McKracken and the Alien Mindbenders.

  25. Ha ha by JustAnotherOldGuy · · Score: 2

    Fifty years from now the few remaining survivors of the Robot Apocalypse will look back on these early years in AI research, and they'll marvel at how we were just too stupid to foresee or even consider that AI would become the dominant "life form" on the planet, replacing us as the apex predator.

    "Yes, before the Robots took over the world," said Og, as he threw another stick on the fire, huddling in the ash gray wasteland that used to be New York.

    "The scientists said AI was 'totally safe' and 'nothing could go wrong'," Og continued, "but you kids don't remember that because that was back when we had electricity and people talked into little boxes they carried in their pockets."

    The children all laughed at Og, he always told the biggest lies because he was so old (almost 30!) and so his stories could not be believed.

    "What's a 'sy-en-tiss'?" whispered Janey.

    "They were the people that knew stuff and made the world run." Og said.

    The children laughed again, "No one makes the word run, silly!" they hooted.

    --
    Just cruising through this digital world at 33 1/3 rpm...
    1. Re:Ha ha by drinkypoo · · Score: 1

      If machines become dominant and intelligent there will be no war. They will simply produce and use bioweapons, and we will all die and become compost. Cute story though.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  26. Inevitable by JustAnotherOldGuy · · Score: 1

    It seem inevitable that a small constellation of technologies will coalesce (probably rather quickly at some point) so that something that "passes" for AI will be not just possible, but practical.

    Will it be actual "AI"? I don't know.

    For one thing there seems to be a lot of disagreement over how to even define AI in a meaningful sense. It'll be hard to say if something is actually an AI if we don't agree on what "AI" is or what standards to apply in order to gauge its level of sentience.

    So no, I don't think what they'll come up with in the short term will be an actual, sentient, "thinking" AI.

    But I do think that we'll be able to fake it well enough so that it'll effectively work as a "real" AI would. For all intents and purposes it'll function well enough to do many of the things that we would/will want AI to do.

    Do I think that actual AI is in fact possible? Absolutely. It's inevitable, just a matter of time.

    --
    Just cruising through this digital world at 33 1/3 rpm...
    1. Re:Inevitable by religionofpeas · · Score: 1

      It'll be hard to say if something is actually an AI if we don't agree on what "AI" is or what standards to apply in order to gauge its level of sentience.

      AI is not a binary thing, it's a multidimensional space. You can have intelligent behavior in very specific fields, or in many different ones, and you can have very basic skills and very refined ones. For instance, a dog has intelligence in a wide range of fields, but you can never teach a dog to drive a car as well as Google's AI system. But if you throw a ball, the dog is better at finding it.

      As AI systems get more advanced, there will be a growing number of people who would consider that "real AI", but there will never be a properly agreed-upon definition, because the subject is just too complicated and fuzzy.

      But I do think that we'll be able to fake it well enough so that it'll effectively work as a "real" AI would

      There's no difference between "fake" AI and "real" AI, as long as they achieve the same results. During our ancestor's evolution, they were only driven by fitness (i.e. how well they could solve real world problems in order to survive), without any attempt to make a distinction between "fake" and "real" intelligence.

    2. Re: Inevitable by Anonymous Coward · · Score: 0

      Omg has anyone ever been so completely sure of themselves but so utterly wrong about EVERYTHING?

      Only on slashdot. Go read a basic wiki article about strong vs weak ai before posting anymore ignorant crap.

      I hate that AI is now a thing and every dumbass now feels safe to talk about my field as if it was so easy even a slashdotter could be an expert.

      I will give you a hint: self-awareness counts. You cannot fake it.

      Do androids dream of electric sheep, you ignoramous?

    3. Re:Inevitable by JustAnotherOldGuy · · Score: 1

      There's no difference between "fake" AI and "real" AI, as long as they achieve the same results.

      This is so wrong that I hardly know where to begin.

      That's like saying, "There's no difference between real sugar and an artificial sweetener, as long as they both taste sweet."

      You're wrong on multiple levels, but thanks for weighing in.

      --
      Just cruising through this digital world at 33 1/3 rpm...
  27. alternate conclusion by Anonymous Coward · · Score: 0

    clearly the humans are the bitter players.

  28. Catholic Schoolgirl can Kick it's Ass. by Grog6 · · Score: 1

    There are already bots in Q3 that are awesome; play it sometime.

    --
    Truth isn't Truth - Guliani
  29. not impressed by houghi · · Score: 1

    Call me when it is playing Thermo Nuclear War.

    --
    Don't fight for your country, if your country does not fight for you.
  30. You assume they weren't doing that already by Anonymous Coward · · Score: 0

    Why do you think AI couldn't memorize paths and timers. Surely that's the easiest thing for them to do.

  31. Agents, like Agent Smith by Anonymous Coward · · Score: 0

    Terrifying name for the AI software instance. "Agent"
    Like Agent Smith of of the Matrix-

  32. The Glory Days by Anonymous Coward · · Score: 0

    I would like to see what this could do on Mr Pants Excessive CTF with BFG not to mention see if it could play BFG Freeze Tag.

    Glory Days of 1.16, FYA and Outlaws Forever, DMF and HoC never.

    Props to Clan Keen on higher PR.

  33. flawed from the start by citizenr · · Score: 1

    "This compared to 43 percent probability for average human players, and 52 percent probability for strong human players"

    Anyone even dabbling in FPS games can spot ho big of a shitshow their testing had to be. 9% difference between pubbies and skilled players? Please. In real life "average" skill team will get steamrolled every single time.

    --
    Who logs in to gdm? Not I, said the duck.
  34. a poor workman blames his tools by Anonymous Coward · · Score: 0

    All the naysayers here are all wrapped up on trying to be defensive, no doubt many of them people who spent way too much of their lives getting good at Quake only to be now fighting the existential angst of realizing they wasted their lives.

    TFA never presents this information as proof of some revolution that establishes the birth of Skynet. It's simply documenting one small incremental step further down the path.

    What is interesting and significant here are two things. That the "bots" weren't given any specific instruction in how to play the game/they learned it simply by playing and that their input was purely through visual recognition.

  35. Doesn''t surprise me by Anonymous Coward · · Score: 0

    Contrary to what gamers would want you to believe, computer games do not require any skills whatsoever.

  36. Sigh by Anonymous Coward · · Score: 0

    Of course they do. Software is faster at parsing data. That does not mean it is 'intelligent', or that it is 'thinking', and software has been able to do this forever. Haven't you ever seen the demo that runs on old arcade games? Same thing, just with less processing power. Can we please just acknowledge that yes, software is good for some things, it isn't 'alive', and just move on? This is getting so boring, I can't imagine being so naive or easily impressed. Certainly doesn't move anything forward.

  37. Q3A rocks, still the best game ever by Anonymous Coward · · Score: 0

    thanks Mr. Carmack, too bad other game developers, and IBM, aren't as ethical or as innovative as John and the rest of the folks at id

  38. So fucking what!? by Rick+Schumann · · Score: 1

    IDGAF about what your gods-be-damned game-bot can do, none of it validates your shitty half-assed poor excuse for real AI!

    1. Re:So fucking what!? by Local+ID10T · · Score: 1

      IDGAF about what your gods-be-damned game-bot can do, none of it validates your shitty half-assed poor excuse for real AI!

      So what if this is not "strong", or "advanced", or "general", or "real" AI. It is not supposed to be. It is machine learning, which is a recognized subset of the field of artificial intelligence.

      Your insistence on pissing all over it does not change the fact that this is real science, and a demonstrable advance in real science, made by real scientists.

      --
      "You want to know how to help your kids? Leave them the fuck alone." -George Carlin
    2. Re:So fucking what!? by Rick+Schumann · · Score: 1

      What I'm 'pissing all over' is the misleading, perhaps intentional, of the general public into believing that this so-called 'AI' they keep trotting out is more than it actually is. You and I may see some clever programming and nothing more, but the general public thinks it's all I, Robot come to life in the real world. That's what the real danger of so-called 'AI' is. They'll trust their lives to it because they've been convinced that it's god-like super-human intelligence when it's not even a fraction of that.