Slashdot Mirror
Ubisoft is Using AI To Catch Bugs in Games Before Devs Make Them (wired.co.uk)
AI has a new task: helping to keep the bugs out of video games. From a report: At the recent Ubisoft Developer Conference in Montreal, the French gaming company unveiled a new AI assistant for its developers. Dubbed Commit Assistant, the goal of the AI system is to catch bugs before they're ever committed into code, saving developers time and reducing the number of flaws that make it into a game before release. "I think like many good ideas, it's like 'how come we didn't think about that before?'," says Yves Jacquier, who heads up La Forge, Ubisoft's R&D division in Montreal. His department partners with local universities including McGill and Concordia to collaborate on research intended to advance the field of artificial intelligence as a whole, not just within the industry.
La Forge fed Commit Assistant with roughly ten years' worth of code from across Ubisoft's software library, allowing it to learn where mistakes have historically been made, reference any corrections that were applied, and predict when a coder may be about to write a similar bug. "It's all about comparing the lines of code we've created in the past, the bugs that were created in them, and the bugs that were corrected, and finding a way to make links [between them] to provide us with a super-AI for programmers," explains Jacquier.
La Forge fed Commit Assistant with roughly ten years' worth of code from across Ubisoft's software library, allowing it to learn where mistakes have historically been made, reference any corrections that were applied, and predict when a coder may be about to write a similar bug. "It's all about comparing the lines of code we've created in the past, the bugs that were created in them, and the bugs that were corrected, and finding a way to make links [between them] to provide us with a super-AI for programmers," explains Jacquier.
I remember it being discussed in my software engineering class that trying to automate bug removal or detection could be shown to be isomorphic to solving Turing's halting problem.
File under 'M' for 'Manic ranting'
The more things change, the more they stay the same....
Anybody else remember LINT? I used to work a project that required that all compiler warnings be dealt with and anything reported by LINT was documented and explained IN THE CODE. It certainly didn't catch everything but it sure kept the code consistent and common logical issues from appearing too often.
Now off my lawn....(snicker)
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
AI: "Based on years of historical pattern matching, your commit has been flagged as 'needs review' for the following reasons:
1 - First name of developer is 'Fred'"
If the dev AI/Automation is getting that good. Why are they not just doing a majority of the game development?
;)
;)
;)
I always hear the horror stories about how bad AI and Automation will affect burger flippers, etc.
Seems to me the real jobs that will be affected are the white collar elites jobs. Bankers, Brokers, Developers, Judges, Lawyers, Programmers, etc. Imagine white collar professions minus the bias/corruption/prejudice because it is all just honest AI & Automation
Oh yea, honest AI/Automaton politicians, Sweet
Just my 2 cents
Brown ball means bug we can fix after the release, red ball means we can't ship until the precog AI is made happy.
Maybe this AI can finally convinced them to give an option to NOT have the elevation angle of driving cameras try to reset to default after a couple of seconds? Drives me nuts in all their games, e.g. Watchdogs and Ghost Recon: Wildlands. Especially important if driving a taller vehicle.
Says it can detect 6 out of 10 bugs? Based on what? How would it even know what a "bug" was in the context of the game? The video also says it creates code signatures for bugs, but doesn't explain how. They explain the concept, but does it actually work? What specifically does it do? Without seeing examples, it's hard to imagine this tool does what they seem to be saying it does.
this is ..... not like LINT. Remember LINT? LINT is still in widespread use. This would be a complimentary tool to catch higher level bugs based on code heuristics in valid correct code beyond the safety checks that LINT looks for. This is not for doing what LINT does, and what LINT does is still very useful.
"Old man yells at systemd"
auto QA testing can just fail silently or pass in a way that any real person will see it as an error.
Also poor ui's / control setups are bugs that some auto system can not find as well but QA testers will find.
Yes, indeed. We realized a lot of those compiler warnings were actually trying to tell us something (WOW!) and cleaned up our code base over the course of a number of years. We are now pretty much at the point where compiler warnings are generally viewed as errors, so the bar is very high and requires additional code review if you legitimately need to submit something that triggers a warning ... "the compiler is lying, I'm right" isn't good enough. We have a number of people on staff who are really good at figuring this out before it ships and shows up as a bug. It's also far more reassuring when you see your code compile really cleanly, makes a lot of other stuff far easier.
It's also worth noting that there's plenty of stuff you can do when you are checking in code if your organization has come up with code guidelines. Checking for things like the present of tabs, copyright strings and a number of other things can be enforced.
But it is the same concept as LINT.... It may be different from LINT but this really isn't a new idea, and it IS LIKE LINT...
There have been static code analysis tools in use for decades now, LINT was among the first of these tools to find wide spread use and many have followed in its foot prints. This is NOT a new idea, even if the implementation method varies from the C program the initial LINT was/is.
Read the Mythical Man Month.... There is nothing new... Each generation thinks their stuff is better, that they found the way, the silver bullet. I've watched a couple of generations of new programmers go though the same "Eureka! I've got it!" claims, only to discover the hard way what I discovered, the devil is in the details and there is no magic bullet.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
How am I supposed to write myself a new minivan now?!
Anons need not reply. Questions end with a question mark.
Have gnu, will travel.
It is so "eightyish".
LINT is something that looks at your code for problems and reports them.
A compiler is something that looks at your code for problems and reports them (and also generates output).
This ubisoft tool is something that looks at your code for problems and reports them.
Your colleague standing over your shoulder is something that looks at your code for problems and reports them.
Your colleague who does code-review is something that looks at your code for problems and reports them.
Your copy-editor is someone who looks at what you've written for problems and reports them.
Your spell-checker is something that looks at what you've written for problems and reports them.
Your grammar-checker is something that looks at what you've written for problems and reports them.
It's kind of silly to say "this is LINT!" or "there is nothing new" as regards this story. The task of having feedback on what you've produced is at least as hold as humankind itself and has existed in every area of human endeavor, in a huge variety of ways, with feedback performed at a huge variety of times. Of course that aspect isn't new. Beyond that, this Ubisoft is as (un-)related to LINT as every single other tool I listed.
Still no cure for Ubi's stupid design choices.
semantics are everything!
See, AI is replacing what we (consumers) used to do, especially for companies like Ubisoft. What do they expect me to do now, play the game without issue and enjoy myself? What am I going to complain about now?
delete this
"I'm sorry. I can't do that, Dave."
Yea, I used to say that kind of thing when I was young too. Hubris lives on in the young. We had all the good ideas then too, we where better educated, fresh out of school and full of promise. But we where as stupid as those who came before us. Wisdom is hard won though experience and I've personally learned the grey beards of my day where right, there is really nothing new. Programming remains the same problem, though the names and faces have changed.
Face it.. At the very best, this is just an extension of LINT and just static code analyzer like LINT was/is. They've been doing this kind of thing long before I came into this field fresh out of school and when you are old and grey, the newbies will "invent" the same thing AGAIN, and you can dig up this post and quote it.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
You should read the article.
It has absolutely nothing to do with LINT and is not the same concept.
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
No, it is not a static code analyzer.
You seem to old to grasp new concepts.
Why don't you read the article instead of continuing to make an idiot out of your self?
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
So you've used Resharper too?
This sounds more like Clippy.
"I see you are trying to write a state machine..."
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
For any given program f running on Microsoft Windows it will halt if you let it run long enough.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Do people really not use -Wall -Werror (or equivalent) by default? Aside from stuff being put on github which apparenrtly has a requirement to spew megabytes of warnings when compiling...
Not a bad thing. That's actually quite useful. State machines and dates and other things seem to be items that programmers always re-invent, despite there being a half dozen of them in the libraries they already use.
Imagine the usefulness one could have if the IDE simply stated "it appears you're implementing a date function. Have you considered the date and time APIs already available to you? Here's some APIs and documentation."
This is especially true for languages with rich APIs that do everything, yet everyone seems to reimplement them wrongly anyways.
Want to impress me further? See to it that it sees I implement bubblesort, but it stays quiet if it knows the data set I'm working with is small. If I make it bigger, it then flags it. (Bubblesort is perfectly fine as a sort algorithm for small datasets - like say, 10 elements)
Good to have Auto QA though to free humans for what they're good at. Also good to have auto-play-testing with AI agents just running against walls etc. If any AIs get stuck or position.z -100 and have fallen out of the game level then you can replay their game route.
Save your play testers for quality not random breaking of the game.
I believe that in the movie, they started using AI to lint their code, then it became self aware and just started writing the code and finally gave rise to the terminators which put all humans into a simulation to harvest their bioenergy. Or something like that...
Let me try again with a car analogy. Someone designs and builds a Tesla. You come along and say "Face it. At the very best, this is just an extension of the '87 Ford F150. There's nothing new. They've just re-invented the '87 Ford F150. Oh the hubris of the young, to think that they could do anything different."
What's weird is that your "prior art" is (1) oddly specific almost in ignorance of the rest of the field that came before and after, (2) misses the point that Tesla has taken one existing technology "battery" and put it into another existing technology "electric vehicle" in a well-designed package that works well and is interesting for these reasons.
Oh, grasshopper... This is not new. It's doing static code analysis and flags parts that may have issues based on where trouble has been seen in the past. This is EXACTLY what LINT is, advice from experience. It's basically saying "Um, you *might* not want to do this kind of thing because it's often a mistake. Are you sure?"
Of course you somehow think that because it's some fuzzy AI technique used it's somehow different? Cute....If anything, it's less effective being AI, but that's another debate you won't understand.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
I like you analogy, but consider this small addition...
Let's say I tell you that we discovered a whole pile of rules for designing cars. Don't try to stamp metal into this kind of shape, mount your glass in ways the flexing of the body doesn't break it, don't use plastic for this kind of part or that, don't use 6V lamps or non-rechargeable batteries, don't put the gas tank too close to the bumper or put electrical wires and exhaust manifolds near it either.... We use these "rules of the road" to validate our design changes as we make them.
You, invented an AI way to look for this same list in your vehicle designs....
The concept of that Tesla isn't new, a battery operated car you can drive on the roads.. Only the implementation is new. Tesla didn't invent the electric car, they just implemented one.
This 'new' program was invented back in the days of LINT, this is just a new implementation of one. It's not even better than LINT if you ask me, but being AI based, there's no way to prove that objectively.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
It looks like you're writing a loop! Should I refactor it into linq and sneakily introduce an O(n^2) complexity?
No, it is not a static code analyzer. You seem to old to grasp new concepts. Why don't you read the article instead of continuing to make an idiot out of your self?
From the Article above:
"It's all about comparing the lines of code we've created in the past"
That sure sounds like "static code analysis" to me. Perhaps you don't understand that that term means?
Static Analysis of source code is looking at the source code for interesting patterns, in most cases looking for common programming errors. LINT did this, this "new" program does the same thing. It looks at the source code for patterns right? Then it's doing static analysis.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
From the Article above:
"It's all about comparing the lines of code we've created in the past"
That sure sounds like "static code analysis" to me.
It IS static code analysis. But that doesn't mean it is "like Lint".
Lint uses a table of HUMAN GENERATED patterns. These patterns are labor intensive to produce, and only find bugs that humans thought to check for.
This new checker looks at a steadily expanding database of bugs, and the fixes for those bugs, and LEARNS THE PATTERNS ON ITS OWN. This means it can have a much bigger set of patterns, including many that a human might have never thought to include. It also means that the system can steadily improve.
Most likely this system will be used as a supplement for Lint, rather than a replacement. But as it learns and improves, it may make Lint no longer very useful.
I mean, maybe just fire Josh?
Staffing decisions are outside the scope of the tool's remit.
... and how does it compare to existing static code analyzers? It's not that static code analysis is a completely new technique.
There has been plenty of long-term research on this topic under the name "defect prediction". It is actually one of the best (or few..) suited application areas in the software engineering processes for machine learning. You get an obvious set of training data. Record all the bugs found, issues reported, match them to fixing patches (or whatever you call it, fix locations to code in practice). Label those places/parts of code as "error-prone" before the fix. Maybe take the "after" part as an example of non-issue. Train your machine learning classifier.
It's been done for a long time. Much like most semi-academic application areas, this is again done with university-company collaboration. Because no company on itself ever wants to get into this type of stuff due to seeing little real benefit beyond your basic, easily available and applicable, analyzers, and writing those (unit) tests that you should write anyway. In similar way to topics such as "regression test optimization". Decades of slapping on it, minor improvements if any, but with creative reporting, always some Uni finding a "partner" to get funded and help them put the academics to little bit of practice, as it otherwise is not so profitable.
So here they found a better known company (Ubisoft) and decided to call machine learning AI, brilliant hype generated. Now lets have a project meeting and report to our funding agency about how we made world-wide news with our super-AI tech and globally big companies. OMG have many more millions you so good. Because the funding agency will not understand jack about what is going on there, and if they do, they will still be impressed someone made something working in some way. Which sadly is a true achievement in that context. ...
Compiler warnings are often viewed as "noise" and disregarded entirely, or "logged and fixed in the next release". We learned our lesson the hard way years ago and have moved to warning-free code as a checkin requirement, but it would not surprise me to find a lot of organizations with date-driven releases who let them slip, especially as the ship date get ominously close. We pretty much use -Wall -Werror, but occasionally I need to deal with stuff from github and it's warning city. It's kind of like learning hygene is really good and then having to go somewhere soap hasn't been invented.
It is not _code analysis_, hence it can not be _static code analysis_
But perhaps you only want to nitpick with words.
The tool knows nothing about the code or the programming language.
It only knows I fixed a bug in line 100 of a text file.
And bottom line you can use the same technique for editing books. "The lector says 'this phrase is bad', the author changes it to 'this sounds better'". The AI checks for similarities in the rest of the book and flags them.
In other words: it knows nothing about the topic it is looking at.
LINT knows a lot about the language it is linting, after all it uses the same parser and AST as the compiler.
Historically you had the compiler and lint because machines had not enough memory to have in depths semantic analysis AND code generation in one program. So they wrote two programs, which share most their code. One does an in depths analysis about the code and outputs warnings (warnings that often _are_ errors). The compiler only wants to compile and emit code. It does not care about many things. Which is also considerably faster on classic machines (machines with a few KB of RAM, e.g. PDP-11 and Apple ][, yes, I programmed in ancient C on Apple ][s - I believe Aztec C)
So, a LINT and the approach of Ubisof: have absolutely nothing in common
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
If this tool doesn't understand the programming language of the code, Then... It's pretty much worthless compared to LINT. Shall we send all our code to a group of English grammar experts for their comments before we allow it to be committed too? What's the point of that?
In any programming language there are commonly used sequences which are prone to errors, which are syntactically correct, but likely wrong. Your AI based search would produce lots of useless garbage if it cannot tell the difference between a comment and actual code.
I'm going to say it only one more time. Static code analysis (looking at the source code) is what LINT did and does. Your brand "new" AI program, does the same thing. If it works as you claim, it likely also produces a lot of false positives OR if the thresholds are set high, misses a lot of common mistakes and you'd be better off tossing the AI part out and just hard code it, like LINT did....
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
This is called static analysis and has been around for YEARS. Tons of companies are doing this.
GCS/MU/P d- s:- a-- C++++$ UL++ P+ L++ E+ W++ N o K- w--- O M+ V- PS+++ PE Y+ PGP t+ 5- X R++ tv+ b++ DI++ D++ G+ e++ h-
I'd call the made-for-VR movie, "Do Androids evolve to become veterinarians specializing in sheep?"
"There is no god but allah" - well, they got it half right.
from my understanding a game "dev" does very little programming in the traditional sense
I work on a VR project at my day job (basically a video game). Your statement that we use a "pre-made" game engine is basically like saying that nobody does any "real" programming if they use a library or some code that someone else wrote. In other words: yours is a no-true-scotsman statement.
These guys are mostly doing scripting
We used both C++ and Unreal Script when we were using UDK (Unreal Engine 3). But everything we do now with Unreal 4 is C++ (and some Blueprint stuff to hook the GUI together).
world building, animation, modeling, progress triggers, hit boxes, etc.
Yes, that's the problem domain. And how do you think these things get implemented? With code.
So assuming I'm correct on that, Lint would be useless for this.
You're not really correct on all that. Also Lint and other static analysis tools are useful. John Carmack has some great insight into all this and how static analysis tools can help and fit into game development. As evidence: the Doom 3 source code is pretty generally very good.