Slashdot Mirror


Japan Researchers Develop Automated Technique For Anime Colorization Using Deep Learning (brightsurf.com)

Japanese researchers have developed a technique for automatic colorization in anime production. "To promote efficiency and automation in anime production, the research team focused on the possibility of automating the colorization of trace images in the finishing process of anime production," reports Brightsurf. "By integrating the anime production technology and know-how of IMAGICA GROUP and OLM Digital with the machine learning, computer graphics and vision technology of NAIST, the research team succeeded in developing the world's first technique for automatic colorization of Japanese anime production." From the report: After the trace image cleaning in a pre-processing step, automatic colorization is performed according to the color script of the character using a deep learning-based image segmentation algorithm. The colorization result is refined in a post-process step using voting techniques for each closed region. This technique will be presented at SIGGRAPH ASIA 2018, an international conference on computer graphics and interactive techniques, to be held in Tokyo, Japan on Dec. 4-7. While this technique is still in the preliminary research stage, the research team will further improve its accuracy and validate it in production within the anime production studio. The product of this will be available for commercialization from 2020.

77 comments

  1. How many frames by AHuxley · · Score: 4, Interesting

    How many artists will get to start their own work rather than filling in frames for others?

    --
    Domestic spying is now "Benign Information Gathering"
    1. Re:How many frames by rhsanborn · · Score: 4, Insightful

      How many artists won't get the entry-level apprentice-like opportunity to learn from people with more experience because this role has been automated?

    2. Re: How many frames by Anonymous Coward · · Score: 0

      Most artists never color anime you know. It is not only specialized but hardly anyone I ever met likes the work

    3. Re:How many frames by mikael · · Score: 4, Informative

      Disney offshored that work from Hollywood to the Far East decades ago, then brought it back when it was automated by computer. Artists from the art schools are way more qualified than to inbetween and fill in paint cells.

      If you look at how they did many of those early animations, there were motion artists who took movies of dancers, rotoscoped the animation to get generic stick figures for reference. These frames were a closely guarded secret as they were reused between animations. When it came to make an animation, they took those reference frames, outlined each with a simple body, get detail artists do a fully detailed frame, then hand those over to inbetweeners who did the intermediate animation frames, and finally to the cel painters who outlined and filled in each individual frame. The deep learning system replaces these stages.

      https://en.wikipedia.org/wiki/...

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    4. Re: How many frames by parkinglot777 · · Score: 1

      Most artists never color anime you know. It is not only specialized but hardly anyone I ever met likes the work

      I am guessing you are from a western country. I highly doubt you really know how Japanese comics were published back in 80's or 90's (before anime). IIRC, Anime nowadays was called girl cartoon back then because the drawing was very girlish and most of them were about young people love stories.

      Anyway, back then the artist (the author) would draw outline of the comic. The artist will have a helper or two (depending on how famous his/her comic is). The helpers are those who "fill in color" and enhance the outline. Some of these helpers will become new artists later on. That is what rhsanborn was talking about.

    5. Re: How many frames by Anonymous Coward · · Score: 0

      I highly doubt you really know how Japanese comics were published back in 80's or 90's (before anime).

      There's just a teensy bit of ambiguity here -- you're not saying that anime wasn't around, back then...are you?

    6. Re:How many frames by jellomizer · · Score: 1

      This is the general problem across the first world economies. The gap between entry level and skilled level employment.
      The good old traditional work your way up method we hear.
      0. Get a degree (lets say accounting)
      1. Start a job in the mail room, you get to know the peoples name and the departments.
      2. Move up to delivery where you get to know the people.
      3. You find begin to interact with the accounting department
      5. Do some internship with that department, enough to show that you know your stuff.
      6. Get into Accounting
      7. Get Promoted
      8. Goto 7 until you are at the top.

      Today for most organization this method is garbage.
      There is no Mail room or delivery.
      Internships are just often glorified coffee servers
      Accounting department will only hire people with years of full experience
      The job promotion path the company wants skills needed for the promotion. So you want to get promoted to manager, what experience do you have as a manager.
      Higher management wants an "outside view" of the organization so they don't promote up.

      This causes the ambitious employee to find a new place to work every few years.

      Now Technology is part of this problem, as it good at historic entry level work. However businesses have failed its employees and started treating them like equipment and not partners on a shared goal.

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    7. Re: How many frames by Anonymous Coward · · Score: 1

      was called girl cartoon

      still called girl cartoon

    8. Re: How many frames by UnknownSoldier · · Score: 2

      I'm not sure how Manga worked but the West there used to be a few departments:

      * Penciller
      * Inker
      * Letterer
      * Colorist

    9. Re:How many frames by AlanObject · · Score: 2

      How many artists won't get the entry-level apprentice-like opportunity to learn from people with more experience because this role has been automated?

      Note that much anime grunt work in Japan has been farmed out to Korea where the labor is cheaper. Just look at the credits of any anime that aired in the last several years -- almost all Korean names. Not only the coloring but the in-betweeners.

      So the problem you are pointing out already happened. Some jobs will remain but mostly for the higher-end stuff.

  2. Pass by stealth_finger · · Score: 3, Interesting

    Once you start to get 3d assets and computerised stuff in anime it loses that look which gives it that x factor. It's too clean and looks wrong. IMO at least.

    --
    Wanna buy a shirt?
    https://www.redbubble.com/people/stealthfinger/shop?asc=u
    1. Re:Pass by GeLeTo · · Score: 2

      This is done with Deep Learning. If the source material used to train the neural network has the "x factor" and does not look clean - so should the output result.

    2. Re: Pass by Anonymous Coward · · Score: 0

      I suppose this is offered by multiple studios - pricing clearly ranges wildly

    3. Re:Pass by stealth_finger · · Score: 1

      This is done with Deep Learning. If the source material used to train the neural network has the "x factor" and does not look clean - so should the output result.

      By its own operation it's going to smooth everything out when it's trying to make an amalgamation of all its source material. It'll probably look good sure but will be missing something.

      --
      Wanna buy a shirt?
      https://www.redbubble.com/people/stealthfinger/shop?asc=u
    4. Re:Pass by skovnymfe · · Score: 1

      I agree. Just look at all the awful netflix ones.

    5. Re:Pass by religionofpeas · · Score: 1

      By its own operation it's going to smooth everything out when it's trying to make an amalgamation of all its source material.

      Deep Learning is good at representing complex patterns, such as which designs can or cannot be used together, so that's most likely not going to be a problem. A standard technique is to use adversarial networks to provide feedback on the quality by comparing the generated material to original sources. Anything that stands out, such as excessive smoothing, will be used to correct the generating network.

    6. Re: Pass by Nidi62 · · Score: 1

      South Parks anime rip looked pretty good.

      --
      The only thing necessary for evil to triumph is for it to be pitted against a slightly greater evil
    7. Re:Pass by AmiMoJo · · Score: 1

      It's been happening slowly for a couple of decades now. An early example would be Azumanga Daioh, which was drawn in pencil as usual but then scanned and inked and coloured digitally. It looks pretty normal, most people at the time didn't realize.

      Later even Studio Ghibli started using digital stuff for in-between frames, although all the key stuff is hand painted.

      3D and digital has massively reduced the cost of producing anime and vastly improved the quality. Early 3D stuff did stand out, but now when they do it well it's seamless. Sometimes they just use it as a guide and then hand draw over it, like they did with the Hollywood movie Tron.

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    8. Re:Pass by Anonymous Coward · · Score: 0

      The BBC actually had an article recently on the strengths and weaknesses of deep learning in art. It seems it might be decent at coloring things, but would be terrible at drawing them as it has trouble understanding interactions between things. It was an interesting "read" even if written to a very low level. It's mostly a gallery with captions.

      http://www.bbc.com/future/gallery/20181127-the-weird-way-machines-with-ai-see-the-world

    9. Re:Pass by AmiMoJo · · Score: 1

      The biggest reason for the declining quality is the reduced time given to production. Anything that speeds it up and leaves more time for review should improve the animation.

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    10. Re: Pass by stealth_finger · · Score: 1

      South Parks anime rip looked pretty good.

      Yeah but it didn't look like Akira.

      --
      Wanna buy a shirt?
      https://www.redbubble.com/people/stealthfinger/shop?asc=u
    11. Re:Pass by mikael · · Score: 1

      I agree. Ulysses 31 is a good example. The French did the story lines, but they needed the Japanes Anime artists help to really give those animations a means to convey intense explosions. Sometimes they did that by mixing purples blues and whites, for supernova, red/orange/yellow/black for an exploding planet, or black/white cracks with god rays for strange exotic matter.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    12. Re:Pass by stealth_finger · · Score: 1

      I was watching a video on youtube the other day with some guy trying to address it, because hand drawing that stuff takes stupid amounts of time. Basically he 3d modelled this ship, animated it and exported it as lines and coloured and animated over the top. It did actually look pretty good at the end.

      --
      Wanna buy a shirt?
      https://www.redbubble.com/people/stealthfinger/shop?asc=u
    13. Re:Pass by stealth_finger · · Score: 1

      That's true but the same could be said of most things lol

      --
      Wanna buy a shirt?
      https://www.redbubble.com/people/stealthfinger/shop?asc=u
    14. Re:Pass by Anonymous Coward · · Score: 0

      I think a lot of people misunderstand "digital".

      When Azymanga Daioh was digitally inked and coloured, this does not mean "the computer did the work". You still had people doing the inking and colouring, just they replaced inking pens and pain brushes with a computer stylus. An image drawn digitally is still hand drawn. Same artist. Same skills. The only thing different is what they're holding in their hands. It reduces costs and time by taking out the large quantity of materials needed, as well as the step of filming the individual cels. Corrections can be made to individual cels without reshooting an entire sequence. And those corrections can be made more easily, again by an artist.

        In this case, the colourists working on inbetween cells are taken out of the process entirely. I am concerned about how this technology will affect the workforce, the other side of the coin is how it could potentially be a boon to independent artists and animation studios. Regardless, this is coming so the industry will have to adapt.

    15. Re:Pass by thegarbz · · Score: 1

      I disagree. I actually quite like the quality of the artwork in the Netflix ones. The big problem with the Netflix ones is that more often than not the story is absolute garbage.

    16. Re:Pass by Anonymous Coward · · Score: 0

      Well, on one hand it is like you say. Automating away coloring means more resources to improve quality of the rest.
      On the other hand this only works if there is no demand for increasing profits.

      With coloring gone that means that means that any future saving has to be done by cutting down on the quality of everything else.

      The best scenario is that Anime will be cheaper. The worst scenario is that the saving will be pocketed as profits.
      I don't see a world where the studios think that they should spend more on quality since they got away with the current level.
      Why spend more than you have to?

    17. Re:Pass by zlives · · Score: 1

      i for one like
      B: The Beginning
      Knights of Sidonia
      castlevania
      and others
      not sure if they are all netflix but i for one enjoy them

    18. Re:Pass by Anonymous Coward · · Score: 0

      It reduces costs because there's no longer any physical materials to deal with, and the organization/storage of such materials (often sold to fans before digital was a thing.)

      It does not reduce the labor costs one bit. If anything, digital just shifts the workloads around. So in the pre-digital days you'd have a lot of people just fixing mistakes or redrawing things. In post-digital, the original artist can just fix things, and if they don't have time for it, they hand the file off to someone else to fix. An entire production line can be setup per scene, and broken up among multiple production lines, so your staff for episode 1, 4, 7, etc might all be the same guys, while episode 2, 5 and 8 might be the other team. Episode 3 and 9 might be worked on by separate teams that aren't in-house. But you still need people who can draw the storyboards, and without those, you don't know what crazy drawings you'll get back.

      Most of the in-house staff at Animation and Gaming companies are the artists. In a gaming context, a game can proceed with missing assets and added in at the last minute. In an Anime or Cartoon production you can keep fixing things until broadcast, and then you have to fix more of it for the Blueray release otherwise you won't make enough money for the next production.

  3. Re:Why don't Africans do this? by Anonymous Coward · · Score: 0

    1) Because "tech" media is dominated by predominantly white males that need to high five themselves. Even if there are inventions from Africans you never hear about them on ./
    2) Colonization of Africa by the Europeans, that implies enslavement, systematic killing, and the depletion of natural resources, threw Africa back in development decades if not centuries. Hence, there are more urgent problems than fucking anime colorization using deep learning.
    3) Get out you mothers basement you incel cunt. Maybe go to Africa to get a picture for yourself.

  4. Text article by religionofpeas · · Score: 4, Insightful

    What's the point of a short text article about graphics without adding any pictures/video clip as demonstration ?

    1. Re:Text article by Anonymous Coward · · Score: 0

      Even the researchers' site has no images or proof about their achievement.

    2. Re:Text article by jellomizer · · Score: 0

      Mostly likely this deep learning algorithm found the GWBASIC PAINT command.

      10 SCREEN 7 ' It is 2018 lets use EGA Graphics
      20 BLOAD "FRAME1.BLD"
      30 PAINT 160, 100, 1

      chances are I just filled in something blue!

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    3. Re:Text article by Anonymous Coward · · Score: 0

      It crashes because it can't find frame1.bld

    4. Re:Text article by jellomizer · · Score: 1

      Well make one. Open a Hex editor and make a 64000 byte file. and fill it with 0's and F's (for black in white) assume a 320x200 screen. so for every 320 bytes the data will wrap down to the next line. and save it as FRAME1.BLD

      That is how you do 4 bit graphics!

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    5. Re:Text article by jellomizer · · Score: 1

      Correction it is only 32000 bytes. 4 bit graphics not 8.

      --
      If something is so important that you feel the need to post it on the internet... It probably isn't that important.
    6. Re:Text article by srmalloy · · Score: 1

      From the reference in the article to submitting the paper at SIGGRAPH Asia, this announcement is an attempt to drum up interest in advance of the conference; releasing multimedia prior to the conference would dilute the interest -- "oh, I've already seen that; I'll go watch a _new_ presentation" -- and reduce the attendance at their presentation. Expect it to be flogged much more heavily, with extensive stills and video, after the conference.

  5. Re:Why don't Africans do this? by Anonymous Coward · · Score: 0

    Why is it that you NEVER get someone with a high IQ making posts like the parent.

  6. Not one sample page, how nice! by Anonymous Coward · · Score: 0

    I suppose an article like that would have samples showing the technique in action, but I guess it's asking too much

    1. Re: Not one sample page, how nice! by Anonymous Coward · · Score: 0

      The article says it's still in development, so presumably there simply isn't any action yet.

      Once again, 'deep learning', 'training' - these are misnomers, as is 'AI' itself. We've been using algorithms for things like tweening and filters etc. for a very long time. Automation isn't remotely new. Why are millennials so retarded about this stuff?

  7. Re:Why don't Africans do this? by Anonymous Coward · · Score: 0

    1) Because "tech" media is dominated by predominantly white males that need to high five themselves. Even if there are inventions from Africans you never hear about them on ./

    Circular argument. African inventions are suppressed conspiracy theory is risible.

    2) Colonization of Africa by the Europeans, that implies enslavement, systematic killing, and the depletion of natural resources, threw Africa back in development decades if not centuries. Hence, there are more urgent problems than fucking anime colorization using deep learning.

    Enslavement, systematic killing, and the depletion of natural resources were endemic to Africa and Africans. The Europeans just made it worse.

    3) Get out you mothers basement you incel cunt. Maybe go to Africa to get a picture for yourself.

    Ad hominem.

    Is that really the best you can do to refute an obvious troll and racist?

  8. Re:Why don't Africans do this? by Anonymous Coward · · Score: 0

    Why is it that you NEVER hear of Africans EVER inventing something like this?

    Do Africans even LIKE anime? Why on EARTH they would even WANT to invent this?

    (it's so fun captioning words)

  9. AI? by mrwireless · · Score: 1

    To whomever wrote this: thank you for calling this Deep Learning instead of going "OMG AI".

    1. Re:AI? by 110010001000 · · Score: 2

      We used to call them "computer programs" but "Deep Learning" sounds more research-y.

  10. Re: This FP for gNAA by Anonymous Coward · · Score: 0

    We were billed for this service and paid but did not receive it. We also receive this service for free from our internal departments as well as for free from vendors we have worked with in the past. If there is some sort of service differentiation I am not seeing what that is. Maybe the geniuses among us can sort it out.

  11. Re: Why don't Africans do this? by Anonymous Coward · · Score: 0

    Is that really the best you can do to refute an obvious troll and racist?

    No.

  12. Re: Why don't Africans do this? by Anonymous Coward · · Score: 0

    Read this guys future posts and that will give you a clue

  13. Re: Why don't Africans do this? by Anonymous Coward · · Score: 0

    Africa had castles and working aquaducts before the wh*te devil made them all into slaves.

    They're only just recovering now, 60 years after commiting ethnic cleaning against white Rhodesians, Hatians, and countless other European folk

  14. hentai by NuclearCat · · Score: 1

    Interesting how it will perform on it :)

    1. Re:hentai by Anonymous Coward · · Score: 0

      Is there any black-and-white hentai that needs colorizing?

    2. Re:hentai by Anonymous Coward · · Score: 0

      Most times, the interesting bits are left out by law. Which is why there are squirting tentacles, flailing all over the place.

    3. Re:hentai by Anonymous Coward · · Score: 0

      There really aren't that many tentacle hentais vs "normal" hentais. Maybe about 5% of the total hentais made have tentacles, going off numbers on kisshentai and myanimelist. There was one manga artist, Toshio Maeda, in the 1980s who was the "tentacle guy" and all the major tentacles hentais from 80s and 90s were based off his mangas. Tentacles are a thing in the horror subgenre of hentai, but most creators and viewers don't create/watch horror hentais.

      And if you watch any actual "tentacles hentais" expecting to see a lot of tentacles, you will in fact be disappointed. Horror-themed hentais tend to throw a short tentacle scene in there as a sort of nod to Toshio Maeda, so they can say they've got tentacles in it. you'll watch three hours of other shit for the 50 second tentacle scene.

    4. Re:hentai by Anonymous Coward · · Score: 0

      Is there any black-and-white hentai that needs colorizing?

      In manga form, yes. It's an enormous market, as these things go. The vast majority of eromanga is in black-and-white, sometimes with a handful of color pages, but often only with color covers.

    5. Re:hentai by Anonymous Coward · · Score: 0

      The vast majority of ero-manga and eroge (ero-game) deals with material that is hard to sell in the first place. That's why many of these things are only available at conventions.

      However there's an even smaller section of "blatantly illegal if they were flesh and blood", most lolicon, BL, furry, and hyper-violent stuff would never see the light of day in the west if it weren't for fans of this stuff in Japan talking about it and pirating it. Then you have the internet dumpster-fire sites that complain that this material doesn't make them hard/wet enough and demand that someone translate, de-censor, and color these things.

      I had to laugh one day, I found one of these ero games on steam and it's been "decensored"... and the bits that were decensored were like, tiny. Like the crotches on the women were glued on stickers for all I cared.

      Anyway, if they can color black-and-white material, they could likely de-censor the black bars or mosaics as well.

  15. Re: Why don't Africans do this? by Anonymous Coward · · Score: 0

    No such thing. White devils are smarter than that. They never colonize

  16. Deep Learning where it counts in 2018 by Anonymous Coward · · Score: 0

    This is the type of task that can really benefit from neural networks and deep learning, tasks that are somewhat repetitive and time consuming and follow a certain pattern and set of rules. Funny how you can always count on the Japanese for optimizing current technology, it's really what makes them formidable

  17. Too bad by OrangeTide · · Score: 1

    Too bad anime plots seem computer generated these days.

    --
    “Common sense is not so common.” — Voltaire
    1. Re: Too bad by Anonymous Coward · · Score: 0

      No. They are usually generated by a hopeful new manager who has no clue how script writers work

    2. Re:Too bad by Anonymous Coward · · Score: 0

      Hell, you need to be a computer to figure out what's going on in some of them... "In this episode, five separate fights will break out between seven different factions consisting of twelve new characters, eight characters you've forgotten about, three characters you thought were dead, and six main characters with new character models. All with the ultimate goal of, um, we'll figure that out in a dozen episodes. And everyone wears black while standing in shadows at night."

  18. TRUMP DID IT! by Anonymous Coward · · Score: 0

    Trump finally did it!

  19. Re: Why don't Africans do this? by Anonymous Coward · · Score: 0

    We wuz kangs

  20. Re: Why don't Africans do this? by Anonymous Coward · · Score: 0

    Lovely

  21. Re:Why don't Africans do this? by QuadEddie · · Score: 1

    Hierarchy of needs applies here. If you civilization never overcomes the basics like security, having a good home, and easy access to a variety of nutritionally-rich food, then you won't advance to abstract concepts like high art. If IQ doesn't allow for the bottom tier of those things, the genetic mutants (higher IQ people) won't thrive and have their genes valued.

  22. Automated upscaling of art with neural networks by ToTheStars · · Score: 1

    In related news, there's a program called "waifu2x" that uses deep learning and neural networks (and CUDA, if you have an NVIDIA graphics card...highly recommended, there's a ~4x speed boost on my machine) to automatically scale up ("super-resolution") and de-noise anime art: https://github.com/nagadomi/wa...

    For Windows users, use this version: https://github.com/lltcggie/wa...

    1. Re:Automated upscaling of art with neural networks by Anonymous Coward · · Score: 0

      That's old stuff, the new hotness is DeepCreamPy, deep learning-based decensoring.

  23. Sort of an empty post by argStyopa · · Score: 1

    RTFA and there's nothing there.... just "we can do this" but no pics (likely because they're going to announce the technique formally in Dec). I expect that this will look more or less something like the cell-shaded early CGI animation.

    Obviously, this is about "optimizing" the drudgework of colorizing, meaning the teams of sub-artists on productions will just be a little smaller.

    I guess everything depends on how persuasive this looks; it could be cool and seems like a relatively low-hanging fruit for automation of the process. Even if it looks shitty, the consumer audience has such low demands that they've (I guess) come to accept the already-significantly-automated interpolation and heavy-trace processes common to the cheaper anime/cartoons.

    --
    -Styopa
  24. Lol. You have no idea... by Anonymous Coward · · Score: 1

    How much modern Anime is CGI. You can only tell when they fall down (like making a character too detailed so the cell shading ends up with artifacts instead of large flat segments that don't change much during a scene.)

    Everything I have watched this year is CGI, and most either uses 'grain' over the CGI, or just leaves it in. Especially bad are the Americanized CGI Animes, like new Berserk, Fist of the North Star Regenesis, Baki (Netflix's adult oriented stab at these), Goblin Slayer (which I can't recommend enough, adult lots of D&D references but a flavor of its own.)

    For anyone who wants to complain, this goes all the way back to Akira and composited work has been going on at least since Techi Muyo: GXP!

    Anime as you knew it is and has been dead for 35 years. A few dinosaurs stayed around after that, but production turnaround and budgets haven't resulted in material that wasn't computer assisted in at least 12 years.

  25. Re: Why don't Africans do this? by Anonymous Coward · · Score: 0

    You do know it was mostly blacks enslaving blacks (and still is)? Who do you think the whites bought them from?

  26. Wait for the bug by Anonymous Coward · · Score: 1

    I bet we will see some weird mistake, like pink hair.

  27. Good by TimMD909 · · Score: 1

    I've waited years for the second seasons for Shingeki no Kyojin and One Punch Man. Get on it, Japan. I'm counting on you to learn Japanese.

  28. Want to try it out? by Anonymous Coward · · Score: 0

    There are a couple ones online if you want to try them. Style 2 Paints will do automatic colorization, and Make Girls Moe will autogenerate anime faces.

  29. Anime's X factor by thegarbz · · Score: 1

    Once you start to get 3d assets and computerised stuff in anime it loses that look which gives it that x factor.

    Anime loses it's great and compelling stories, character development, excellent voice acting, and overall joy to watch because a computer was involved in drawing?

    1. Re:Anime's X factor by stealth_finger · · Score: 1

      Not unless a computer does the story, characters or voice acting. I'm only talking aesthetics here.

      --
      Wanna buy a shirt?
      https://www.redbubble.com/people/stealthfinger/shop?asc=u