Slashdot Mirror

← Back to Stories (view on slashdot.org)

Startup Uses AI To Create Programs From Simple Screenshots (siliconangle.com)

Posted by msmash on from the interesting-things dept.

An anonymous reader shares an article: A new neural network being built by a Danish startup called UIzard Technologies IVS has created an application that can transform raw designs of graphical user interfaces into actual source code that can be used to build them. Company founder Tony Beltramelli has just published a research paper that reveals how it has achieved that. It uses cutting-edge machine learning technologies to create a neural network that can generate code automatically when it's fed with screenshots of a GUI. The Pix2Code model actually outperforms many human coders because it can create code for three separate platforms, including Android, iOS and "web-based technologies," whereas many programmers are only able to do so for one platform. Pix2Code can create GUIs from screenshots with an accuracy of 77 percent, but that will improve as the algorithm learns more, the founder said.

4 of 89 comments (clear)

  1. I RTFA by Anonymous Coward · · Score: 5, Informative

    It only generates the layout files for the different platforms.

    1. Re:I RTFA by HornWumpus · · Score: 4, Interesting

      Yep, everybody did win. Back in 1990, when Rapid Application Development (RAD in the hype of the day) tools did this.

      Your IDE still has this feature. Drag and drop UI drawing is better than having any UI inferred from a drawing. How do you draw a mask?

      Trying to do this well, transparently for multiple different devices plus 'browser' is challenging, to say the least. But...GOOD NEWS...each of these markets is big enough to support a UI team of it's own. Claiming to do it well, automatically from a 'napkin sketch', for all significant platforms is braggadocious to the point where adults start to whisper about where the person's keeper is, calling him 'Sheldon'.

      But we all remember being 22 and doing similar; 'that's easy, just...' The time it takes from 'that's easy' to 'uhh....shit' is what separates success from failure, long term.

      The best, this will do is produce a 'wrong' (control behavior from a drawing?) UI for a 'sketch artist' who hasn't bothered to learn to use his IDE. Somebody still has to come along, muddling through the messes (one per target), and fix it.

      --
      John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
  2. RAD Redux by Tablizer · · Score: 4, Interesting

    We've had RAD systems for decades. They make the first 80% easy, but not the last 20%. One is always dealing with things like legacy databases with goofy schemas and domain-specific intricacies.

    Tools that may take longer to lay down the basics but can be tuned easier for specifies still seem the best bet.

    Plus you have issues of mobile devices such that UI's need to be "responsive" to different screen sizes. These can take a lot of experimentation to get right because context is involved. They are solving 1990's problems.

    --
    Table-ized A.I.
  3. Re:no need for AI by ShanghaiBill · · Score: 4, Interesting

    Programs that can auto-generate glue code from a GUI input have been around for decades. You just need to fill in the stubs with the other 99% of the code. The problem with these GUI input systems is that the code they generate is fragile and even small changes are often more difficult than just starting over. I have never found them useful, but they are popular for iPhone and Android apps.

    I read TFA, and it says almost nothing, but I think the new thing about this system is that you don't need to use a GUI input, and instead you can just show it a picture or screenshot of an existing GUI (say, your competitor's product), and it will auto-generate code to create that GUI, with stubs for the actual functionality. That seems pretty slick.