Startup Uses AI To Create Programs From Simple Screenshots (siliconangle.com)

An anonymous reader shares an article: A new neural network being built by a Danish startup called UIzard Technologies IVS has created an application that can transform raw designs of graphical user interfaces into actual source code that can be used to build them. Company founder Tony Beltramelli has just published a research paper that reveals how it has achieved that. It uses cutting-edge machine learning technologies to create a neural network that can generate code automatically when it's fed with screenshots of a GUI. The Pix2Code model actually outperforms many human coders because it can create code for three separate platforms, including Android, iOS and "web-based technologies," whereas many programmers are only able to do so for one platform. Pix2Code can create GUIs from screenshots with an accuracy of 77 percent, but that will improve as the algorithm learns more, the founder said.

Comments

no need for AI

[bugs2squash]

I'm pretty sure code generators have been able to accept input from graphical layout editors for a while. Just what is this AI "inferring" ?

Re:no need for AI

[ShanghaiBill]

Programs that can auto-generate glue code from a GUI input have been around for decades. You just need to fill in the stubs with the other 99% of the code. The problem with these GUI input systems is that the code they generate is fragile and even small changes are often more difficult than just starting over. I have never found them useful, but they are popular for iPhone and Android apps.

I read TFA, and it says almost nothing, but I think the new thing about this system is that you don't need to use a GUI input, and instead you can just show it a picture or screenshot of an existing GUI (say, your competitor's product), and it will auto-generate code to create that GUI, with stubs for the actual functionality. That seems pretty slick.

Re:no need for AI

[hsmith]

I guess one good thing is this isn't "blockchain" based - so that is one improvement.

Re:no need for AI

[smallfries]

The output from a layout editor is a structured description of the components and their layout. This is inferring that description from a .png - the LSTM is building a description of the structural relationship between the widgets from the input image.

It looks pretty cool, although quite simple. The intermediate token stream that it is inferring may be more interesting as a design tool than the neural network on the front-end that is building it.

Re:no need for AI

[Wootery]

Who will save us from the cyber AI revolution?

Idea for an Arnie flick...

I RTFA

[Anonymous Coward]

It only generates the layout files for the different platforms.

Re:I RTFA

[kaizendojo]

Which frees designers to work on UI and coders to concentrate on tighter code instead of the tedium of coding UIs from sketches. Everybody wins.

Re:I RTFA

[HornWumpus]

Yep, everybody did win. Back in 1990, when Rapid Application Development (RAD in the hype of the day) tools did this.

Your IDE still has this feature. Drag and drop UI drawing is better than having any UI inferred from a drawing. How do you draw a mask?

Trying to do this well, transparently for multiple different devices plus 'browser' is challenging, to say the least. But...GOOD NEWS...each of these markets is big enough to support a UI team of it's own. Claiming to do it well, automatically from a 'napkin sketch', for all significant platforms is braggadocious to the point where adults start to whisper about where the person's keeper is, calling him 'Sheldon'.

But we all remember being 22 and doing similar; 'that's easy, just...' The time it takes from 'that's easy' to 'uhh....shit' is what separates success from failure, long term.

The best, this will do is produce a 'wrong' (control behavior from a drawing?) UI for a 'sketch artist' who hasn't bothered to learn to use his IDE. Somebody still has to come along, muddling through the messes (one per target), and fix it.

Re:I RTFA

[MangoCats]

That "tedium" of coding the ui is usually the easiest part of app development, for me at least.

Does the neural net also recognize dynamic UIs with swipe, pinch and twirl responses?

Neat academic project... but 77% accuracy just to do the layouts? I've got interns better than that.

Re:I RTFA

[Fnkmaster]

Agreed. We built a nearly identical system (with OpenCV for morphological analysis and neural networks) about a year ago, as part of a larger AI-powered mobile app development platform.

77% accuracy is not very impressive, but unclear what the training and test sets are here.

The biggest functional win from this is actually getting sensible layout params from a designer's UI mockup - i.e. figuring out should this be right justified/left justified, should there be margin/padding here, etc. We solved that problem pretty well.

Other challenges involve asset up-scaling, background image color extraction, etc. If you can take rough image mockups and output well polished asset packs, with vectorized images, layout files and stub code for developers to work with, that's a pretty significant win.

We got that far with the project, but ended up shifting direction to a somewhat different market where there was more growth potential - literally nobody wanted to invest further in mobile app tools in 2016, AI powered or not.

So yeah, cool proof-of-concept, but as a standalone offering this doesn't create much value. As part of a larger toolchain might be valuable.

RAD Redux

[Tablizer]

We've had RAD systems for decades. They make the first 80% easy, but not the last 20%. One is always dealing with things like legacy databases with goofy schemas and domain-specific intricacies.

Tools that may take longer to lay down the basics but can be tuned easier for specifies still seem the best bet.

Plus you have issues of mobile devices such that UI's need to be "responsive" to different screen sizes. These can take a lot of experimentation to get right because context is involved. They are solving 1990's problems.

It just accepts an image instead of a drawing?

[dpbsmith]

Was the first version of ResEdit released in 1984 or 1985? In any case, for more than thirty years, there have been developer tools that allowed you to draw a UI screen, while simultaneously creating a WYSIWYG screen image, an object-oriented description of the elements in the image (e.g. "a checkbox at 50,100"), and code to generate the image.

As nearly as I can tell, the only novelty here is the ability to work off a static image file, rather than being able to work off the time-sequence of the series of drawing manipulations used to draw the file. This wouldn't be a big deal even if it worked, since it doesn't take very long for a human to look at a UI screen and draw a duplicate layout using a UI layout tool.

As for "77% accuracy," I have no idea what that means or how you calculate the percentage, but sounds like "it doesn't work," because the amount of work needed to correct something that is only 77% accurate is probably about the same--quite possibly more--than the amount of work needed to create it from scratch with a good layout tool.

Furthermore, it is very common for a UI layout to contain elements that are only conditionally visible. An obvious one would be a tabbed panel. A screenshot can show you the control that are in the frontmost tab page, but has no information at all that would allow pix2code to even begin to guess at the controls and other elements that are present in the other tab fields. Therefore, to get even a complete visual record of the interface, it is necessary to have some kind of procedure or script that results in every UI element being systematically revealed. That's not trivial. (Imagine some of the currently fashionable designs that save screen real estate by putting larger parts of the UI on invisible trays that only slide into view when needed).

Re:Not good

[HornWumpus]

The ones that could were/are _terrible_coders_. UI configuration is more complicated than a screenshot can capture. If they complete it from a screenshot without asking questions, fire their incompetent asses. They missed a lot, guaranteed, assumed even more...

UI Mockup is a small part of systems analysis. Comes relatively late in the process. Mostly the UI 'falls out' of the data structure or data/business layers, depending on how you 'squint' at the process.

But that's all old stuff, doesn't involve AI...doesn't reflect 'UI designers' ability to fuck the whole process up while applying a glorified skin to the app. Now get off my lawn...

Ghost of HyperCard

[swell]

A reminder of the day when you didn't need a crack team of engineers to produce a simple educational program or hyperlinked ebook. Every teacher a programmer. Every student a programmer producing value added content, whether 5 or 55 years of age.

We talk about advancements in the industry but we've taken a giant step backward in terms of creative output. See 'Inigo Gets Out' https://www.youtube.com/watch?...

Why this announcement is harmful.

[sbaker]

I see questions on Quora and similar places from kids who are thinking of taking up a career as computer programmers - on commonly asked one is "If I become a programmer, will AI make my career obsolete?" - and this is a very valid concern. If I were a truck driver, I'd be really worried that self-driving trucks would take my job 5 years from now.

This announcement (which effectively says to the layperson "Programmers are about to become obsolete") will have a chilling effect on those people who are just thinking about getting into this field.

In truth - this AI program will never see the light of day - it can NEVER "write a program from screenshots" because the necessary information to do that isn't present in the screenshots - even in principle. What HAPPENS when you push this button? All the screenshots tell you is that there is a button...and MAYBE...if the screenshot is somehow linked to other screenshots...it might tell you that pressing the button takes you to another panel. What it doesn't tell you is that pressing that button caused the camera to take a photo, for the software to reconstruct a 3D image of a person from that photo, that this has to be sent off to the server to match other 3D images, that the resulting match produces that person's name - which the program is then given from the server - and which then results in that "NAME" field on the next GUI panel to be populated with an actual name and not the "John Doe" that the GUI designer put there so the programmer would know that this is where the name goes.

By itself - this announcement can be laughed at and called bullshit by anyone who has anything to do with writing programs (and I'm 100% sure it's being laughed at right now) - but the CHILLING effect that such ridiculously over-stated claims make on those who might be considering entering the industry is a very, very bad thing.