Slashdot Mirror
Startup Uses AI To Create Programs From Simple Screenshots (siliconangle.com)
An anonymous reader shares an article: A new neural network being built by a Danish startup called UIzard Technologies IVS has created an application that can transform raw designs of graphical user interfaces into actual source code that can be used to build them. Company founder Tony Beltramelli has just published a research paper that reveals how it has achieved that. It uses cutting-edge machine learning technologies to create a neural network that can generate code automatically when it's fed with screenshots of a GUI. The Pix2Code model actually outperforms many human coders because it can create code for three separate platforms, including Android, iOS and "web-based technologies," whereas many programmers are only able to do so for one platform. Pix2Code can create GUIs from screenshots with an accuracy of 77 percent, but that will improve as the algorithm learns more, the founder said.
I'm pretty sure code generators have been able to accept input from graphical layout editors for a while. Just what is this AI "inferring" ?
Nullius in verba
And just like that, yet another field obliterated by AI. One wonders why anyone bothers going to school these days? At this rate, we'll have AGI soon enough and all the old models go flying out the window at the speed of light.
This just shows how easy it is to app an app that apps other apps, unlike LUDDITE software!
Apps!
It only generates the layout files for the different platforms.
Since when is html/css a program??
77% is not good. I would not say it's outperforming any coders.
"First they came for the slanderers and i said nothing."
We've had RAD systems for decades. They make the first 80% easy, but not the last 20%. One is always dealing with things like legacy databases with goofy schemas and domain-specific intricacies.
Tools that may take longer to lay down the basics but can be tuned easier for specifies still seem the best bet.
Plus you have issues of mobile devices such that UI's need to be "responsive" to different screen sizes. These can take a lot of experimentation to get right because context is involved. They are solving 1990's problems.
Table-ized A.I.
https://xkcd.com/1838/
Was the first version of ResEdit released in 1984 or 1985? In any case, for more than thirty years, there have been developer tools that allowed you to draw a UI screen, while simultaneously creating a WYSIWYG screen image, an object-oriented description of the elements in the image (e.g. "a checkbox at 50,100"), and code to generate the image.
As nearly as I can tell, the only novelty here is the ability to work off a static image file, rather than being able to work off the time-sequence of the series of drawing manipulations used to draw the file. This wouldn't be a big deal even if it worked, since it doesn't take very long for a human to look at a UI screen and draw a duplicate layout using a UI layout tool.
As for "77% accuracy," I have no idea what that means or how you calculate the percentage, but sounds like "it doesn't work," because the amount of work needed to correct something that is only 77% accurate is probably about the same--quite possibly more--than the amount of work needed to create it from scratch with a good layout tool.
Furthermore, it is very common for a UI layout to contain elements that are only conditionally visible. An obvious one would be a tabbed panel. A screenshot can show you the control that are in the frontmost tab page, but has no information at all that would allow pix2code to even begin to guess at the controls and other elements that are present in the other tab fields. Therefore, to get even a complete visual record of the interface, it is necessary to have some kind of procedure or script that results in every UI element being systematically revealed. That's not trivial. (Imagine some of the currently fashionable designs that save screen real estate by putting larger parts of the UI on invisible trays that only slide into view when needed).
"How to Do Nothing," kids activities, back in print!
An accuracy of 77% is consider a passing C grade (unfortunately) in school and is completely unacceptable in the real world business environment. It needs to be getting an A+ to be acceptable in the real world. Fire the sucker.
Sigh. Here we go: that isn't 'AI'. Can we please start calling it something more accurate and honest? Can millennials please step outside of their teenaged fantasies and join the rest of us here in reality? It would be so much easier. Double sigh.
Computer languages are almost all variations of English,(While, If, then, else, go, return, are all english words).
But human derived language, particularly alphabet based ones, s are not appropriate for coding.
Alphabet based languages are designed to represent an infinite set of words. Computer languages use a small set - often less than 100.
excitingthingstodo.blogspot.com
coders! welcome to the chopping block lol...
Um. Say what?
Any set of symbols is an "alphabet" in computer science parlance. Furthermore, although computer languages have a limited number of keywords, they also have variable names, etc.
Meh. I think parent is satire. Too subtle a WTF to be real.
Alphabet based languages are "Dart" and "Go" right?
Welcome to Simulink. We've been using it for a while to solve quite a few complex tasks.
You are familiar with the Historical Documents, are you not? https://youtu.be/5wXDnJt3cUI?t...
This has been a test. If this had been an actual Sig, you would have been amused.
I don't know what languages you're using, but every one I use allows you to define additional words to do other things. They're called functions, procedures, subroutines macros, whatever. I don't know any computer language that uses a small set of words. Even assembler allows you to use macros that you can call all sorts of names. And then there's labels for jumps in both low and high-level languages.
Since you used english as an example, there are far more valid words in programming languages than in the english language. $WHAT_THE_FCUK, _this_is_not_a_valid_english_word_, gfx10(), none of these are english words. Even assembler instructions aren't english. JMP, LEA, SHR, XOR, etc.
"Transparent" is a shit show that trades on every stereotype going. A man in drag is NOT a transsexual.
The web page is based on old school HTML Java Script + some JQuery.
Judging by the video, it looks like they have created a machine learning tool that can recognize text and buttons.
And from that recreate the exact same, in this case bootstrap UI.
So you can create a mockup in lets say Axure or Indigo, then save it as a PNG file in order to interpret and recreate it.
Doesn't seem any special by todays technologies.
This is a UI generator that might get handy when it understands photos taken from a white board full of crazy doodles
But to create a program you also need to care about the business logic.
Try to dig out all kinds of domain specific junk from someones passive aggressive head just by looking at a picture.
That can be difficult.
Such details will be out of reach from most AI in the near future (+10 to 20 years).
The devil is in the details as they say.
So are we all going to end up as small back end devils then? when the all mighty AI have taken over the UI
I wrote OCR software years ago just for fun. Writing algorithms that could identify boundaries each typeset character from a scanned magazine page as well as images was pretty hard to do... but I did it... with 98% accuracy... and no real AI... just algorithms like SPF.
What this guy did barely counts. It just identifies rectangles (poorly) and guesses what kind of rectangle (poorly) and spits out code.
It's a cute project for a high school kid.
I'll give thing a screenshot of my program: a big button that says "Fix my problem," or maybe "Enhance". With that I guess I did the hard part, so now I'll just kick back and wait for the magic AI to generate the code to make it work..
What happens if you take the screenshot of the neural network program and feed it to itself, will it show its source code in the output ?
A reminder of the day when you didn't need a crack team of engineers to produce a simple educational program or hyperlinked ebook. Every teacher a programmer. Every student a programmer producing value added content, whether 5 or 55 years of age.
We talk about advancements in the industry but we've taken a giant step backward in terms of creative output. See 'Inigo Gets Out' https://www.youtube.com/watch?...
...omphaloskepsis often...
Challenge accepted - see you at the coffee machine - remember your cup!
Requiem for the American Dream
As ANYONE who writes programs with GUI's will tell you - a system like this can AT BEST write 10% of the code needed in an actual "Program"...because the GUI layer isn't that tough to write - and there is no way on god's green earth that something that looks at screenshots can infer how the other 90% of the code has to work. Add to this that if it's even 99% accurate - how will a human programmer fix the remaining 1% of the bugs?
So - programmers everywhere are laughing at this claim...it's patently obvious that it can't do that.
It's over-sold. No, it isn't using "AI to create programs from simple screenshots" - that's a flat out lie.
Now - if instead, we were told that this could "create GUI layouts from pencil sketches" - then I would have been impressed...if it could do it from already designed-inside-the-computer wireframes - then I'd be less impressed...and with only 77% accuracy - I'm thinking...yeah...um - and if the code it generates isn't beautifully commented with good variable names that reflect the functions of the individual widgets and decorative elements - then it's worse than useless.
This is a very premature press release and all it's done is to make everyone who understands the problem even a tiny bit - call "bullshit" - which destroys the reputation of those involved and makes it much harder to get people interested even if it does eventually become useful.
So - come back when you get at least 99% accuracy. Be honest about what this does (No, it doesn't "write programs") and *CRUCIALLY* explain how a regular programmer will fix the 1% (or currently 23%) of the errors it makes.
Human written code will have useful names for each widget and other graphic on the screen, so if that widget isn't doing what it should - we can fix it easily. If this thing generates spaghetti code with names (at best) guessed from the icons on the screenshot - then if it's ANYTHING short of 100% perfect every single time - then the effort to debug the code it generates will utterly erase any benefits it provides.
Furthermore - if I do use it to generate the GUI code with 99% perfection - then fix the 1% that isn't - and then the design team want to add another button - or move something - the AI will go off, do it's thing and make a new 99% perfect GUI code...will I then have to go in and fix a different set of 1% screwups? Because if I do, then this AI will not only be somewhat painful - it won't help the coding of GUI's at all, it'll actually cost programmers a LOT more time to fix any bugs it produces.
www.sjbaker.org
I see questions on Quora and similar places from kids who are thinking of taking up a career as computer programmers - on commonly asked one is "If I become a programmer, will AI make my career obsolete?" - and this is a very valid concern. If I were a truck driver, I'd be really worried that self-driving trucks would take my job 5 years from now.
This announcement (which effectively says to the layperson "Programmers are about to become obsolete") will have a chilling effect on those people who are just thinking about getting into this field.
In truth - this AI program will never see the light of day - it can NEVER "write a program from screenshots" because the necessary information to do that isn't present in the screenshots - even in principle. What HAPPENS when you push this button? All the screenshots tell you is that there is a button...and MAYBE...if the screenshot is somehow linked to other screenshots...it might tell you that pressing the button takes you to another panel. What it doesn't tell you is that pressing that button caused the camera to take a photo, for the software to reconstruct a 3D image of a person from that photo, that this has to be sent off to the server to match other 3D images, that the resulting match produces that person's name - which the program is then given from the server - and which then results in that "NAME" field on the next GUI panel to be populated with an actual name and not the "John Doe" that the GUI designer put there so the programmer would know that this is where the name goes.
By itself - this announcement can be laughed at and called bullshit by anyone who has anything to do with writing programs (and I'm 100% sure it's being laughed at right now) - but the CHILLING effect that such ridiculously over-stated claims make on those who might be considering entering the industry is a very, very bad thing.
www.sjbaker.org
Well played :)
We need more RPGs!
Startup Uses Bullshit To Defraud Investors
Looks like a fun project, but ... the three platforms they mention have somewhat different UX guidelines; it's a lot 'deeper' problem to translate a gui into something idiomatically appropriate for the mentioned platforms.
Not to say this isn't a good, interesting start on that very thing.
FWIW BASIC before 1983ish was like that. Subroutines were defined as the line numbers of the first statements in them (ie GOSUB 1000, not GOSUB DrawBox(x, y, 16, 24))
Not that I'm disagreeing with you in spirit, just technically you're wrong, which is the best kind of wrong, or something ;-) You youngsters have it so good with your procedures and functions and classes and, uh, methods, and properties and... back in my day we had line numbers and variables with one or two letters maybe followed by a dollar or percent and we liked it no we didn't it sucked.
You are not alone. This is not normal. None of this is normal.
Well, except that there were other languages before basic that didn't use line numbers. That moldy golden oldie, assembler, jumped to offsets (usually defined as a symbolic name), not line numbers. But if you want to call me a youngster, at my age I'll take it as a compliment :-p
BTW - you could use variables longer than 2 letters, just that only the first two were significant, so COUNT and COW were the same. As long as you kept that limitation in mind, it made for more readable code. And if you didn't, well, have fun debugging ... But at least it made code more readable.
"Transparent" is a shit show that trades on every stereotype going. A man in drag is NOT a transsexual.
This will be great for automating the process of generating fake landing pages for phishing attacks. Think phishing worms.
Depended on the version of BASIC, I know most Microsoft versions did allow you to say "APPLE=2: PRINT APLPE" (sic) and it'd print 2, but there were quite a few that didn't and rigorously enforced the 1 or 2 letter limit.
OTOH, the BASIC I used on the ZX81 (my first home computer!) did allow you unlimited variable names. And I guess as GO SUB (sic) accepted formulas in Sinclair BASICs (alas not Microsoft's) you could have used named subroutines (ie 10 LET DRAWBOX=1000 / 20 GO SUB DRAWBOX) so in that respect...
You are not alone. This is not normal. None of this is normal.