True Visual Programming

eberta writes "We are still stuck with text programming for the most part. I can think of only a few truly visual programming environments like LabView and none are really mainstream for application development. Being recently unemployed and having ample spare time, I have started a pet project to work on my own version, GIPSpin (Graphical Interface Programming System). With multithreading becoming an increasing issue in software development, I'm wondering why hasn't there been more focus on visual programming. I see so much possibility of making coding easier and handling threading issues semi-automatically by allowing the user to graphically guide the auto-threading AI. Right now it seems the industry is focused on figuring out how to get just small chunks of code auto-threaded either through hardware or compiler technology, with longer term solutions like OpenMP still text based environments."

Comments

Don't reinvent the wheel ...

[bitsformoney]

... check what is/has been out there before. I.e. something called ProGraph has been mentioned on /. a couple of times before.

Just three words

[TuringTest]

Programming By Example

It's inefficient

[0x461FAB0BD7D2]

There are ways of visual programming right now, such as WYSIWYG HTML editors (Dreamweaver, Nvu, Frontpage) and Visual Basic.

However, I can't see this happening for Perl, PHP, C, Java, etc. Everyone has their own style of coding with their own ideas, many of which are abstract and cannot be effectively visualized. To make an IDE which effectively deals with all the quirks programming has would be quite a feat, but would be so bulky that text-based programming would be the most efficient.

There might be a place for visual programming in rapid application development or some simple programs/scripts like HTML pages and the like. Beyond that, I'd doubt it.

Re:It's inefficient

[drxenos]

I don't think you understand what "visual programming" is. None of the example you cite are real visual programming. Visual Basic, et. al., are not true visual programming. They are IDEs to help in building software. Visual programming requires no (or almost no) coding. It is all graphical. One example (I forget the system) involved using blackboxes. The boxes has particular functions (i.e., a signal processor). The boxes had connections for there inputs and outputs, and properties to modify their behavior. You just inserted the boxes you wanted, and connected them controlling the flow of your data. This system reminded me of designing digital logic circuits graphically. It's hard to explain, but it was actually more powerful than my description. There are also systems to generate code from UML.

Why hasn't there been more focus?

[p3d0]

Here's why... Why don't you reply to this note and show me a sample code snippet from your visual language?

Check out the Aardappel language. I think it's a very interesting, powerful language, with one major flaw: it's a visual language. This makes it so awkward to deal with that even Aardappel's own inventor broke down and created a textual equivalent language for the sample code in his PhD thesis.

Text is better than pictures for describing anything complex. We have thousands of years of experience to back this up.

Re:Why hasn't there been more focus?

[LWATCDR]

"Text is better than pictures for describing anything complex. We have thousands of years of experience to back this up."

You mean things like aircraft, spacecraft, buildings, or any number of things that you would use something like a blueprint or map to describe?
I do agree that I have yet to see a good visual programing language but to say that text is better than pictures for describing anything complex is incorrect. It really depends on what you are describing. A set of blue prints for a home beats written text hands down.

Re:Why hasn't there been more focus?

[nickos]

Surely Befunge proves that visual languages can be a usable alternative to textual ones :)

vv < <
2
^ v<
v1<?>3v4
^ ^
> >?> ?>5^
v v
v9<?>7v6
v v<
8
. > > ^
^<

Another implementation.

[saintlupus]

Here's a visual programming environment called SIVIL that one of the professors here at Canisius has been working on for a while.

http://www-cs.canisius.edu/~meyer/VP/home.html

--saint

Because...

[Anonymous Coward]

I'm wondering why hasn't there been more focus on visual programming

Because in the 1980's people realized that visual programming was a dead-end idea. Trust me, one of the products I work with lets you program "workflows" using a visio-like toolkit. It's the most unproductive thing I've ever had to use.

Seriously, researchers have beaten this one to death already. Even as far back as The Mythical Man Month it was already recognized that graphical/visual programming would not give us any improvements on efficiency.

I guess the old addage of "reinventing lisp, badly" still holds true. "Computer Science" sure seems to ignore a lot of its past research.

Re:Because...

[WetCat]

... by Anonymous Coward ... Trust me,

Why should I trust the Anonymous Coward?

Problems with 'Visual Programming'

[drspliff]

Visual Programming has always been a sort of industry goal, every product you see out there is trying to make it easier on the developer, be it UML or another modelling software, iMatix's Libero or another code generation framework.

One of the reasons I think Visual Programming won't catch on for a long time, or will take serious innovational leap is because with existing solutions the developer looses too much control over the path of execution, optimizations, memory management and all the other lower-level stuff we developers have to tinker with.

There have been numerous frameworks which have tried to bridge the gap, one that sticks in my mind is SilverStream's/Novell's exteNd Composer/Director. They follow the basic roles of a point-and-click programming environment, flow layout, assisted statement creation etc.

But there is only so much you can do before you end up just writing solid code again. I don't want to sound like an elite-ist, but personally I think all these high-level visual programming environments will lead to 'Joe Blogs' developing your [name critical financial or business application here].

And not to mention the thousands of zealots out there who you'll have to bring kicking and screaming into the new 'visual' era.

Rant or rave, new developments in this area can be a great aid to experienced developers, but in the wrong hands they can cause more harm than good (Visual Basic anybody?)

Moderate: -5 Zealot bait

Re:Problems with 'Visual Programming'

[kentquirk]

Back in '93 or so I was the UI designer on a visual programming version of VB that Lotus developed (and later sold to Revelation software). We had "links" between UI elements and code snippets so that you could build basic VB apps without actually breaking out the text editor. It worked pretty well as an accelerator for basic application development, but eventually you ended up writing text-based code. It just saved you time getting started.

I liked the idea so much I went out and built a game around it -- MindRover (http://www.mindrover.com/) where players constructed robots using purely visual programming techniques (no text-based code at all). It worked, and it was great for a game, but it made for a cumbersome development process. It's hard to look at an image and see everything you need to see.

Visual programming is cool and fun. It makes the learning curve for programming a little flatter, and it can be a nice way to prototype stuff and get the early versions of an application running. But using it to build apps that are more than toys is a much bigger challenge, and I haven't seen anyone yet who has done it effectively. The original poster's research looks an awful lot like a demo I saw at MIT Media Lab about 5 years ago. IMO, it exposes a bit too much of the syntax of conditionals and things in the visual UI. It's not far enough from textual programming to be helpful.

We think in Language

[sepluv]

Humans think internally about the world outside using language (hence multi-lingual people being faster thinkers as they have more paths to persue at once). Ask a psychologist.

We do not think using a poor graphical representation of the Real World. Given this, text is the best way of representing stuff on a computer (except were graphics are explictally necessary).

Now, when we get realistic VR systems that actually feel like RL, this may be a different matter (although source code would have to be represented as text at some level as both computers and humans think one-dimensionally with strings of text or numbers).

At the moment all we have is slow 2D graphics on flickery, bright, flat screens. We have far too go.

Re:We think in Language

[Bastian]

I agree for the most part, but I would revise it to recognize that we do think about a lot of stuff visually (being able to do so is absolutely necessary for tool use, and I know I understand things better when I have pictures to work with.)

However, the language is essential. It has been our primary mode of communication since time began, and trying to get too far away from it in any situation where you have to explain or describe complex ideas seems like folly to me. I think there is probably a reason why, for the most part, written languages such as ancient Mayan and Chinese have not been nearly as popular around the world as phonetic systems of writing. Personally, I think that the reason is that collections of very abstract things like sounds (which words describe) and a few shorthand glyphs (={}*-) are much closer to the thought-stuff that floats around in my brain than collections of concrete pictures.

Why hasn't there been more focus on VP?!

[uradu]

Because, while on the surface it looks like a really neat thing, in the real world productivity and usability quickly take a nose dive. Visual tools are good for smaller graphs where you can keep most of the view onscreen at once (e.g. filters or rules like in email apps, or workflow graphs), but once the graph grows beyond a certain size it quickly becomes unmanagable. Besides, what's quicker, typing "for (int i = 0; i max; i++)" or dragging an "if" element from a toolbar and dropping it on an empty area of the form, activating and filling out its fields, and connecting it to the rest of the program flow?

I've been forced to use them for doing IVR

[OhPlz]

I used to write IVR applications, interactive voice response. They're the automated phone systems that everyone loves to hate.

While I had been doing it for ten years it never failed that when there was a regime change within the company some new high level manager would read an add in some random telephony journal and think he could magically reduce our IVR development time ten times over.

Yep, everytime it was one of those graphical languages. Look! Just connect the voice prompts to touch tone input nodes then uhm, magic.. and we're done!

At least in the products we were forced to try it never worked out that way. Any type of call flow the GUI developers didn't have in mind would be like pulling teeth. You'd have to work around the narrow vision they had. That defeats the purpose of it being simple or rapid.

Worse yet, there was no way to optimize anything. On an 800 line dead-head time costs money. If I'm trying to ask someone for a PIN so I can start billing them for the call I need to shave off every fraction of a second I can. After hundreds of thousands of calls those fractions add up! But since you have no low level control of anything you're completely at the mercy of the developers that wrote the language.

Then there's code re-use. I hadn't seen a package that let you easily create libraries. If a problem was found with a frequently used "drawing of code" you couldn't use any automated tools to find or fix it, you have to open each by hand and look. I hate things you can't grep on.

Anyhoo, UI editors are about as visual as I prefer to get. It's an interesting project and for some applications it might make sense. I haven't found one yet.

I like it!

[WetCat]

Also for control process applications an interesting thing is
http://babelfish.altavista.com/babelfish/trurl_pag econtent?lp=ru_en&url=http%3A%2F%2Fwww.ipu.rssi.ru %2FLABS%2FLAB49%2Flab49rad.html
made by Russian Institute or Control Sciences.

Re:I like it!

[Jerf]

Keep going, and don't let the nattering neybobs of negativism here at /. get you down.

You know, I understand what you're getting at here, but you need to understand where the "nattering naybobs" are coming from. We're not saying "We've never seen this before and we're sure it won't work, don't try, fingers stuck in my ear, blah blah blah!" We're saying "We've seen several implementations of this idea, they all suck in the same fundamental way, and we've never seen anybody with a functional solution to the suckage problem."

You want to try your hand at it, be my guest. If you have taken the time to understand history, what has failed in the past, the problems encountered by others, and you still want to try, well, go for it. It's your time.

But if you think, and I get this vibe from the original poster, "Hey, there's nobody who's tried this great idea, I think I'll start a project on it!"... you're fucked, plain and simple, and throwing a bit of water on that ethusiasm is a favor, right up there with "Your first programming project should not be a real-time strategy RPG." Encouragement is great and all, but you should not encourage someone who's just been mountaineering for a week to take on Everest. Discouragement can be a positive thing.

At the very least, educate yourself before starting. There's a lot of failure in this field.

Two of my "general principles" come into play here:

1. "When a lot of other very smart people have tried to solve the problem, but you think you have the answer, you almost certainly don't, and you certainly don't if it seems 'simple' or 'obvious' to you." This fits that in spades; a lot of very smart people have tried this and failed.
2. "There exists a lot of solutions that only work in demos, or worse yet, when you just 'imagine' it in your head, but crash and burn when faced with the real world. Anything works in a demo, or in your head, even the absolutely impossible. If you've never seen it outside of a demo, there's a reason." And that fits this whole "graphical programming" to a T; it's trivial to create an "ooo, ahh" demo with the execution flying around the screen, two or three 'if's and a "for". But there aren't very many functions, let alone programs, that work with two 'if's and a for loop. (Another thing that fits here is many interesting AI solutions in academia that are claimed to be generally powerful, with varying justification, but in five years of working with the solution they've only managed one very particular demo.)

Go for the gusto, but if you just close your eyes and hum when people try to explain why it's a hard problem, you're dooming yourself from the getgo and wasting your time.

INPUT DEVICES

[ericandrade]

Computer interfaces suck.

Programming languages are visual and depend on the computer screen or whatever the computer outputs. Changing the way it is disposed on the screen is just trying to make it cute.

The major restraint right now on programming is INPUT.

The mouse dates back 50 years,
the keyboard's even older than that, and it's designed to slow down users! (both cause +RSI and other crap, and are very slow and innacurate)

For example, where are dual cursors?
There needs to be more OS implementations and design to have superinterfaces. Make the long, tedious, well planned out programming that will accelerate future programming. And think out of the box: Future programming will not be done with a keyboard and a mouse.

http://sloan.stanford.edu/MouseSite/1968Demo.htm l

Why can Stephen Hawkings write speeches, scientific texts and do tons of complex things with a single thumb clic?

Where are the standards for new interface developpement?
The OS developpements to support the hardware?

Screens are getting bigger, why do I have to rely on a menu in the top left of the screen if i'm working in the bottom right of a 2nd screen?

Design and manufacture of new technology is slowed down by the limited ways we have to transfer, collect, manage and create the complex data we have to deal with. F*ck Moore's law! With the keyboard and mouse, the bottleneck is between us and it. The idle time prooves it.

Computer's are an instrument. Instruments are allways hard to master. They evolve. They're not supposed to only get cuter.

Imagine a computer using 50% of it's processing power to know what you want /to say/it to do... /rant

Re:INPUT DEVICES

[pavon]

The mouse dates back 50 years
The steering wheel dates back 1000s of years, yet we still use it because it is an effective interface.

the keyboard's even older than that, and it's designed to slow down users!
No it wasn't - that's a myth. Besides the best text entry devices which are designed to be as efficient as possible, such as Dvorak (use it myself) and chording, are only incrementally better than QWERTY - not the revolutionary jump you are looking for.

For example, where are dual cursors?
Dead on the research table where they belong. They never provided any significant improvement in performance, and even slowed users down due to having to stop and think about what they were doing - even after spending a significant time learning the system.

Why can Stephen Hawkings write speeches, scientific texts and do tons of complex things with a single thumb clic?
You can too - on a cell phone. They are smart ways to make the most of limited input capabilities; however, they are incredibly slow compared to the keyboard and mouse, and no one would choose to use one unless it was the only option available.

With the keyboard and mouse, the bottleneck is between us and it. The idle time prooves it.
Yet, much of the idle time is due to user thinking, not slow user input - you don't notice yourself being slow, because you are preoccupied, but you do notice when the computer slows you down even slightly. Honestly, ever since I learned to to touch-type, I have been able to type as fast as I can think, and there isn't really any reason for me to want to input text faster. Once I get a wacom tablet, I'll likely be able to say the same about the 2d/3d graphics work (err play) that I do.

Most of the instances where I notice the computer slowing me down - making me do more work then I ought to - is due to poor design and integration of the software, not the keyboard and mouse. For example, I had to start up and entirely separate program just to spell-check this post.

Where are the standards for new interface developpement?
I don't know - why haven't you come up with one?

Re:INPUT DEVICES

[RevAaron]

Squeak Smalltalk can do multiple cursors. Someone would have to do a bit of simple hackery for Windows/Linux to take two PS/2/usb mice and feed it to Squeak, or also there are some easier (but more expensive) ways to do it with two squeak processes. The usual use of two (or more) cursors in Squeak is for the remote access/collaboration software built in- you can have 5 people connected remotely, each with their own cursor with its own focus, as well as the person who is sat down in front of the computer. No problem, it's all there.

It may be a place to start playing if you're interested in that sort of thing.

But we *do* think visually

[TuringTest]

Humans think of the world using language, but we also think of the world using visual, spatial, temporal, sensorial... reasoning.

Do ask a real psychologist, she will say that there are different kinds of think. Textual is best suited for abstract, logical reasoning. But associative thinking is often better done visually. In the Programmer's Guide to the Mind you have an interesting classification of all these.

A programming environment should take care of all these kinds of thought, not just support the logic abstractions as they do now. A promising field of research is Programming By Example. This programming style tries to build the final program by using concrete reasoning over samples of data, instead of forcing you to think of the general, abstract procedure.

I like it!

[ka9dgx]

It fits in with some of my own views of the world, with the concept of Rich Source, etc. As long is this is a two-way tool, and can be used to offer another view of the source, as opposed to being only graphical, you've got a great tool idea going here.

Keep going, and don't let the nattering neybobs of negativism here at /. get you down.

--Mike--

Graphical Programming is Inefficient

[ActionAL]

Believe me we use an old Honeywell system at work called MeasureX which uses graphical programming. EVERYTHING IS BLOCKS AND LINES CONNECTING EACH OTHER. It is the most inefficent way to read code possible. The human brain can read text code insanely faster than trying to decipher a huge picture with tons of blocks and lines strewn everywhere with complex connections and pipes from page to page. It is the most horrible thing I have ever seen. And if you have one missing connection or pipe, goodness good luck trying to debug it.

Unreal 3 engine uses visual programming

[Junks Jerzey]

The text-based UnrealScript has gone away, to be replaced by a fully-visual language, Kismet.

Re:Unreal 3 engine uses visual programming

[Anm]

While UnrealEd3 does visualize much of the content creation process, including materials/shaders, physics constraints, particle systems, AI waypoints/hints, etc. It doesn't not visualize the underlying logic. As a developing working on two Unreal projects, I'm assuming I would have heard this from Epic.

For more on UnrealEd3 (and the engine itself) look at:
http://www.unrealtechnology.com/html/technology/ue 30.shtml

Anm

Visual Languages are Lacking

[fenris_23]

A programming language is a language used to communicate both to a machine and to other humans. Language features help us encapsulate, hide, and organize complexities so that communicating very complex ideas to a human or machine is more efficient and maintainable.

Non-written languages do not provide the same depth and strength. For example, a CD of James Joyce's Ulysses is not as accessible and understandable as the same book in written form.
Furthermore, how would you express those concepts visually? In my opinion, we developed our forms of written communication over the years because it is the most efficient and expressive.

Take hieroglyphics for example. Everybody knows that the Egyptians used a written language of symbols referring to entire concepts rather than words. However, many people do not know that in every day practice, the Egyptians developed a linear form of the same language. Similarly, Asian cultures have adapted their languages to a linear form to use with computers because it is easier than adapting a computer to work with more complex symbols.

Also consider that amount of complexity that can be expressed in a written (text) programming language. When you begin thinking about designing a visual language, you begin thinking about logic flow and control structures. However, you should begin at the most basic level. A programming language's lexicon has both closed and open classes. The keywords are closed but the open class of identifiers is infinite. Furthermore, the idioms used to express these identifiers in various statements are also practically infinite with respect to designing a visual language. Statements can be combined into idioms that vary between languages, programmers, development teams, and application domains.

Worse, is the problem of side-effects. Many programmers using languages such as C and C++ use side effects all of the time. How do you adequately express that in a visual language?

UML is a visual language that has seen a massive amount of research and development. Much progress has been made but even the most die-hard UML designer still has to go down to the code-level to fix the various idioms they wish to express in the programming language that the UML cannot express.

Ultimately, I think the biggest problem is the lexical and syntactic constraints. A programming language allows one to easily expand the lexicon of a programming language as well as various syntactical forms. To do this visually, you will have to create a symbol set to handle each form. If you tried to implement this dynamically like how a written language works, then you are really just developing a written language that uses pictures for words. In that case, you are wasting your time and may as well stick with text.

Different tools for different purposes

[booch]

I think the parent poster has a lot of important points. But one thing I haven't seen mentioned in this entire thread is that each type of programming/thinking/communicating has its place. While visual layout of complex program logic is not a good way to program, a written procedural language is probably not the best way to create a GUI. We should be using descriptive languages (whether visual or text) for GUIs and such.

For example, a typical how-to book is mostly text, but it also includes diagrams of how to put things together, and pictures of the completed project. All of these are essential to the book. Another good example is teaching. Teachers employ multiple methods of teaching, as different people learn differently, and different topics lend themselves to different styles. Some people learn best by doing; others by watching; and others by reading.

I think the right thing to do regarding visual programming is to relegate it to where it works best. I suspect that this would be mostly related to things that are already visual -- i.e. graphical (GUI) layouts, etc. I think it's also useful to be able to visually represent program code "in the large" in a graphical manner, to see the inter-relations, but I don't think manipulating that type of graph is useful.

I remember the old NeXT Interface Builder. You would take components, and connect them together. You'd have to write code if you were doing anything complex, but you could set up the GUI, including callback functions, visually. I think it's still around in Mac OS X. Things like GLADE work similarly. I'd like to see these types of tools used more.

Just choose the right visual representation

[c0d3h4x0r]

Visual representations work better than textual representations for most technical things, but only if you choose the right visual representation.

Example of a good visual representation: music composition software with virtual modules/machines/synthesizer that you graphically plug together into a virtual "rack" of equipment for your song. Software like BuzzTracker or Psycle, which take this visual approach, are far more efficient to work in than the old textual interfaces provided by programs like ScreamTracker3 or Impulse Tracker.

Example of a pretty good, but still imperfect, visual representation: the desktop GUI. From within the GUI, you can accomplish just about everything you could possibly ever need to do with your computer. It's almost never necessary to pull up a command prompt to get something done, because the GUI provides an equivalent, more-understandable, typically more-efficient way to do it. Of course, there are still cases where the GUI provides a shitty, inefficient visual representation (or worse yet, no visual representation at all) and you do still have to resort to the command line to get something done, or to do it more efficiently. This just illustrates how choosing the right/optimal visual representation is the real challenge, and it also illustrates how it's an ongoing work-in-progress to pick the "optimal" visual representation.

Example of a bad visual representation: most visual programming models developed thus far. As others have pointed out, most visual programming models put together so far are too high-level to be realistic programming environments for real-world purposes. This doesn't mean that visual programming will never work. It just means that no one has offered up a decent enough visual representation of programming yet.

Another thing worth noting is that when you try to develop a high-level "wrapper" layer which rides on top of a lower-level "intermediate" layer, which in turn rides on top of a lowest-level "base" layer, that the layering prevents the top layer from being as elegant and usable as it otherwise could/should be.

Classic examples of this phenomennon:

• GNOME/KDE/front-ends (wrapper layer) on top of Linux CLI tools/textual config files/X11/subsystems (intermediate layer) on top of the OS kernel (base layer), instead of integrating the desktop GUI model in a standardized way throughout the entire system
• Visual programming model which attempts to translate pictures (wrapper layer) into equivalent common C/C++/Java/C# programming ideas (intermediate layer) and then into machine code (base layer)

In other words, layering forces higher layers to have to be designed to accomodate the design of the layers underneath it, which goes directly against the idea of designing the user-facing (top-most) layer for optimal usability and human understanding.

I think one of the biggest reasons visual programming has not really succeeded so far is because all the approaches to it have been attempts to "visualize" existing programming models as set forth by C/C++/Java/C#/Basic-type languages. That won't work because those programming models were never designed to be visual in the first place. This approach forces the top-most layer (the visual stuff) to be designed in a way that accomodates the intermediate layer, rather than permitting it to be designed in the most human-intuitive way.

Instead of trying to create a visual representation of those existing programming models, the right approach (whatever it is) will ultimately prove to involve an entirely new programming model constructed specifically for it, rather than reusing all the same constructs and ideas established by existing textual languages.