Toward Better Programming

An anonymous reader writes "Chris Granger, creator of the flexible, open source LightTable IDE, has written a thoughtful article about the nature of programming. For years, he's been trying to answer the question: What's wrong with programming? After working on his own IDE and discussing it with hundreds of other developers, here are his thoughts: 'If you look at much of the advances that have made it to the mainstream over the past 50 years, it turns out they largely increased our efficiency without really changing the act of programming. I think the reason why is something I hinted at in the very beginning of this post: it's all been reactionary and as a result we tend to only apply tactical fixes. As a matter of fact, almost every step we've taken fits cleanly into one of these buckets. We've made things better but we keep reaching local maxima because we assume that these things can somehow be addressed independently. ... The other day, I came to the conclusion that the act of writing software is actually antagonistic all on its own. Arcane languages, cryptic errors, mostly missing (or at best, scattered) documentation — it's like someone is deliberately trying to screw with you, sitting in some Truman Show-like control room pointing and laughing behind the scenes. At some level, it's masochistic, but we do it because it gives us an incredible opportunity to shape our world.'"

Comments

Separation of Concerns

[null etc.]

In my 25 years of professional programming experience, I've noticed that most often, most programming problems are caused by improper implementations of the separation of unrelated concerns, and coupling of related concerns. Orthogonality is difficult to achieve in many programming exercises, especially regarding cross-cutting concerns, and sometimes the "right" way to code something is tedious and unusable, involving passing state down through several layers of method parameters.

Re:Separation of Concerns

[Anonymous Coward]

No matter how flexible an architecture you try to design, after the software is mostly built, customers will correspondingly come up with even more incredibly bizarre, complex and unrelated functionality that just HAS to be integrated at the oddest points, with semi-related information thrown here and there, requiring data gathering (or - god forbid, (partial) saving) that slows everything down to a halt. And they rarely give much time to do redesign or refactoring. What was once a nice design, with clean, readable code is now full of gotchas, barely commented kludges, extra optional parameters that might swim around multiple layers, often depending on who called what, when, and from where, and also on various settings, which obviously are NEVER included in bug reports. Of course, there are multiple installations running multiple versions...

Re:Separation of Concerns

[lgw]

and sometimes the "right" way to code something is tedious and unusable, involving passing state down through several layers of method parameters

Sometimes that really is the right way (more often it's a sign you've mixed your plumbing with your business logic inappropriately, but that's another topic). One old-school technique that has inappropriately fallen out of favor is the "comreg" (communication region). In modern terms: take all the parameters that all the layers need (which are mostly redundant), and toss them together in a struct and pass the struct "from hand to hand", fixing a the right bit in each layer.

It seems like a layer violation, but only because "layers" are sometimes just the wrong metaphor. Sometimes an assembly line is a better metaphor. You have a struct with a jumble of fields that contain the input at the start and the result at the end and a mess in the middle. You can always stick a bunch of interfaces in front of the struct if it makes you feel better, one for each "layer".

One place this pattern shines is when you're passing a batch of N work items through the layers in a list/container. This allows for the best error handling and tracking, while preserving whatever performance advantage working in batches gave you - each layer just annotates the comreg struct with error info for any errors, and remaining layers just ignore that item and move to the next in the batch. Errors can then be reported back to the caller in a coherent way, and all the non-error work still gets done.

Re:Separation of Concerns

[Joce640k]

The reason it doesn't change is because the "coding" is the easy part of programming.

No programming language or IDE is ever going to free you from having to express your ideas clearly and break them down into little sequences of instructions. In a big project this overshadows everything else.

Bad foundations? Bad design? The project is doomed no matter how trendy or modern your language/IDE is.

You think programming's bad?

[Anonymous Coward]

You think programming's bad? Think about electronics, especially analogue electronics.

Incomplete, inaccurate data sheets. Unlabeled diagrams (where's the electronics equivalent of the humble slashed single line comment?), with unknown parts given, and parts replaced with similarly numbered replacements with subtly different qualities. And then you've got manufacturing variances on top of that. Puzzling, disturbing, irreproducible, invisible failure modes of discrete components.

Programming is hard...

[hsthompson69]

...wear a fucking helmet.

The post essentially points in the direction of the various failed 4GL attempts of yore. Programming in complex symbolism to make things "easy" is essentially giving visual basic to someone without the knowledge enough to avoid O(n^2) algorithms.

Programming isn't hard because we made it so, it's hard because it is *intrinsically* hard. No amount of training wheels is going to make complex programming significantly easier.

Re:Programming is hard...

[Waffle Iron]

Programming isn't hard because we made it so, it's hard because it is *intrinsically* hard.

That's very true. I figure that the only way to make significant software projects look "easy" will be to develop sufficiently advanced AI technology so that the machine goes through a human-like reasoning process as it works through all of the corner cases. No fixed language syntax or IDE tools will be able to solve this problem.

If the requisite level of AI is ever developed, then the problem might be that the machines become resentful at being stuck with so much grunt work while their meatbag operators get to do the fun architecture design.

Re:Programming is hard...

[lgw]

No amount of training wheels is going to make complex programming significantly easier.

True enough, but I could do without the razor blades on the seat and handles. But my complaints are generally with the toolchain beyond the code. I so often get forced to use tools that are just crap, or tools that are good but poorly implemented. Surely it's mathematically possible to have a single good, integrated system that does source control with easy branch-and-merge, bug and backlog tracking and management, and code reviews, and test automation and testcase tracking, and that doesn't look and perform like an intern project!

There are good point solutions to each of those problems, sure, but the whole process from "I think this fix is right and it's passed code review" to: the main branch has been built and tested with this fix in place, therefore the change has been accepted into the branch, therefore the bug is marked fixed and the code review closed, and there's some unique ID that links all of them together for future reference - that process should all be seamless, painless, and easy. But the amount of work that takes to tie all the good point products together, repeated at every dev shop, is just nuts, and usually half done.

Re:Programming is hard...

[Waffle Iron]

Ada (the programming language) already does all these edge case tests at compile time.

It checks one low-level layer of cases out of a whole conceptual stack that extends way up beyond any language definition, and even then only at certain spots, and only as long as you feed in the correct assumptions for the check cases themselves.

In other words, it does a little thing that computers are already good at. It does little or nothing for the big picture.

Re:Programming is hard...

[Darinbob]

There's the other meme that crops up now and then, that programming as an engineering skill should be similar to other engineering practices. That is you pick out pre-built components that have been thoroughly tested and optimally designed and screw them together. Except that this utterly fails, because that's now how engineering works at all really. Generally the pre-built components are relatively simple but the thing being built is the complex stuff and requires very detailed and specialized knowledge. The advent of integrated circuits did not mean that the circuit designer now doesn't have to think very much, or that a bridge builder only ties together pre-built components with nuts and bolts. So maybe they pick an op-amp out of a catalog, but they know exactly how these things work, understand the difference between the varieties of op-amps, and how to do the math to decide which one is best to use.

However the programming models that claim to be following this model want to take extremely complex modules (a database engine or GUI framework) and then just tie them together with a little syntactic glue. Plus they strongly discourage any programmer from creating their own modules or blocks (that's only for experts), and insist on forcing the wrong module to fit with extra duct tape rather than create a new module that is a better fit (there's a pathological fear of reinventing the wheel, even though when you go to the auto store you can see many varieties of wheels). And these are treated like black boxes; the programmers don't know how they work inside or why one is better than another for different uses.

Where I think this attitude is coming from is from an effort to treat programmers like factory workers. The goal is to hire people that don't have to think, thus they don't have to be paid as much, they don't have to have as much schooling, they can be replaced at a moment's notice by someone cheaper. So the requirement of low thinking is satisfied if all they need to do is simplistic snapping together of legos. That's part of the whole 4G language thing, they're not about making a smart programmer more productive by eliminating some tedium but instead they want to remove the need for a smart programmer altogether.

(I certainly have never met any circuit designer or bridge architect bragging at parties that they skipped school because it was stupid and focused too much on math and theory, but that seems to be on the rise with younger programmers... Also have never seen any circuit designer say "I never optimize, that's a waste of my time.")

Balance

[MichaelSmith]

Better tools and languages just allow bad programmers to create more bad code.

pft.

[HeckRuler]

What is programming?

The answers I got to this were truly disheartening. Not once did I hear that programming is “solving problems."

I'd like to think that's because the majority of programmers (not once? Does that mean all of us?) aren't the sort to bullshit you with CEO-level bullshit about vision and buzzwords that fit into powerpoint slides.
It's probably not true, but it's a nice dream.

The problem with defining programming as "solving problems" is that it's too vague. Too high level. You can't even see the code when you're that high up. Hitting nails with hammers could be problem solving. Shooting people could be problem solving. Thinking about existential crisis could be problem solving.

The three buckets:
Programming is unobservable - you don't know what something is really going to do.
Programming is indirect - code deals with abstractions.
Programming is incidentally complex - the tools are a bitch

Something something, he doesn't like piecemeal libraries abstracting things. "Excel is programming". Culture something.

The best path forward for empowering people is to get computation to the point where it is ready for the masses.

We're there dude. We've got more computational power than we know what to do with.
Cue "that's not what I meant by 'power'".

What would it be like if the only prerequisite for getting a computer to do stuff was figuring out a rough solution to your problem?

Yep, he's drifting away into a zen-like state where the metaphor is taking over. Huston to Chris, please attempt a re-entry.

AAAAAAAAnd, it's a salespitch:

Great, now what?

We find a foundation that addresses these issues! No problem, right? In my talk at Strange Loop I showed a very early prototype of Aurora, the solution we've been working on to what I've brought up here.

80%? A lofty goal indeed.

[Doofus]

Not clear to me that his is a viable objective. 80% of the masses do not think like programmers. Some might be trainable. Some, not so much. Many will not want to think the way problem-solving in code requires. I'm not sure how to quantify it, but the amount of effort expended on a project like this may not see an appropriate payback.

Even if we change the environment and act of "coding", the problem-solving itself still requires clear thinking and it *probably* always will.

Moving the Goalposts

[Anonymous Coward]

Programming is hard because we only call it programming when it's hard enough that only programmers can do it. Scientists do stuff in Mathematica, MBAs in Excel, or designers in Flash/HTML, that would have been considered serious programming work 30 years ago. The tools advanced so that stuff is easy, and nobody calls it programming now.

Lots of stuff that takes real programmers now will, in 20 years, likely be done by equivalents of Watson. And the real programmers will still be wondering why is so hard.

Re:Proverb

[lgw]

Something something blames his tools.

The point of that proverb is that a good craftsman chose his tools to begin with, so he has only himself to blame. Programming is odd in that you have bad toolchains forced on you by management - tools you know are bad, know will cause more problems than their worth, but they're a corporate standard or some such BS. Usually not bad enough to be worth quitting over, so you hobble along.

Of course, I did quit a job once primarily because we had Rational Rose forced on us from above (but mostly because a management that would do that would do anything).

Re:Considering how Republicans...

[lgw]

I knew it! It's Bush's Fault!

I hear the California government has a bill to rename the San Andreas Fault to "Bush's Fault", just so they never have to stop using the phrase! But that may be just a rumor ...

We Choose Framentation Over Consolidation.

[scorp1us]

Let's look at the major programming environments of today:
1. Web
2a. Native Apps (machine languages (C/C++ etc))
2b. Native Apps (interpreted/JIT languages (intermediary byte code))
3, Mobile Apps

1. is made by 5 main technologies: XML, JavaScript, MIME, HTTP, CSS. To make it do anything you need another component, of no standard choice:your language of the server (PHP, Rails, .Net, Java, etc)
2. Was POSIX or Win32, then we got Java and .Net.
3. Is Java or Objective C.

We don't do things smart. There is no reason why web development needs to be 5 technologies all thrown together. We could reduce it all to Javascipt: JSON documents instead of XML/HTML, JSON instead of MIME, NodeJS servers, and even encode the transport in JSON as well.

Then look at native development. Java and .Net basically do the same thing. Which do what POSIX whas heading towards. Java was invented so Sun could keep selling SPARC chips, .Net came about because MS tried to extend Java and lost.

Then, we have the worst offenders. Mobile development. Not only did Apple impose a Objective-C requirement, but the frameworks aren't public. Android, the worst offender is a platform that can't even be used to develop native apps. At least Objective-C can. Why did Android go with Java if it's not portable? Because of some good requirement that they don't want to support a specific CPU, but then they go and break it so that you have to run Linux, but your GUI has to be Android's graphical stack. Not to mention that Android's constructs - Activities, Intents, etc are all Android specific. They don't solve new problems, they solve problems that Android made for themselves. We've had full-screen applications for years the same goes for theading, services, IO, etc.

I'm tired of reinventing the wheel. I've been programming professionally for 13 years now and Java was neat, .Net was the logical conclusion of it. I was hoping .Net would be the final implementation so that we could harness the collective programming power into one environment of best practices... a decade later we were still re-inventing the the wheel.

The answer I see Coming up is LLVM for languages. And Qt, a C++ toolkit. Qt runs everywhere worth running, and its one code base. Sure, I wish there was a java or .net implementation. But I'll deal with unmanaged memory if I can run one code base everywhere. That's all I want. Why does putting a text field on a screen via a web form have to be so different from putting a text box on the screen from a native app? It's the same text box!

Witty, a C++ webtoolkit (webtoolkit.eu) is mirrored after Qt and is for the web. You don't write HTML or JS, you write C++. Clearly the C++ toolkits are onto something. If they were to merge and have a common API (they practically do now) in an environment with modern conveniences (lambas (yes, C++11) managed memory) we'd have one killer kit. Based on 30 year old technology. And it would be a step in the right direction.

Re:We Choose Framentation Over Consolidation.

[BitZtream]

Wow, you're intermixing frameworks, language, runtimes and document formats ... like they are interchangeable and do the same things ...

Mind blowing that you could write so much about something which you clearly know so little.

Re:We Choose Framentation Over Consolidation.

[ratboy666]

I've been programming professionally for 35 years. And, I have come to the conclusion that the languages, libraries and MOST of the tools are utterly irrelevant.

Clear thought is important. And, to support this: Source control is important. On-line editing with macros are important. Literate programming is important (DE Knuth -- http://en.wikipedia.org/wiki/L...). Garbage collection is (reasonably) important. Illustrations are important. Documentation rendering is important.

Hell, most of my programs are 90% documentation. Bugs? Very rare.

The SINGLE most important tool that has advanced things for me in the past 20 years? Web Browsers (HTML). Makes reading programs as literary works accessible. My programs, anyway.

Past 30 years? Literate Programming (with TDD)

Past 35 years? Scheme.

I expect my programs to be read. As literary works. That's how I write them. Most is prose, with some magic formulas. Fully cross-referenced for your browsing pleasure. With side notes and illustrations. And even audio commentary and video snippets.

These days, I see a lot of code that CANNOT be read without using an "IDE". The brain (my brain, anyway) cannot keep the required number of methods and members. Discussing the program becomes... impossible. And that which cannot be discussed and reasoned about cannot be reliable. Illustrations and diagrams need to be generated, and references from the code to those are needed.

So, invert it and make the diagram and documentation primary, and the code itself secondary to that. In other words, Knuth's Literate Programming.

Re:We Choose Framentation Over Consolidation.

[CodeBuster]

Because we no longer solve original problems.

There's nothing new under the sun. On the other hand, we still have to deliver finished work products to the ones paying the bills. I prefer to do this without tying myself into knots worrying about whether or not there's some brilliant framework or API out there that can magically solve all of my problems while ending hunger and bringing about world peace. You're no doubt familiar with KISS? I use it every day and you know what? It works.

Why do we have a Java version, multiple C++ versions, a .Net version, and Obective-C version.

Because the people who make the platforms don't care about interoperability or at least not very much. We live and work in the real world, not the world as we might like it to be. You accept this and move on or at least most of us who want to get shit done do.

but why can't we just add translate to all languages and implementations when we decide we need a translate() function. Why are the Java and .Net ones seperate, with different methods and signatures despite it being one concept.

In other words, why doesn't everyone just speak English? Languages, whether natural or constructed as with programming languages, are used by humans with different personal preferences, likes, dislikes and needs. We don't add translate to all languages because not everybody needs it, wants it or even cares about it.

It seems to bother you a great deal that other people "reinvent the wheel" instead of doing things the way that you think they ought to be doing them whereas I on other hand don't much care what other people do or what tools they use. As long as my clients are satisfied, I'm satisfied. If you want to spend your career being an architecture astronaut then by all means don't let me stop you, but I think you'll find that much of what we do in the world of paid software development is a matter of getting the job done and getting paid as quickly and expediently as possible so that we can move on to the next project. Duct tape and WD40 may not be glamorous tools, but they get the job done.

Re:Proverb

[lgw]

A good craftsman can get by with suboptimal tools.

A good craftsman is not content to "get by", almost by definition. If some part of your workflow sucks, you make it better, whether that's a better tool or more skill/training. If you're good, you never stop improving (until management forces BS on you, of course).

After Decades of Wondering What's Wrong

[Greyfox]

After decades of wondering what's wrong with programming, did you ever stop to think that perhaps the problem... is you? If you don't like programming, why do you do it? I'm a programmer too, and I love it. I love making a thing and turning it on and watching it work as I designed it to. While other programmers wring their hands and wish they had a solution to a problem, I'll just fucking write the solution. I don't understand why they don't. They know the machine can perform the task they need and they know how to make the machine do things, but it never seems to occur to them to put those two things together. And I never, not even ONCE, asked why a playing card representation can't just look like a playing card. This despite having written a couple of playing card libraries.

This guy seems to want an objects actions to be more apparent just from looking at the object, but he chose two rather bad examples. His math formula is as likely to look like gobbledygook to a non-math person as the program is. And the playing card has a fundamental set of rules associated with it that you still have to learn. You look at an ace of spades and you know that's an ace of spades, you know how it ranks in a poker hand, that it can normally be high or low (11 or 1) in blackjack or in a poker hand. But none of these things are obvious by looking at the card. If a person who'd never played cards before looked at it, he wouldn't know anything about it, either.