Slashdot Mirror
Programmers and the "Big Picture"?
"Back working on my undergrad (computer engineering) I remember getting frustrated at the comp-sci profs that insisted machines were simply 'black boxes' and the underlying hardware need not be a concern of the programmer.
Of course in embedded systems that's not the case. When developing code for a medical device, you've got to understand how the hardware responds to a software crash, etc.
A number of Slashdot readers dogmatically responded with "security through obscurity" quotes about the shuttle's missing secret box. While that may have some validity, it does not respect the needs of the entire system, in this case the difficulty of maintaining keys and equipment across a huge network of military equipment, personnel, installations."
I like to use jpeg compression. PNG is great for line-drawings, etc though. If you really need the whole big picture, and size isn't an issue, then yes, PNG's 24-bit mode is pretty good.
it's been time for a paradigm shift for years now.
Karma: Bad (mostly affected by being such an asshole)
I don't have as much experience as some, but I've always wondered about coders who restrain themselves in the 'world' their code runs in. It overlaps, I think, with the problems of sysadmins who leave systems/gateways/firewalls and whatnot wide open to the world.
If a coder isn't ignoring the fact their code isn't going to be running on the exact same shell as they are, they're ignoring that it won't always be running in the exact same OS, or exact same network. Tragically, when it breaks it can then break BIG.
Note I also don't have enough experience to offer a solution other than "get a clue!". It's more work until you embed it in your habits to take notice of these possibilities.
Most programmers who are going to come across a "black box" have enough experience to be able code for the situation. Isn't that skill a trait of a good programmer?
:)
Then again maybe Im missing the point
If you get an error, type "OVERRIDE" or "SECURITY OVERRIDE" and then try the optimize command again.
Yes. And no.
(Ask a uselessly-general question, get a uselessly-general answer.)
I think perhaps the question was meant to focus on how big picture programming (not images) might have helped or hindered the abilities of the investigators in determining the cause. On the other hand, images of the destruction might come in handy for investigative purpoases as well. Either way, and regardless the intent of the original question, what might systems engineers improve upon or how could they improve upon current paradigms to assist in investigative methods without using "black box" technology approaches?
Rivendahl
... there is nothing that has not already been thought
I think the problem increases as programmers are less and less a part of the complete systems development life cycle and are contracted to work on individual components of an overall system. Especially during the maintenance phases of a system's life, the inexperience of new programmers on a project is probably more to blame than 'training' per say to think in a black-box mentality.
Be very, very careful what you put into that head, because you will never, ever get it out. -Thomas Cardinal Wolsey
I don't know what you're trying to say here man, but no amount of programming or "Fatal Error: Wing no longer attached to craft" terminal prompts would've saved them from what happened.
If you're trying to make a case for programming paradigm shifts based on security procedures, it isn't working in this context.
Hades, PoD: Official Advocate
I am taking courses toward my degree and I must say that in my intro to programming course, the instructor was constantly stressing the need for 'black box' programming. In addition though, he also stressed that while keeping things black box, you also need to keep your mind on the whole project, always watching out for possibly security problems, etc.
I believe that some people tend to get tunnel vision and concentrate wholely on the bb theory, without taking into consideration the whole program. This does usually lead to problems and errors in the code.
Regards,
jlk
Ignore Linux. It is going away, after all, IBM has just announced that they're dropping it on itanic: http://www.infoworld.com/article/03/02/10/HNitaniu m_1.html
Many times, management is the cause of preventing developers to see the "big picture". Sometimes it's "Here, code this" and you don't get a lot of opportunity to ask the questions you know need to be asked. Sometimes you have to hope resolutions to these types of issues are built into the requirements specification or will be ironed out in quality assurance measures.
The developer is only one in a group of responsible parties in any given system, and his/her output depends largely in input from others. If a developer is kept "out of the loop" on things (or is lazy and stays out of the loop opn purpose), you're going to see these problems.
Often it's like blaming clogged fuel injectors _for_ cheap gasoline instead of _on_ it.
Keeping the "big picture" in mind is a good thing for managers and designers. For people implementing the finer details, though, it can be a distraction and a poor use of their time. Someone implementing or verifying flow control in a ground-to-space network link does not need to know much about the format of data carried over the link. Someone doing circuit layout or design for a cockpit control widget does not need to worry about reentry dynamics and airflow. Similar examples can be found in any large system design.
It is the responsibility of the higher level designers and managers to encapsulate the big picture into digestible, approachable chunks. To the extent possible, they should be aware of and codify the assumptions and requirements that cross multiple domains -- when those are documented, it is easier to test the system for correctness or robustness, as well as to diagnose problems.
When everyone on the project tries to orient their work to what they each perceive as the big picture, you end up with enough different perceptions that people work against each other. Breaking down the system into smaller, more defined, chunks combats that tendency.
People tend to focus exclusively on their area of expertise.
:D
Otherwise they become managers
Sent from your iPad.
If i write a component that takes in X1 and outputs X2, isn't it the designer's job to make it look pretty? I mean, supposedly they were the ones that came up with needing the component in the first place, to accomplish some function or other, and thus make the user happy.
stuff |
Now, the amount of abstraction possible does differ depending on what you're doing. Embedded systems programming is hard, and you do have to know details of the machine. But I ask you - do you insist on a gate-level understanding of the embedded CPU, or will you settle for knowing the opcodes and their timing characteristics?
Because, in embedded programming, you need to know more about the device, it's proportionately harder to do. That's one reason, apart from power and cost considerations, that embedded systems tend to be simple - the simpler the system, the easier it is to think about, to prove correctness or to at least enumerate possible pathways and handle them.
But even in that case, you need to be able to ignore some implementation issues or you can't do it at all.
PHEM - party like it's 1997-2003!
Exactly. With a simple hardware system, you can afford to think of the underlying hardware while writing the software. However, try writing software with the hardware in mind when the hardware is the shuttle. The 'black box' mentality is important because you can easily get completely overwhelmed by trying to think of how every little piece of hardware fits into your software design. The hardware should work and not be concern. Instead of worrying about what the hardware will do if your software crashes spend that time making sure your programs are stable and won't crash.
There's a growing sense that even if The Future comes,
most of us won't be able to afford it.
-- Lemmy
... but the lack of experience.
Programmers have to consider subsystems as abstractions: there's a limit to how many things the brain can deal with at one time. We know that this kind of thinking produces cleaner designs which are less susceptible to bugs and security holes.
Knowing the limitations of the "black box" and what will break the abstraction is the product of lots and lots of experience. I don't believe there's any way to teach this - it's something that you just have to live through.
That's why senior level developers can continue to justify their existence (and higher price tags)!
on the new Death Star, I found that trying to envision the "Big Picture" interfered with the specific requirements of my task. I needed control mechanisms smart enough to deal with Storm Trooper suits, regular Empire uniforms, robots with various temperature ranges, Wookies. It needed to be able to maintain a comfortable temperature range in the beam tunnel vicinity even during firing. And it needed to be efficient enough that they wouldn't shift power from the exhaust port shields to the jacuzzi heaters like they did on the old Death Star.
The only thing that school prepares you for is to get an entry level job where you can gain the experience to write reliable software.
School will get you up to speed on new terms and concepts, but the only thing that will make you better at writing good code is to read good code, write your own code and compare it to the good code, notice the difference, and adjust your approach until your code is "good".
Programming classes may encourage a 'black box' approach to programming, depending on what language you use. The reason for this is that it all relies on how high-level the language is; if you're using PHP then chances are you won't be worrying nearly as much about the hardware of your system than if you're using C, assembler, or machine code.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
The problem with these programmers is that they rarely understand what can and does go wrong with the outside world. It is always amazing to me that there are people out there that assume that everyone has a 100BaseT Ethernet hub between the front end and the back end or other stupid assumptions.
The issue that crops up most when programmers think in black box terms is that today's software is not spec'd out enough so that the end user does not get what they wanted but the programmer did not solve it by asking. Too often the problem is very fuzzy and thus the programmer is there to help clarify not just implement.
Without a well rounded programmer looking at the overall system (or his/her boss), you will wind up with chatty, buggy applications that was what the user asked for but not what they needed.
There was a great Slashdot article/discussion on leaky abstractions. Search and you will find.
The usual problem applies here. While the world evolves... university professors teaching material usually doesn't.
...what e-mail program should I use?...let me consult my magic 8ball! *slosh slosh* hmmm... "outlook not so good"
We need new paradigms, and more best-processes. We need a design for system mentality and a design for integration production engine.
Or just realize that spaceflight is risky, and it's not always the great unknown that will bring us down.
I'm not advocating carelessness, just to point out that these space machines we build tend to last 15 years between failure and are so complicated it makes the MS Windows operating system look like Linkin Logs. REAL engineers built that sucker, and they aren't infallible, but they are thorough.
I'd bet my life on their processes any day.
Computer science professors and courses are more concerned with the methods, ideas, and logic of computer programming and design. The idea is to create a totally abstract system, hardware or software, that can then be implemented on any system. This is the purpose of "black box" programming.
While I agree with you that programmers should understand the hardware they are writing for, any knowledge of that hardware is biasing their creation of a system to run on that hardware and further removing itself from computer science's notion of total abstraction.
I recently installed the recent version of the accursed RealOne player to watch an rm file. I hate Real player more than can be described by words and it just seems to be getting worse.
So I pop it up to view the file, and what happens? I get the movie playing in a window on top of the Real advertising/browser thing. It spontaneously pops up a "help" balloon giving me a tip for how to use the browser window. The balloon is sitting RIGHT ON TOP OF THE GODDAMNED MOVIE IMAGE. It goes away after a few seconds of frantic clicking, but the point is clear, these programs are often a monstrous brew, created by too many chefs. They just throw in features, and there doesn't seem to be someone sitting at the top, deciding whether these features actually contribute to improving the final product, or just make it worse.
Then there's Office, which, by default will turn 2-21-95 into 2/21/95. ????? I have to dig through numerous help pages to figure out what subpanel of the preferences menu will deactivate this. Worse, I enter 23 hello on a new line in Word, and hit enter, it auto indents, adds a 24 and positions the cursor after. !?!?!!?!?!?!?!!?
How many times I've had to fight this particular feature I can't tell you.
And it's certainly not just a closed source thing either, if anything, some open source GUI packages are even worse. Although, to be fair, I don't expect as much from open source stuff, because noone's getting paid. But when a program created by paid programmers is just badly done, I get infuriated at the incompetence, at the hours wasted taking a usable product and making it actually worse by throwing in garbage features.
It's been said a million times, but if we made cars the way we make software, noone would get anywhere.
I think I'm going to roll up Asshat Linux 0.1 today. It will be the exact same thing as gentoo, but the fortune database will be nothing but slashdot posts and RMS quotes.
--
the strongest word is still the word "free"
I've worked on large systems where I've had a lot to do with many parts of the system, and not been able to hold the whole thing in my head all at the same time.
And as a someone that's had a lot to do with the building of these systems I have a better chance than most. New programmers would have no chance. We need black box systems to enable us to continue working.
The Real problem is that the black boxes we define don't have nice sharp edges, so when we put 2 boxes next to each other there are cracks. Cracks for the crackers to crawl through.
Joel Spolsky wrote about it and called it The Law of Leaky Abstractions
DWR is Ajax for Java
why not enclose that link in tags . Not that it really matters since its a dead link anyways.
i could not think of anything clever.
You need to give your black box objects a coat of super-black paint... security by invisibility...
--sex
Very popular slashdot journal for adul
It is essential that every programmer in a big system only thinks about his own problem, and uses the other parts as a black box.
Say I want to use some library. Then it has a documented API, which explains how I can use it. I don't need to know more. For me as a programmer, that means:
I'm certain that without these black boxes, no big software engineering project would be possible. The human mind can't keep track of everything in a whole system at once (except for some simple cases - like embedded systems, perhaps).
It is done sometimes - I believe perl looks inside a file struct when reading/writing files on some platforms to get faster I/O than standard C, for example. But that's only as an optimization after coding the general case, and even then I don't believe it's a good idea.
For hardware, the story is much the same. Any speedups specific for the hardware are optimizations, and they should only be looked at when the program works, after profiling, when there's a speed problem, and the algorithm can't be improved.
Remember the rules of optimization: 1) Don't do it. 2) (for experts only) Don't do it yet.
Black boxes in software engineering are your friend.
I believe posters are recognized by their sig. So I made one.
And heck, I don't know. I mean, is it great that I can now call malloc(1000000); and get a valid pointer that's just usable? Yeah probably. In DOS, I wasn't shielded from the memory manager as much, and to do something like that, I had to write my own EMS memory swapping code! That was a PITA, and kept me from the true task I was trying to solve.
So a modern 32-bit malloc() is a black box for me. Cool. It's empowered me very nicely.
However, something like WinSock has become a big black box for people too. Okay, great. So it's really easy, in 5 function calls, to open a socket across the internet and send data. But you've missed the nourances of security. So now your app is unsafe, because you weren't forced to know more about what's going on in the "black box."
Well, that's all I can really say in my post. Black boxes are a darn complex issue to talk about. Anyone who attempts to distill this down to a "yes" or "no" answer is probably missing a lot of the completixy at hand in the issue.
Very interesting question to put forth, though. Good topic.
Nothing is so smiple that it can't get screwed up.
I concur that there is a lot of this kind of niavete out there, but on the other hand, there are always the few that will go above and beyond. While my schooling tends to focus on a more abstract approach with emphasis on OOP, I have also started working on embedded systems in my own time.
It seems apparent enough to me, that any passionate CS student will not be satisfied with a mystically based understanding of computer architecture; and will in turn educate themself. I propose then, that any kind of 'black-box' mentality is more a reflection of the students' drive then their education.
________________
"A man prepared who hesitates, is lost." -Dante The Divine Comedy: Inferno Canto XXVIII, 99
why's it always got to be the "black box"? Huh??
This is my sig. Its pathetic.
Okay the "big picture" college profs should be showing you is this one.
Is black box programming a pipe dream? I wouldn't go that far, as software engineering/compsci is a relatively new "science". At any rate, I know that I am very reliant on knowledge of the underlying platform that my code is running on. When a piece of software (especially one that I didn't write) doesn't work, I often resort to tools like truss/strace, lsof, netcat, /proc,etc... to help me determine what is going on "under the hood". I can figure out what ports, files, dlls, and logs the software is using in a matter of seconds, instead of resorting to a dubugger or printf's.
;-)
I'm no superstar engineeer, but I find this methodology (my window into the black box) so valuable that I'm often frustrated by collegues who refuse to learn more about an OS/VM/interpreter and make use of it. It is also what most frustrates me about troubleshooting in windows.
While it's true that I don't know much about windows, I get the feeling that these kind of observation tools that are so common on unix-ish machines aren't quite so prominently available on winderboxen. Sure, you can figure out a lot about a problem using MSDEV (what I remember from college, where VC++ wouldn't stop opening everytime netscape crashed), but it isn't available on ANY machine that I ever troubleshoot.
Hell, even when I'm programming java I use truss to figure out what the hell is wrong with my classpath
I suppose I'm not too threatening, presently, but wait till I start Nautilus
A black box is something you don't know anything about. You make it not a black box by learning something about it. Most of software development is spent learning about the system (reading documents, searching indexes, walking through code with a debugger). How much you can get done is determined by how little you can get away with learning before fixing a problem.
That's not universal, it might not be the case with the shuttle software, but it's true for a lot of software. It's definitely true for my job.
Try again: here it is
I'm not sure if there was a lack of communication with your prof, but the concept should have been "SOFTware as black boxes". This is the concept of data hiding, which is a good thing. The cornerstones of software engineering are abstraction and encapsulation, and data hiding is a big part of encapsulation.
The hardware is (to an extent) a "black box" from the standpoint of any higher-level language, including C and Ada. That is the whole point of software portability, which is also a good thing. Both of those have been used for a tremendous number of embedded systems (particularly Ada, which is used for quite a lot of the space shuttle software). One must know that one's algorithms will execute deterministically in the required time, but knowing in detail how the data and instructions flow from memory though cache and processor is emphatically not required in 99% of cases.
Detailed knowledge of computer hardware is helpful to software engineers, but by no means essential. Talk to the hardware folks if you have a question. ;-)
By the way, don't forget another important axiom:
"Premature optimization is the root of all evil."
Galileo: "The Earth revolves around the Sun!"
Score: -1 100% Flamebait
There are many things computer science education does not teach the average student about programming. This is burdened by the fact that programming can vary significantly across areas of CS (i.e. networking vs. database implementation) and even within the same area (GUI programming on Windows vs. GUI programming on Apple computers).
When I was at GA Tech the administration prided themselves on creating students that could learn quickly about technologies thrown at them and had broad knowledge about various areas of CS. There was also more focus on learning how to program in general than specifics. This meant that there was no C++ taught in school even at the height of the language's popularity because its complexity got in the way of teaching people how to program.
Students were especially thought to learn how to think 'abstractly' which especially with the advent of Java meant not only ignoring how hardware works but also how things like memory management work as well. In the general case, one can't be faulted for doing this while teaching students. Most of my peers at the time were getting work at dotcoms doing Java servlets or similar web/database programming so learning how things like how using linked lists vs. arrays for a data structure affects the number of page faults the system makes were not things that they would really have to concern themselves with considering how things like the virtual machine and database server would be more significantly affect their application than any code they wrote.
Unfortunately for the few people who ended up working on embedded systems where failure is a life or death situation (such as at shops like Medtronic) this meant they sometimes would not have the background to work in those environments. However some would counter that the training they got in school would give them the aptitude to learn what they needed.
I believe the same applies for writing secure software. Few schools teach people how to write secure code not even simple things like why not to use C functions like gets() or strcpy(). However I've seen such people become snapped into shape when exposed to secure programming practices.
If you are in a very small team, you obviously need to be aware and conscientious of the system as a whole, but on a larger team, if everybody has a view of the whole project with their own vision, everybody goes a different direction. It is better in this case to have each individual or group be concerned with following the specifications for their individual componant, and having lead programmers/designers integrate as needed. I've never worked in embeded systems, so I cant tell you if that holds up.
People who think they know everything really piss off those of us that actually do.
My undergrad studies for computer science included fundamental understanding of gates and boolean logic. We also studied some of the microcode that goes into processors. So we went from the level of the gates to simple chips to the basics of processors to assembly to operating systems to applications. It wasn't taught in that order from the ground up. Algorithms were studied at the same time as chips, but it worked out well. Anyone getting a comp sci degree from Pace U. in NY has at least a fundamental understanding of computers from the ground up. However I have a coworker (developer) with an electrical engineering degree. He has much better knowledge of the electronics from beginning to end and he's a great programmer.
I find knowing how things work from the bottom up makes me better at building on top. I find the most ignorant and least innovative developers to be those with only a high level understanding of how the underlying software and hardware works.
Developers: We can use your help.
The best case scenario will always be for each member of a development team to understand every nuance of the system and every detail of its interface with the underlying hardware. However, it simply isn't practical (and for some systems it might not even be possible).
Go to The Hun if you want "big" pictures.
Krystal_Blade
It will be easy to motivate our fellow man; there is hardly anything people treasure more than not being annihilated.
More specifically, the Big Picture is Ninnle Linux!
http://www.fastcompany.com/online/06/writestuff
Every Java programmer should at least look at the source for the Java base classes, and ultimately should understand the VM. C++ programmers should at least read "Inside The C++ Object Model." C/C++ programmers should peek at the assembly their compiler creates. Python or Perl programmers that have a good understanding of the internals of their interpreters are going to write better code.
All these abstractions are there so you don't have to sweat details all the time. But this shouldn't be misconstrued as "never."
It depends on what field you're in. Are you writing a portable software component that is basically a business logic module, or are you writing code that is the whole system (embedded micros, etc.)? Software components, if they need to be portable, should be blackbox-esque.
Embedded stuff? Hell no. Software is there, usually, to facilitate the hardware. The smaller, tighter, and faster you can get the code, the cheaper the hardware to run it on, and the cheaper the overall design. That was one of the things that annoyed me most about college programming classes, etc. - the fact that everything was abstracted into a world where all hardware was equal and all code was perfectly modular.
I'm an embedded programmer. The hardware is mother, the hardware is father. Sure, usually I have at least a bootloader and/or possibly a separate kernel that abstracts the hardware just a bit. However, while abstractions are nifty and help prevent code duplication and coder error, the machine itself certainly doesn't need them and all they do is chew up processor cycles. That leads to needing bigger, more power hungry hardware to keep ahead of all the bloated code. The trick is balancing the two.
Sure, I can write C for a PIC with 64 bytes of memory and 2K of program space as well as I can write C for a 16-way Alpha server monstrosity, but the style of code I write will be massively different. There ain't no malloc, no filesystem, etc. on a PIC. You can't interface to a ODBC client to store data - you have to write the flash interface routines, check for errors, figure in wait states, etc. To manipulate certain registers, you have to execute very specific assembly in a very specific order, or you risk falling into bugs in certain revs of silicon.
I also tend to think that programming is moving bits around between registers. It's a fun way to flip transistors, essentially. Likewise, I've always thought that computer engineers should start at machine architecture, then move to assembly, then C, then C++, etc. Make them really think about what the abstraction each language buys you vs. what it costs in bloat and inefficiency (because no optimizing complier is as good as an hard-working ASM guru).
I'm not against portability, but I am against bloat and inefficient coding in cases where it doesn't buy me (or future maintainers) anything.
ND
We're taught black-box programming in school because of the simple fact that hardware changes, frequently. Abstraction is necessary, or we would all be writing in assembly.
But, of course, encapsulated programming must be designed that way. A top-down, system-wide approach to security, error-handling, IPC, etc must be established in advanced and trickle-down to each component, even if it means that your software simply returns a -1 on a failure, but the higher piece in the heiarchy sees your -1 and the -1 of other parts and determines that a boolean value of FALSE is deserved, which it returns to its parent component, etc. Eventually, a high enough component will recognize the true problem and do what ever it needs.
The point is, black-box programming works, but only if there is an underlying plan in effect. It's the five (or is it 6?) P's of a successful project in action!
Do Slashdot readers think that the theories used to teach (and learn) programming lead to programmers that tend to approach problems with a 'black box', or 'virtual machine' mentality without considering the entire system?
Having a human brain leads programmers to approach problems with a 'black box' or 'virtual machine' mentality.
I don't think we were built for a natural 'big picture' view. We were built to understand our little piece of the African savanna from inside that box.
All the 'big picture' stuff is doable, but not as naturally. We will always feel a little more comfortable inside the box.
Humanity will always have to force itself to think outside the boxes we constantly make to aid our system of modeling reality through perception.
In theory, a perfect system could be built from a lot of communicating "black boxes" without knowing the whole system. That's what the fun is all about. In the Real World however, this approach is not possible since there are too many broken black boxes and too many unsupported standards that make a secure approach impossible without knowing or at least controlling (=choosing which black boxes to use) the whole system.
Fight hunger. Filet a politician and send him to a 3rd world country of your choice.
Hrmm. Some of the philosophies of Unix and its revolutionary system of pipes tend to emphasize individual components, each doing its job well. (Though Perl as the swiss army chainsaw with sometimes surprisingly better performance has something to say on that...)
I'm probably living in a dreamland, but I really think small teams, where people can realistically have a hand in all and therefore knowledge of all parts of the system, can do almost any software project. It seems to me that "mythical man month" scaling problems really start to attack productivity, even with medium size teams.
SO YOU'RE GOING TO DIE: The Comic for Dealing with Death
I've been doing pda programming for both the pocketpc and the palm os.
The application for both is intended to be identical, but the api is different for each device.
I designed the app originally for the palm, but now I am porting it over to the pocketpc. Unfortunately, the api is different enough that little of the code is portable.
If I had known I would be coding for both, I would have tried to design the code to be more portable. Knowing the requirements of both systems might have allowed me to factor out the device-specific sections.
Where I work, we have two software groups (in practice). One is an integration group, and they actually sit in the lab with the hardware. They all know how to reset the hardware, replace cards, even perform diagnostics on the system. Their code is almost invariably good, and their knowledge of the system is almost as formidable as the hardware designers. Sadly, they are limited to writing BIST and self-test code, not application code.
The app guys, on the other hand, don't even sit in the same room as the hardware. They have to call someone to hit the bleedin' reset button, and most of them barely know how to power the equipment up. They code at an abstract level, and frequently miss nuances of the hardware that were explicitely designed in for the task at hand. Their code is *always* bug ridden, and when we have serious problems on the system bench, nine out of ten times we can point the finger at a software engineer.
So, yes - I can say from experience that software engineers could benefit from paying attention to the entire task, not just the code they are being paid to produce.
One side of the issue is that if you attempt to look at any sufficiently large and complex system it will overwhelm you with its complexity. The human mind can only deal with a certain amount of complexity at a time before overloading. That's the reason why object-oriented methodologies were invented, to attempt to chop up a large and complex problem into smaller and more manageable pieces, so you can deal with certain things as "black boxes" and move on to the bigger picture. Sort of like zooming in and out. A graphic artist would never think of working at a single zoom scale when editing a picture, she would zoom in for fine work and zoom out for an overall view. Treating things as black boxes is done so that you don't lose sight of the forest for the trees, not the other way around!
But of course, as my own experience in embedded systems development and electronics work has taught me as well, it does no good to simply leave things as black boxes. You also have to know how the black box works on the inside before you can go on to treat it as a black box. I had to learn the ins and outs of semiconductor and transistor physics before I learned how to use logic IC's, which have these components as their basic building blocks, so that I'd understand the limits and quirks of these devices. I think the big problem we have is that people are generally unfamiliar with how the many black boxes they use actually look like on the inside, so if their system winds up eventually tickling limitations or quirks (which, as the complexity of the system they're building grows becomes more and more likely), they have no idea what the hell is going on or what to do about it. In other words, too much zooming out, not enough zooming in, so you get work which has too many rough edges and not enough fine detail.
Salon had a highly insightful article some years back about this very topic as it pertains to software engineering: "The Dumbing Down of Programming", by Ellen Ullman, Part One and Part Two. She talks about the way too much knowledge is disappearing into code, and the problems that causes.
Qu'on me donne six lignes écrites de la main du plus honnête homme, j'y trouverai de quoi le faire pendre.
"Black box" thinking provides the ability to encapsulate or modularise certain functions. You can test the black box to the specification (if the specification is good enough), and determine whether the black box will do its job in the specified context correctly.
Enter "reuse." This can change the context in which the original black box was required to operate. It may work, but it may be degraded in some way (for example, a data management system designed for a year-old HP rp7410 server having to be shoehorned into a five-year-old K460). Or the black box will have to be modified to deal with a new set of conditions (what was a legitimate error in the old context is OK in the new one, and the black box has to have additional input to tell it which context applies). Either of these can be managed.
Things get difficult when this black box is picked up without careful examination and embedded in something completely unknown to the original black box design team. The specification for the black box may not match this new context, and things will break. What's changed? The picture, not the black box. It is not the designers' fault that the black box doesn't fit the new context perfectly. But designers must be prepared to adjust them to new contexts if appropriate. And design managers must be aware of this possibility.
In short, the black box methodology is fine, but it is limited in that it does not recognize changing contexts. Everyone involved in the design process must understand this.
The Seventh Rule: Take others more seriously than yourself, particularly when you are leading them.
My sense is that this does happen, but not usually because of any flaw in the programmers or engineers. More often, it's the result of management not giving more information than is necessary to complete the task.
It's important to take the whole team, as a group, through the big picture. Even if it is just a short overview meeting, there is significant value in making sure that everyone knows that their assignment is part of a larger whole, showing how the pieces are intended to come together, and giving everyone a context for their individual bits.
My experience is that being given all the additional info doesn't take too much time, doesn't overwhelm anyone, and produces far more usable results. Letting everyone work in a vacuum, the other extreme, tends to cause integration nightmares and lots of wild tangents that make sense ONLY in the context of one little bit while working against the overall goals.
http://drteknikal.blogspot.com/
first off im just wondering after reading these posts how many people thought you were discussing the flight data storage in airplanes referred to as "black boxes" *grin* :P
on a serious note, in my undergraduate studies in computer science, we were first taught how everything works with hardware, then how to hardwire program it, then microprogramming, and so on. By the time you progress to the next step you cannot forget about how things work and relative efficiency, for example whenever I do anything now I am always reminded of that damned microprogramming.
I dunno, if you are taking a black box approach to programming, chances are that you arent in a coding position where you need to take hardware into account. If you are in a situation when you cant/shouldnt take a black box approach I would hope that you have had enough schooling and/or experience/knowledge to take a more advanced approach
But if you are using frontpage...it doesnt matter what you do
[I can picture a world without war, without hate. I can picture us attacking that world, because they'd never expect it]
The project I am currently on has developers at 3 Sites(each 1500 miles or so from each other) working on different aspects of the project. One of these Sites has a very open minded Manager who stresses that even though the part being worked on by his people is only a small subsystem of the whole project, that no code is developed unless the design engineer has talked to the Guys from the Other Sites and made sure that interfaces are well Designed and Issues are worked out. One of the Other Sites takes the BB aproach and is severely locked in tunnel vision on the parts of the project that are in there ball court. The Guys working under the better Manager consistantly deliver on time, with very few defects. On the Other hand the Guys locked in Tunnel Vision consistantly Deliver late due to the higher instance of defects. There is a time and place for Limited BB type of programming but I believe it is bad programming to not at least Stick your head out of your Cube for a few minutes and see what is going on with the rest of the project.
--Im an oven mitt, not an engineer! (SLArbys Radio Commercial)
I started my career (long ago, in a galaxy far away) developing embedded systems, and much later, when running an R&D lab, came to the conclusion that, excepting (importantly) user-interface design, embedded systems were the best crucible in which to learn the right balance between modularity and holism in systems design and implementation.
It's easy for programmers who have only worked on PCs to lose sight of the notion that programs affect the world, but when you are controlling big machines that, improperly instructed, will destroy themselves and the people around them, you begin to think twice about your coding tricks, your testing, and the interaction of your component in the system as a whole.
But there is an underlying assumption in the question that modular design and system holism are mutually exclusive, and I don't accept that either. I also except user-interface design, which is more sociology and psychology and neurology than computer science.
You are correct, however, in supposing that security is particularly vulnerable.
Here's one (true) story, which I will deliberately leave unattributed: a programmer is writing code to control the dual vertical bandsaw in a sawmill -- two huge saws, each 12 inches of high-tensile stainless steel with 3-inch teeth, stretched tight between two six-foot diameter wheels and running at 10,000rpm. A log is pulled on a chain through the middle, so a cut can be made on both sides. Logs enter the system, are measured with a laser scanner, and a queued (physically and in the control program) before entering the bandsaw.
The old fart programmers used to simply store log data in an array of sufficient size to hold the maximum number of logs that could ever be in the system, but are cognizant of the problem of "phantom logs" when a log falls off the belt or otherwise leaves the system in an uncontrolled way. The clever young programmer decides to use newly-learned techniques of memory allocation and linked-list design, and build a replacement.
During mill installation the system is tested and appears to run well. At the end of the shift, however, as the last log is about to be run through the system, the operator discovers that there is no data in the queue for the last log, but decides to run it anyway. The computer dereferences a null pointer, grabs garbage data, and tells the bandsaw to set to an impossible position.
Because the mill is still being installed, the stops on the bandsaw have not been adjusted, and the saws set to position "0" -- and run into the chainguide in the middle. High-stress stainless at great speed meets six inches of fixed steel, and the saw blades explode, burying foot-long shards of stainless steel sawblades up to four inches deep in the walls of the mill, destroying the operator's booth, and causing tens of thousands of dollars damage to the mill.
Whose fault was it? The operator, for running the phantom log? The hardware installation guys, for not setting the stops on the mill? Or the programmer, for not constraining the output of his program, testing more completely, and using simpler techniques. Answer: all of the above. Better modules would have forestalled the problem, and better systems holism would have forestalled it as well. A combination would have given an even better margin of error.
This has led me to the following conclusion: in order to get a CS degree, every programmer must write code that will lower a 10-ton machine press a maximum speed to within inches of his chest, and then stop it. We would have more careful programmers if this were the case. If they went on to write security code, we would have fewer holes.
gnet
...and just what the fuck made this space tragedy more horrible than the one seventeen years ago? Same number of deaths, same situation, but this one on landing rather than launch. So much more horrible.
Some of you have complained that the Space Shuttle reference is gratuitous in this article.
To you people, I must point out that in this post-columbine, post-9/11 world, those people who are able to leverage all available datum into a synergetic process are the most likely to accept success in their lives.
Slashdot is jumping the shark. I'm just driving the boat.
I think the question is confusing the "black box" concept with "design patterns". Unless you are programming assembly for Atari 2600 and need to pay attention to how many CPU cycles each instruction takes, there is no reason to consider underlying hardware (embedded issues aside). A black box in this classic sense is simply an abstraction for the zillion types of hardware ever made. The STDOUT file handle is a perfect illustration of this as it has migrated from physical devices to VT100 telnet sessions across the world. Design patterns, OTOH, can easily lead to "cargo-cult" programming, which is always bad.
Don't laugh, but this is one of the reasons why it's important to have solid requirements BEFORE you being coding anything. Most projects don't, I know, but something as complicated as the space shuttle would need to be completely spec'd out beforehand. After proper requirements and specs are laid down, the programmer should then approach the system as if it were a black box...with a lot of restrictions.
The idea behind black box development is that you don't need to know what the rest of the system does...your component takes input and delivers output. That's a Good Thing (tm). Requirements are what tell you how to design and implement your black box, ie, you can't have more than 1ms latency between input/output, you can't assume some system variable is going to be out there, you can't assume your process won't be interrupted. Given these sorts of requirements, your part of the system SHOULD be a black box...someone else should send you the inputs and know what kind of outputs they're getting out. Assuming you correctly followed the requirements (that's what QA testing is for), they know what they're getting.
--trb
Ah, I think we have someone here who DOES see the big picture.
..."
There are lots of times, at least in my experience, where it's not the programmer's fault in how the program works.
I've seen specs come down from higher-ups who have no idea what they are asking for. I'm a little bit luckier though. The analysts we have here tend to spot these problems long before I get to program. But occasionally some do slip through. I have loads of fun ripping these things to shreds. I feel like a professor at college with my little red pen. "Ah, wrong...can't do that! What does that say? etc, etc
Aside:
That is also usually a good stall tactic. If I'm swamped with other projects, I'll send them a flurry of notes, overwell them with spec questions. It usually gives me a few days.
It's tough to think outside the box, when there is no box.
Sean D.
"Hmm. I am to metaphor cheese as metaphor cheese is to transitive verb crackers!"
You must consider the implementation, ie. real-world limitations and requirements/trade-offs otherwise your solution will not be acceptable, but if you go too far in this respect, you'll create a system almost no one else will be able to comprehend and maintain.
On the other hand, if you go fully-elegant (applying every theory that might be applicable) to the construction, you'll wind up exhausting processing power/memory, etc...
That's where judgement comes in; knowing where to cut, and specifically what to document to give maintenance programmers a leg-up on whatever short-cut you engineered to accomplish the feat.
Make sense?
I don't know the meaning of the word 'don't' - J
How's this one any more "horrible" than the other shuttle lost? People's priorities are distorted alright. It's a tragedy - yes, but it's also a tragedy when hundreds die in a regular air crash. Yet, do we mourn them the same?
I believe the opposite.
I am IN school where they mainly teach us abstract concepts and not specific programming languages.
What good would it be to learn a specific programming language in the rapidly changing technological world? Keep your Cobol and teach me recursive binary tree algorithms.
If you cant go and teach yourself how to apply these concepts to specific languages, either you arent meant to be coding or your school did not help teach the second step of algorithms which is the application
Basically, since we were mainly taught algorithms, our assignments would be in a random language like C, java, etc, and we would code portions of these algorithms. Sink or swim, but it worked, I know everyone in my graduating class can code in any language efficiently and effectively.
*go waterloo*
[I can picture a world without war, without hate. I can picture us attacking that world, because they'd never expect it]
Well, programming is, at its root, about controlling complexity. A good program (not that *I've* written one) will have sub-components within it that largely act as black boxes to one another. It is a great and rare skill to recognize where the boundries are in your program and establish them early, to avoid painful refactoring later.
In my experience, it is when something *isn't* a black box that things can get seriously fsked. "What? I set a global variable and now the app seg faults when I click that drop down?" Ahhh, not "black boxy" enough.
My perspective is from a higher level than embedded though. Embedded is a whole different game, although the "controlling complexity" insight of higher level programming languages no doubt sill applies (as far as it can go)
Cheers, pratOuch.. looks like somebody hit a nerve.
The correct level of abstraction for a project is often hard to find, even for very experienced programmers. Sometimes you have to raise the level of abstraction of an overall system, or you will never get to the point where you can move forward on your piece of the process. Generally, I've found that the problems lie in the areas where the pieces don't quite fit together properly; namely where one person's code doesn't follow the contract a second person was expecting. A lot of the time, even for mid-size problems, it would be impossible for one developer (or a team) to have an end-to-end understanding of the problem space.
As far as I can tell, there is only one way to avoid the "black-box" problem, and that is to have one person code the whole thing, which is very likely infeasable. The further you get from "your" part of the system, the more abstract it is going to get. If your abstraction is faulty, there is going to be trouble, but I wouldn't say it was caused by treating the problem as a black box.
In high school I did a 2 year Computer Studies course.
During that period, one night I went to a heavy party and then spent the following day trying to write functional code whilst suffering a hangover.
This was the only experience from the course which mirrored anything which happened to me since I started programming professionally
Yes, I think the 'black box', you-don't-need-to-know-the-hardware approach is dominant in teaching. It's a very useful abstraction, it works most of the time, so I think this is fine.
Of course, any decent programmer (or researcher) should be aware of where this abstraction breaks down and where you do need to know the hardware (or lower layers) of your system.
But frankly, I don't have any idea of how to teach this... Dumbly insisting on looking at the hardware when doing your Quicksort certainly doesn't cut it. I really hate Knuth's only-assembler approach in TAOCP. He has a very valid point, but does he really have to make it on every single page? Worst is, I'm not even sure if he reaches his goals, or whether people don't simply turn to other books...
I guess the best approach to teaching would be to stick with 'black box' as the basic assumption but to point out examples where this model in insufficient, and then hope the students get it...
Part of the reason that the Big Picture difficulties crop up is that programmers are problem solvers. Their problem is "How Do I Do X".
And so they write something that Does X.
This goes wrong when the problem isn't "How Do I Do X", but is "How Do I Do X, Given Y". In these cases, Y may or may not be available to the programmer. The programmer may not understand that Y is important. The programmer might not be able to determine how Y applies, or how to get Y out of The Big Picture. Or the programmer may just be lazy and figure that someone else will take care of Y for them. Or the programmer decided to write the most generalized method for Doing X, so they can Do X Anywhere.
Solving this problem of ignoring Y is going to take education: First, know that Y exists. Second, find Y. Third, code for Y. Finally, when Y is "important enough", recycling code from somewhere else won't cut it.
If I have been able to see further than others, it is because I bought a pair of binoculars.
Your comment is funny in a juvenile sense, but also indicative of one of the problems that some developers have - they are so impressed with their own abilities in their given narrow niche (i.e., they are geeks) that they fail to understand that they are not a one-man band and would be useless without the rest of the players. While a given manager may not understand in as great a detail what the specialist programmer does, a good one understands how to guide and support it, integrate it into the overall technical and business picture and by extension, the world. No I am not a manager, but I do understand that there is a value in the contributions that they make.
I agree that designers in general (both embedded software developers as well as hardware developeres) tend to confine themselves in their own level of abstraction. The black box model here helps as a divide-and-conquer mechanism. If for exmaple you are writing device drivers, you would abstract hardware to memory locations and registers. And at the same time, you hope that the device API - the abstraction you are building for the layer above you - is general enough to cover all applications.
That by itself is a good thing. However, it assumes that there is some intelligent being that has outlined the layers of abstraction and black-boxes upfront. This is hard, and relies on extensive experience. How would one go about and select an OS for an embedded system, for example? How much knowledge about the application or device is used in making this selection ? (And how much of this choice is 'just guessing') ?
Thus people use black-box models (like OS'es) because they are known to yield some results, not necessarily the best one.
My question is, is it really feasible to know everything about a given system? A jack of all trades is a master of none, after all. And that isn't even taking into account classified systems, where different parts of the working system are not only farmed out to different departments, but sometimes different companies. This is why very detailed software specifications are so important. It is the responsibility of the hardware engineers to tell you what the machine expects. Can you, who specializes in programming, really be expected to know everything about the hardware if you didn't design it?
Tell you what, let's forget about programming and consider other things where responsibilites need to be split up, like building a house. While a builder may do general construction, there are areas where he may not be so good in, so things like electrical, roofing, masonry and such will be farmed out to subcontractors who specialize in said activities. The builder still knows the big picture, but can't handle the miniscule details. If he tried to, certain aspects of the house would probably be poor quality and take way to much time to do. The way I see it, programming is no different. It's about making the highest quality product possible in the shortest amount of time. And like everything else, it can't be expected to be perfect.
-R
The short answer is "yes and no". Of course.
... not to mention really weird things, like using an FFT in a DSP to implement a matrix multiply for graphics. (Sounds weird, works good.) The problem is that people are being trained to think that they can choose any kind of black box and expect non-functional issues to be handled by Someone Else.
In fact, it's worse than that because the common mythology is that you shouldn't consider any non-functional requirements until you're finished with the code -- as expressed in the common idea that you should defer optimizing for performance until the very last step in development.
Everyone with more than about 20 minutes of experience knows this isn't the way it's really done -- if it were, then every web system would be built first as one big honkin' monolithic program, and refined from there. Instead, we make lots of decisions (like use J2EE or not, distribute the servers or not, use an in-memory DBMS or MySQL, and so on) very early, in order to get somewhere close (we hope) to a system that does what we need.
We call this "architecture" and in general we don't teach it in school. This is in part because it's not very well understood as a discipline, IMAO because research in architecture has concentrated on representations rather than methods. (This is not to say that it's bad, don't mistake my point. People like Dave Garlan, Mary Shaw, and Paul Clements are doing good useful work; it's just not directed to figuring out what architects do rather than what their work products should be.) Len Bass has started doing something along these lines, and I have a paper in preparation. (You can read some good stuff about architecture here.)
The problem isn't that people program to "black boxes" though: in fact, you have to manage designs with abstraction at any time, and in an embedded system you invariably end up working with black boxes as well. For example, do you do an FFT in a DSP chip, do you implement it in a general-purpose processor?
Most of the programming courses I've taken focus on teaching you the language, and some of the tricks on using the language.
Didn't learn how to really program until I got into the real world and dealt with real problems.
I think a mix of self-taught and instruction yield the best mix.
.
I am the lord of the pun. Dance Knave!
In order to do a good job on your module, you need a solid understanding of how the components you directly interact with function. In addition, a superficial understanding of other components is useful.
For example, let's say you are working on the software for automatic transmission control in the car. You need an intimate understanding of the hardware you are running on, that's directly related to your job.
However, you also need a solid understanding of how the automatic transmission works. Understanding the mechanics of the gear change is important to understanding timing issues, errors that can occur, and how to deal with them.
It is very useful to have a good understanding of how a car works in general, to get an idea of how your product will be used. This allows you to optomize your product for likely scenarios.
Sometimes, for personal satisfaction, it is nice to know how the windshield wiper mechanism works, but it doesn't help you in any way to make your automatic transmission control better.
-Alison
I started programming over 20 years ago, starting on main-frames, moving to minis, PCs and now the Internet. Worked through the accompanying shifts in paradigms; batch processing to client-server and now internet/stateless. I believe that you are half right. I feel that recently schooled programmers come out of school pretty much thinking that they 'know it all' (present company excluded of course) and that they have nothing else to learn. I work with coders that range from 10 to 20 years my juniors and it constantly amazes them what this 'old boy' brings to discussions, designs and problem resolutions.
When I started coding, the batch job was king. It either had to run right or if it failed, it had to fail right. You built code to expect the unexpected. As the systems changed, the paradigms adapted or were reinvented. They all still tried to solve the same problems, how do develop a system quickly, efficiently and robustly (emphasis on robustly). The concept of building components or black-boxes is nothing new. What does appear to be new is the lack of building in recovery from problems or even the anticipation that there could be problems. What I tell my coders is that everything they develop should be with the mindset that it is going to be controlling their mother's heart/lung machine. Everything should be designed with that level of coverage and recoverability. Most systems today are not designed or coded that way. Look at all of the issues Win/Tell systems have with SIMPLE things like buffer overflows. That is inexcusable. It is easy to blame this on MS but that is a scapegoat. The problem is in how students are taught and how they are mentored after they leave school.
How is it where you work? Do you take the time to have weekly meetings with the other developers? To review and discuss code and coding issues? Or does the code go from the developer to QA (if you even have any) and into product with no peer review? This is not a problem that can be solved over-night but it is one that we need to work on and it will NOT be solved by changing languages or paradigms. Compiler cycles are just too cheap and it is just too easy to hit the compile button. To some degree, I have seen the enemy and they is us.
I have to use this cause I can't afford a real sig...
Frankly, I believe there is a lot of bad project management going on. That also applies to software development, not just integration projects. Usability issues much too often arise out of not spending enough time in prototyping and usability testing.
When it comes to seeing the big picture, well, let's face it - in the corporate world, having an open source-like eyeball count on everything kills productivity. However, the people who do the initial design REALLY should spend some time making sure that their design will work. They also should be kind enough to give the programmers a slight briefing on what sort of project they are part of.
Stop the brainwash
The "Black Box" design theory abounds because of the freedom it offers programmers from the dark ages of having to know the underlying hardware intimately before anything could be accomplished. It's what allows programmers to devote all of their time to doing what matters rather than pouring over volumes of errata and arcana.
The reason Windows became so popular, for example, is because its API offered programmers a way to manipulate graphics without having to make graphics calls. Variation from driver to driver was of no concern, and shouldn't be -- that's an IT issue which can be repaired without redoing the entire application.
And in a perfect world, there's no problem. If a driver hooks into an API properly and documents any disperity, then the black box theory holds true. Problem is, driver aren't perfect. A lot of them are designed for bare bones functionality, and only optimized as necessary (hence how Nvidia's still squeezing substantial horsepower out of my ancient GeForce GTS with every new driver release). Obscure hardware cases always cause trouble, which is why Dells are (sometimes) more reliable than "no name" machines with "better" hardware. Dell has the clout to make sure the drivers are as seamless as possible.
What's the solution for embedded developers? Design and test the drivers in house, so the black box coders have a shoulder to cry on when hardware doesn't act properly. But it should not be the core developer's job to know what goes on with the hardware. That kind of thinking bloats budgets, increases the complexity of the project and ultimately the cost. Modularity, even though it makes things more difficult to map in total, makes things easier to deal with on a micro level. If the application works when unit tested but fails on the release machine, then it's the driver's fault. Much easier to fix than it is to perfectly replicate the release in your tests.
Expecting EVERY software developer to be an electrical engineer as well is absurd unless you intend to pay them for both degrees. Better to keep it modular and put the pressure on the hardware abstractor to do a good job of catching the tiger's tail.
Hey freaks: now you're ju
The big difference is that when you are actually designing and coding (verbs!) you have to look into those black boxes. If you don't understand the subsystems/objects/subroutes that your code interfaces with, you won't know what boundary conditions to test, what assumptions the other subsystems are making, etc.
So now I always write well abstracted code (just like your Comp Sci 101 prof taught), but design with the big picture in mind.
Mental models are fundamental to how humans think about systems. We will always view systems as a collection of "black boxes" that interact in defined and predictable ways. The "black boxes" may not literally be black boxes, but they may be assumptions like "This set of linear equations approximates the true behavior of the system closely enough under all situations of interest."
Of course things get exciting when these assumptions turn out to be not good enough in all sutations of interest(try googling Tacoma Narrows Bridge).
The human mind is not unlimited in the amount of naked complexity it can deal with. Therefore we need our "black boxes" as firewalls to contain the complexity beast. However, a good founding in heuristics should train us to recognize situations where our mental models of system components are not serving us, or better yet to proactively look for ways in which our assumptions may fail. Black boxes should not be taken as given, but things that can and should be opened up, tinkered with, and rebuilt.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
I think he is one of the coolest trolls on slashdot ever. The fact that some of his postings got moderated insightful and interesting says a lot about how dumb slashdot mods are. If you didn't know him you would have a hard time telling if he's just a retarded linux user or if he's trolling.
So, keep going man. You should create a new account and use a new subject though.
As a programmer myself, I have come to realize that programmers are not always afforded the opportunity to "see the bigger picture." On larger projects, programmers can be assigned stubs or modules to write with guidelines on how to handle specific events. But in cases where a potential event hasn't been specifically defined -- yet the programmer knows it could occur -- a "default" event must be coded. This was ever so evident in Windows 3.1 when the error "not enough memory" could have meant one of dozens of problems.
"How do you expect me to see the forest with all these damn trees in the way?!"
Dude. I have never laughed so hard! I can only imagine a bad cartoon:
Sick man, sick. I am very proud of you. ^_^
> SELECT * FROM brain_cells WHERE synaptic_rate > 0
0 row returned
Modular programming makes things easy. So, black box is the way to go, no question about it. But, the question of whether knowing enough about the black box is enough for a good programming method is misleading. Here's why. A module runs inside some environment. It has inputs and outputs, and other factors, and the environment that it runs on. To know the black box, you have to know the environment, so there you go, you must program modularly, but must know enough about the environment the module is in, so you can fit it in better. Obviously, you only need to know the black box to program the black box. The question is how much you need to know to know about the black box, because knowing the black box also means you have to know the black box skin, color, shape, sturdiness, etc., which the environment dictates.
consider even something as simple as a file write.
even with C most programs make plenty of assumptions.
When was the last time you wrote code that wrote the data to the kernel buffers, waited until the buffers flush and then read the data back uncached from the device and test it for integrity?
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
Here are some thoughts that go beyond programming and include engineering as well. And not just systems vs black block, but concepts as well.
Here are some random thoughts.
Take the slide rule. Back in the days before destops, calculators, and palmtops, we had slide rules to do division and multiplication. You slide the rule for the numerator over the denominator
(I think, its' been so long). You then look at the
result.
The thing is, you can see how 'close' the result is to whatever you desire (in a circuit or system). You can intuit how close thing are. You can easily 'play with the numbers' with a slide rule in some cases. Slide it a little to see what it would take to get the desired results. A teeny amount, alot; whatever.
With digital calculators, it's a harder (for me) to see the changes visualy. All you see is a quanitive value. I can't look at the physical distances on a slide rule and make inferences.
I can remember doing the same intuiting with meters. In the days before digitization and computers, we had analog meters. A needle would point to the value (voltage, amperage, whatever). Often the 'movement' of the needle is almost more important than the actual value itself.
Take the tuning of a final output circuit in a radio transmitter. You dip the plate and tune for
proper power. With an analog meter, you can see the needle do a quick dip. Sometimes with a digital meter, you can miss the dip, espcially if the circuit has a high Q value. The motion of the needle of the meter controls the speed at which I turn the various knobs.
With a digital meter, I feel removed from the process of tuning.
Monitoring the electrical service for a facility, whether it be a radio transmitter facility, or even a computer room; I am much more comfortable with an analog voltmeter and amp-probe. It's far easier for me to watch for hiccups (needles jumping rapidly or slowly) to indicate something is happening.
I feel that all of these examples are important in my desire to be a part of the overall system, rather than being only a blind black box. I use my overall knowlege of what is happening in the system as a whole to get a 'feel' if what is happening right there and then.
With only abstract figures and a blind black box interface, I would feel much alone and out of touch with the reality of the system.
I think the same can be said about programming. In all of the projects I have been involved with, I have been fortunate enough to see the overall picture of the system at a high enough level to be able to able to be a 'part of the system' rather than a disconnected black box'. This is certainly true in my background in writing scripts to monitor the health of databases and operating systems.
Mark
Cleara
Gotta love that guy from Amsterdamn :-)
If anything, my experience bears out the opposite. The people I've worked with who didn't have any formal computer science education have more trouble looking at the big picture. They want to pick out a pre-packaged "solution" and plug it in. If it doesn't fit, they don't know what to do. If it breaks something else, they'll come in tomorrow to try to fix that.
The also tend to be "network administrators" and other "I think computers are cool but I can't hack"-type jobs. They're an embarrassment to my profession.
Joel Spolsky says it best in his Law of Leaky Abstractions. In other words, no box is totally black.
I think the problem is not so much the "black box" mindset, but rather the perfect black box mindset.
Being an EE who now does software design myself, I try to decompose a problem into smaller problems, and decompose the solution into smaller parts. However, I don't make the mistake of thinking that my smaller parts are each perfect - I try to ask "Now, if component X malfunctions, what effects will it have on this higher level assembly Y?"
The problem is that many time CS folks are not taught that the system can be imperfect, so by exclusion they believe it to be perfect - one plus one will always come back two, disk writes will always succeed if there is enough space for them, and so on. Folks are not taught that sometimes 1.0 + 1.0 != 2.0 (rounding errors), that disks sometimes fail (sector not found - abort, retry, cancel), and so on.
In Circuits 1, an EE-to-be is taught the idea of the perfect op-amp - infinite gain, infinite bandwidth, infinite possible output voltage, infinite input impedance. He is taught to use this model to analyze a circuit.
He is then IMMEDIATELY taught that the model is BS, and starts to add to it - finite input impedance, finite gain, finite bandwidth, finite offset voltage, finite output impedance. The EE-to-be is taught to apply those non-ideal behaviors when needed, and taught to judge when they can be ignored.
Sometimes I think the best thing in the world would be if CS and EEs had to work with robotics as part of their job. When they have to deal with sticky steppers, dust-clogged optics, and misfiring soleniods they will learn to be a bit more paranoid.
www.eFax.com are spammers
Programmers that suck work as programmer in the industy..
;)
Programmers that rock study e-technics and program in their spare time
There's two issues here that are being confused. Black Box programming is a GOOD thing because it forces programmers to realize what their inputs and outputs need to be. Black Box programming with unit/integration testing is a GREAT thing because it rigirously tests the interfaces between the black boxes to ensure buffer overflows, etc. don't happen.
Looking at the "big picture" is part of design, not coding, per se. If you design first and THEN code, you already examined the big picture and can happily code your black boxes without worry.
What you are talking about has been a serious concern for a number of researchers. In particular David L. Parnas has been working on this problem for many years now. The basic problem is that software specification often lacks in precision required to make appropriate decisions when writing the software. This is compounded by the lack of precise specification of the OS itself.
Take a look at the CRL, SERG and SQRL reserach documents at
In particular read the CRL Report 241 - "Predicate Logic for Software Engineering" - it covers some of the fundamentals. Then read up on CRL Report 259 "Formal Documentation of Well-structured Programs". There is tons of other interesting reports there that address some aspects of this very issue.That is what this is for, to remove the mop and bucket code that the developer need not spend 90% of their time coding, and 10% problem solution.
Managed code is the future.
That's really multiple questions rolled into one there. :-)
:-) The question isn't *should* you think in black boxes, it's at what level do you modularize. THAT'S the tricky part.
:-) Experience will only get you so far. At some point, you HAVE to know the Why in addition to the How. And of course, the Why is useless without the How. It took me a while to wrap my head around C memory handling, but once I did (and it was one of those lightbulb moments), my code got a LOT better and a LOT cleaner. I was tutoring someone in C++ at one point, and he simply couldn't grasp arrays and iterative loops until I explained it to him in terms of memory offsets. (Another lightbulb moment.) Of course, neither of us could have had any idea what the theory actually meant unless we'd actually written something so that we could apply the theory in our minds, and then go back and REapply it to the problem.
1) Blackboxes. You have to code in a black box. Well, either that or write the entire system, from the OS through the UI, in binary. I'd advise against it.
In general I'd say that everyone on a project should have a cursory understanding of every other part of the project. The UI guy doesn't have to know SQL, but he should know the basics of an RDBMS, and know that, say, user login information and their preferences are stored in separate areas. And vice-versa, the DB guy shouldn't have to worry about details like what color the widgets are going to be, but he should know that the user is expected to access both their login information and preferences at the same time 90% of the time, because that means combining the two will not only speed up access (fewer joins) but will result in less processing to format correctly, which means fewer opportunities for bugs. (Just as an example.) Making sure that everyone has that cursory understanding of the whole system, and that things are "chunked" properly, is the job of a good manager.
2) Theory vs. Practice. In theory, there is no difference between theory and practice.
That's a problem of the educational system. They don't integrate theory and practice enough. They do all theory, and then you get out into the real world and do all practice, and the two never meet. If they did, we'd have better programmers and better programs.
--GrouchoMarx
Card-carrying member of the EFF, FSF, and ACLU. Are you?
This reminds me of how the romans used to test their bridges: they put the designer under the bridge while marching over it with the entire legion.
Of course, a bridge i a MUCH simpler thing than a program, but, hey, 2000 years, all the bridges are still there !!!
We learn from history that we learn nothing from history - Tom Veneziano
I think that the problems that are mentioned in the question are not mostly due to people having a "black box" or "virtual machine" mentality.
In fact I believe it is exactly the opposite that holds true. "Black box" mentality is good when it is applied correctly and sadly most people don't posess these abilities.
You, in most situations, don't need to know the nitty gritty specifics of the underlying hardware, you do, however, need to know about the characteristics (there is a difference but I won't go into that here) of the underlying hardware; stuff such as latency issues and memory footprint. I conjecture (as have others more clever than me) that the reason that we have so much junk software is that most programmers don't know how do design interfaces for black boxes and put them intelligently together (or don't think that they have the time to do so).
Strive to build robust black boxes that have well designed interfaces and implement those blackboxes well and remember: premature optimization is the root of all evil.
I took a couple of CS courses in college as part of my Math major. They were full-blown CS courses, not courses that had been altered for us Math majors. And they were nothing more than problem-solving courses -- and the problems being solved were so utterly asinine that it was laughable. However, when I studied in Germany I took a CS practicum course where we were assigned the task of creating a graphics program in X Windows on SunOS 4. The class was divided into groups: GUI, backend algorithms, SCM, QA, and requirements and management. There were design sessions and reviews, unit and integration testing, etc, etc, etc. It's the closest I'd ever seen to the real world in academia. I've never heard of any American college or university offering such a course, and no one I've interviewed ever had such a course. That's not to say that it's not offered somewhere, but it just doesn't seem all that common. And that's a real shame.
While there's definite benefits of treating software components as "black boxes", I agree with the asker of the question that there are some definite negative side-effects.
For instance, we've got a couple of developers that just don't know how to work with the team, and figure that they can go sit in their dark cubes and code away their component as a black box that with simply fit in with everybody elses stuff. Common problems that arise are:
- Different logging schemes
- Different configuration schemes
- Different admin-alerting mechanisms
- Components that don't match the design pattern that all other components follow - thus making them harder to understand.
- Components that expect some type of "global" data to exist, that simply doesn't.
These issues have led to no end of grief for those of us who do communicate with each other about what they're doing.Abstraction is great!, but you still need to make sure everything fits together correctly, and not just at the interface level.
Two other thoughts:
(1) Maybe software black boxes are too easy to create. That's probably true for just about anything, but consider how easy it is to scrape together a .DLL with a few hasily thought out entrypoints and release it to clients. Maybe API design should be more rigorous. Here's some flamebait: I happen to think the NT Kernel API is very well thought out, whereas the Linux Kernel API is, well, not as much.
(2) Maybe someone shouldn't be "allowed" to use an API until they've, well, sortof taken training on it to understand the subtleties of its use. Maybe just learning "the basics" about an API isn't good enough. How much of software instability could just be explained by a programmer not understanding the function s/he is calling?
Thanks for the reply
Nothing is so smiple that it can't get screwed up.
I was at a company that was so into the 'black box' idea that simple things like annoying bug fixes became imposible.
Basically (for the programming geeks) they set up an error handling policy where (deep in the program execution) it trapped all exceptions including fatal programming errors and sent them up the chain. At each level of the program it caught the exception created a new one and dicarded the old one. So by the time the user actually got an error message it was about some butterfly flapping it's wings in Africa and not about the null pointer you tried to write to.
Anyway this IMHO is an example of 'black boxes' going too far. If you build a well designed architecture and proper object API schemes then you shouldn't run into issue like such. But if you put on the blinders and assume that whatever your doing in your black box is of no concern to the rest of the world then you are destined for doom.
You have to look at what is going on around you to properly deal with things. This goes for real life as well as programming. Far too many drones with blinders who can't see past there own nose are walking around lately.
> SELECT * FROM brain_cells WHERE synaptic_rate > 0
0 row returned
The real issue may rather be that most people tend to think in term of solutions rather than problems.
The prerequisite to the solution is unfortunately the ability to state the problem correctly.
Even when we do so carefully, we usually lack of an accurate awareness of the real need.
Unconscious needs lead to poorly stated problems which then lead at best to inappropriate solutions.
These inappropriate solutions are finally stacked one atop the other and called "layers of abstractions".
CS teachers, students and programmers are just human. Nothing wrong with that provided they all know their biggest weakness.
When I was taught, the black box mentality related only to specific functions/methods. e.g. function DoThingsToX(X,Y,Z) would change only X and have no side effects, or at least no undocumented ones. That was it. But in programming, all we were concerned with was one part of a project - with specifications on what that part should do, and on what other parts would did. Analysis & Design covered the big picture, where how the interactions of the lower parts should be planned. All requirements and interactions of what was to be programmed would be worked out ahead of time.
Of course, this only involves "white paper" (new) projects. In the real world, I have only been involved in two or three of these, and at the risk of inflated my ego they went pretty well. But 97% of my career has been divided between supporting or adding functionality to existing software. In my opinion, this is where a black box mentality can cause problems. I 've regularly run into functions/procedures that didn't work as described, or expected, or weren't tested, or made undocumented assumptions, and so on. However, that didn't stop me from ensuring what I worked on did what it was supposed to, and passed testing, and effectively was a black box in and of itself.
As to the professors that insisted the machine is a black box, I have to say that generally speaking I agree. For what I work on, as long as the hardware is up my only concern should be that memory requests work. The part that I think is confusing is that what was implied was hardware is a black box if you aren't directly concerned with it. If my software is to control a specific network card, then I better know everything there is to know about that network card. If my software just needs to talk through it to a network, then all I need are the protocols and error messages, and it is a black box for how it does what it does.
R: That voice. Where have I heard that voice before? B: In about 365 other episodes. But I don't know who it is either.
No, when something bad happens, it normally highlights smaller problems that all are part of the situation. So, instead of mourning the loss of 7 people who you probabky didn't know, which is futile, get something out of it.
I also don't understand why this is worse than other people who die in air crashes, as soldiers, in cars, or whatever. They've done something with their lives. Get over it.
It's no worse than the Challenger disaster, and certainly no worse than the Apollo fire. Those men died horrible, slow deaths, for no reason.
I'm an embedded systems engineer. I've designed and programmed industrial, medical, consumer, and aerospace gear. I was engineering manager at a contract design house for a while,
and currently employed at a petroleum distributing facility that specializes in distributing directly to small and mid sized gasoline engine supply tanks through unmanned network connected POS terminals. Oh yeah, they also sell beer and smokes too.
Based on the amount of hard time job hunting posts here lately I assumed this was the start of another one...
If you're really an engineer, then you shouldn't have any trouble seeing the big picture.
Unlike, say, managers or interns, Engineers are trained to think through all the consequences of an action.
If you can't predict the effects of your software code on not just the rest of the project, but the economy and society as a whole, then I guess you've been slacking off.
(nobody flame me without reading the cartoon)
O boy is the real world gonna hit you hard in the face.
It is true that most, if not all, abstractions are leaky. But it is still essential to be able to work in "black box" mode to contain complexity when necessary. It is just as important to be able to flip back and forth between levels of "nested black boxes" when necessary. Of course no single person can learn everything, which is why there are specialized developers, and management software engineers. Meaning at higher or lower levels of abstraction can be preserved (ie. abstraction leak prevention) when working from a particular level: To ensure that everything is sound and complete in other levels, you usually have superior ranking software engineers looking over the shoulders of the code monkeys. So, if software fails because of naivete on the part of a particular developer, it's most likely an engineering management and/or software architecture problem. You can't blame a single developer for not knowing everything. You might (and probably should) blame his/her managing engineering for not ensuring that everything fits together at higher levels. Or you might the software analysts/architects for not designing everything to fit together properly. If you ranting against the general usage of abstraction in CS, you are naive. Everything humans know is an abstraction. Computer engineering is an abstraction. Electrical engineering is an abstraction. Biology, chemistry, physics and everything in between are abstractions. Mathematics is perhaps the ultimate abstraction of all. Unless you are suggesting that we all should attain some sort of zen like state where all the semantic levels converge into a giant mass, you cannot escape the "black box mentality". (Trying to suggest that programmers need to code in only machine? Or maybe raw electrical impulses?) Rod
On the one hand, you should be able to look at a computer as a black box. If it's not an operating system, and it's not a driver, you shouldn't have to know what sort of system your code is running on. Portability is a wonderful, wonderful thing. On the other hand, you should always take into account what system your program will run primarily on, and you should always be aware of how the systems under your program probably work so that you don't duplicate functionality, try to out-guess the compiler, or make all sorts of horrendously expensive blocking calls that you don't need to make. I'm an undergrad at Georgia Tech, and I've found that one of the big differences between a solid degree in computer science and a weak one is that the better programs open the black box as much as possible, especially later in the program. Sure, the early classes are taught in pseudocode and java and such, but the farther along the education gets, the more we have to take classes like ECE 2030 (which explains transisters up to CPUs) and Design of Operating Systems (which explains printf down to the CPU). Another big difference is theory and knowledge of design paradigms, from the simple, like hash tables, to the more unusual, like factory classes. It makes a big difference to see the big picture, but then again it's quite possible to write perfectly acceptable code without the slightest idea how the API works. Otherwise nobody could write Windows software. Caching and pipelining and all that stuff is useful to remember, but there's a reason most of it is completely transparent -- so you don't have to know it's there.
A VM is a good thing. A crummy developer can crash the VM, but not the OS. The developers making the VM should be top-notch naturally and work with the best tools/research available. This is why Java on a mobile phone is good, you may crash the VM, but not the underlying OS (ie. the phone).
AFAIK mission critical aerospace software gets written in completely checked code, ie. errors cannot occur unless dealt with, I believe that ADA is such a language?
Unable to read configuration file '/bigassraid/htdig//conf/14229.conf'
Geocrawler error message.
ANYTHING of sufficient size and complexity is by definition something that no one of us can comprehend in its entirety. This being the case, there's no hope of ever seeing to it that everything from minor annoyances to catastrophic failures will be abolished.
My experience with this sort of thing isn't in software, it's large scale construction projects. Launch pads, to be precise. The basic goal is "build something that the Space Shuttle can successfully fly off of on launch day." In the real world, NOBODY knows exactly what this is going to involve down to the finest detail, and the possibility for malinteractions of that detail.
Fortunately, the launch pad, once built, more or less just stays put and continues doing the same job over and over. With software development, no such stability is feasable. We're still learning more and more about how computers work, both hard and soft ware. In this phantasmogoric landscape, with things morphing from this to that with bewildering speed and little overall pattern, the guys who have to grind out the code (and all their bosses right up to the CEO) have no prayer of ever getting it right. Don't be so hard on yourselves, it's a situation that you can NOT control fully. Just do the best you can and let Charles Darwin sort out the mistakes.
Is it fascism yet?
In the lovely concrete blob where I went to college, I think they more or less got it right.
First they teach the basics of programming (variables, environments, scoping, addition, subtraaction, subroutines, etc.) and all of that happy stuff.
Then they teach you how to build a NAND gate using transistors.
Then they teach you how to build a CPU using NAND gates.
Then they teach you how to call subroutines using RAM and registers and pointers and that CPU you just built.
Then they go back to the basics of programming, now that you have a full appreciation for everything that happens when you add two numbers together, and proceed from there into the nether regions of algorithmia.
To me it seems to be a good approach -- it gives you the basic destination first, then it gives you the foundations to get there, and then it goes onwards. It means you've been exposed to every level of what you're working with (to a point -- we never doped silicon to build NAND gates!), so you know what's down there, and you also see the importance of ignoring what's down there when it doesn't matter.
Well, the previous shuttle accident only involved a bunch of teachers. They deserved what they got, what for torturing poor, helpless school children with their homework and their quizzes day in and day out. This one involved real, honest-to-god astronauts, and thus was a true tragedy.
Outa school for 20+ years but was always taught the black box. The courses I've taken since have all taken the black box approach.
One of the priemere embedded systems languages, Forth was invented by Chuck Moore. I like Chuck Moore's 1% Code Page. His introduction:
Seastead this.
I'm an embedded systems engineer.
Really? What are you embedded in?
Just hold tight. We'll use some Practical Extraction and Reporting Language and have you out of there in no time.
- Hail to our fearless misleader! Fool speed ahead!
If you're doing a one-off development, and everything is going to be for that specific purpose and no other use, sure, Big Picture stuff can be useful.
But if you want to make reuseable code that you can then take on to your next project... the Big Picture is a hindrance, not a help.
Most people tend to do one or the other, and not both. Writing good reuseable code *requires* abstracting away details that would otherwise get in the way of writing modules/objects that are general purpose.
As with so many things, this is a balancing act, and you get better with experience. You get better at asking about constraints, flexibility, limitations, interface requirements, error handling, and all the other things you have to keep in mind when writing code you will be proud to admit you wrote 5 years down the line.
Don't expect to leave college knowing everything you need to know for the rest of your professional life. Continue learning and picking up skills, and you'll justify those fat raises and bonuses you want.
This is my sig. There are many like it but this one is... Oops. Frank, I've got your sig again! Where's mine?
web development as an example?
What the hell is the point of supportable software? How the hell do you plan on keeping your job when the next guy (who's in India and $5/hour instead of $55/hour) uses your "excellently programmed code" to work yourself out of a job?
Seriously, screw the next guy. Just follow your companies procedures (no more no less) and move on your merry way. I've never met a software engineer that got paid less than a "support technician". The difference is if the support person could do the programmers job, they would be (I was a support person initially while trying to get the necessary experience to become a good developer).
Your first few jobs usually pay shit because you know shit. Once you've been in a couple of situations you can learn from you become the 3-4 year experienced developer most jobs are looking for (not too expensive, but knows his/her shit).
I make over 100k in Oklahoma and it's no coincidence my first 2 jobs were in Dallas Texas. I used Dallas to get a better job (there are almost zero entry-level positions in Oklahoma) where I really wanted to work. No matter how much schooling/self-learning you do there is no equal to real world experience and making someone elses job easier isn't the domain of the programmer, it's the IT manager/CIO/Team leader's job to set the goals and timetable of the project(s). In my case I'm the project manager and one of the developers so I make damn sure our specs, design docs, methodologies are followed precisely (since I've got to mop up the messes).
How does a team insure that the first few thousand lines of code, the first board, etc, fit neatly within the system as a whole once it is completed? The best we can currently do is follow some established principles of design and implementation.
Someone said it earlier: Have some knowledge of the entire system, but write code in the black box principle. It is the only way to minimize maintenance of one module if a change is made in another module. Eg, by all means, understand how a module works behind the API, but when coding as a client of the API, you must treat it as a black box, unless of course you enjoy tedious maintenance.
It is essential the programmer or at least the designer know about the entire system or the program code could do something completely inappropriate when handling an exceptional condition. Eg, exit(1) is likely a bad choice for an aeronautical system.
On the other hand, it is entirely impossible for a designer or programmer to keep an entire system of any complexity in their head at one time. Hence, it is necessary to modularize. And for the robustness of the system as a whole, the modules need to be loosely coupled: if one should fail, all others should recover gracefully.
Regarding GUIs, I must disagree. It is a different beast. The complexity comes from dealing with the loss of control flow and ambiguities brought on by different execution order (I clicked there, then here and then the whole screen turned blue). Backend code should be as far removed as possible from presentation code. In a windowed system, the programmer should code for event driven programming only. If the backend makes assumptions about the GUI, then any GUI changes will mean a rewrite of the backend.
If you black box the backend, then the GUI can select from the backend as required, and the chance of an exceptional condition is minimized.
BTW, it probably no consolation at all, but none of my university prof would have considered suggesting that a programmer needs to know nothing of the underlying hardware. There was a great posting on Leaky Abstractions that makes a great case for understanding the implementation of a module.
This only works to a point, and the fundamental problem as others have pointed out is that the 'black boxes' are almost never specified to a level of detail adequate to fully describe their behavior. Things like side effects, performance criteria/guarantees, behavior on edge conditions etc are very frequently overlooked. Your statements are idealized versions that might have been taken from some software Methodology book - they completely ignore the real-world problems like bugs, partial/incomplete implementations, outdated specs/documentation and a whole host of others.
This doesn't mean that you always have to be explicitly focused on these issues, but the overall success of the system as a whole can be critically dependent on them. In a perfect world, of course, this wouldn't be an issue. But assuming the viability of black-box treatment in most real-world projects is the source of many problems, and the truth is that a relatively small portion of the population is capable of maintaining a sufficiently broad view of a system to be able to effectively respond..
There are many, many problems that can answer the question you have posed. Most have to do with improper resource allocation and/or improper analysis (which can, itself, be considered improper resource allocation).
If a project doesn't understand each of the components involved in the overall system, then that is improper analysis (or improper communication of the analysis, or improper interpretation of the communication).
However, even as an "embedded engineer", you do not need to understand every single bit and byte from power source to CPU to each sub-system behind its interface. If you did, then there's no point in having interfaces! No point in designing sub-systems...
I do agree that many (all?) hardware/software systems out there have deficiencies. All of these deficiencies could be overcome by pouring the right amount of the right resources at them. However, everything has its limits and the final GA'ed product is an acceptance of those limits (costs, timelines, resource availabilities, politics, accountabilities, etc...)
Yes, many engineers come up short on stability and/or usability in their products. Sometimes this is due to their own limitations, sometimes it is due to limitations imposed on them by others.
by Joel Spolsky, describes exactly this problem.
t ra ctions.html
http://www.joelonsoftware.com/articles/LeakyAbs
you probably forgot one...
:)
10. Manager
The human mind can only consider so much at once. Using black boxes ALLOWS one to view the larger picture.
In my experience, the single biggest factor keeping engineers from considering the bigger picture in their designs is management. The moment an engineer tries thinking outside the box that management has assigned them, management starts worrying about them upsetting the project schedules, or that thinks that the engineer is wasting time that could be better spent on their little black box. After all, they hire system analysts to worry about the overall picture - why would anyone working on a component have any real insight into the big picture?
For example, in electrical engineering, it is common to simplify complex circuits by breaking them into smaller circuits, analyzing each of the smaller circuits, and then replacing the smaller circuits with appropriate black-box equivalents. With these black boxes in place, the original circuit becomes much easier to understand.
However, as in any modeling exercise, it is crucial to choose appropriate models and to understand the limitations of the models chosen. While a simple resistance model might be a good substitute for DC and low-frequency circuits, it would be inappropriate as a substitute for higher-frequency circuits where capacitance effects come into play.
So, returning to the original poster's questions:
Yes, in these days too much stock is placed in the idea of letting somebody else worry about the complexity. This is especially so in mainstream industry, where one of the key selling points of software development systems is that with Magic DevStationPro X you no longer have to worry about the details but instead just use some brilliant Wizard or API to work at "a higher level." This applies not only to software development but also to user-level domains such as operating systems and applications. For example, a common notion in industry is that by using Microsoft operating systems on servers, administrators no longer need to know how to administer servers; rather, they need only know how to use the GUI administration tools. In other words, the pitch is that you need not concern yourself that the GUI tools present a mere model of the underlying system. Let the model be the system and reap the rewards.That's hogwash. The model approximates the system, no more. In engineering, this is well understood, and I suspect that in good CS programs the same can be said.
Abstraction is a powerful tool. It is widely applicable, effective, and well founded -- when used appropriately. It probably ought to be used more often. Nevertheless, it is not a substitute for rational thought. Nor is it a replacement for being responsible for the entirety of the systems we build.
It would certainly explain some of these problems, but I suspect that far more errors are the result of interface errors. One of the tools that goes with abstraction is composition -- breaking things into pieces, treating the pieces individually (at fine granularity), and combining the pieces (at a larger granularity) to yield a system. The risk of combining pieces is that in order to put them together properly, the boundaries where they coincide -- their interfaces -- must be well understood and compatible. Since the individual pieces make natural units for delegation, pieces are often assigned to different people who may have slightly differing understandings of the boundary conditions. As a result, "interface mismatches" are a significant source of error in software systems.Certainly abstraction plays a role here. Each piece can be thought of as a black-box model. The limitations of that model, and the assumptions under which the model is valid, are certainly important characteristics of the piece's interface with the world. Yet, these characteristics are frequently neglected in documentation and often go uncommunicated across delegation boundaries. This sad fact makes interface mismatches an especially harmful side effect of using abstraction and composition in common software development practice.
Nevertheless, abstraction is a powerful and genuinely useful tool. It is also a necessary tool if we are to build increasingly complex systems. Like any tool, its uses and limitations must be understood if it is to be applied effectively. Thus, getting back to the original poster's question about whether the use of black boxes is harmful, my answer is, No.
The problem isn't abstraction, the problem is improper use of abstraction.
Easy, automatic testing for Perl.
This is the number one problem I have with most of my programming courses. I think the black box issue is a side-effect of the real problem; trying to over-simplify the programming process into a bunch of simple (and often trivial) examples. I don't actually disagree with this from an educational standpoint, except that it is taken too far.
How many computer science depts have you work on a major project(s) over the course of your degree? How many, instead, have you implement algorithms to walk a chess board without hitting any space twice, or teach you how to draw on a DOS window? =) Do you really expect these students to just walk into a real company and start being productive? Many will be almost totally unprepared.
While these skills (and the practice) are certainly not without value, these projects do not teach skills like integrating software components, teamwork, or how to validate incoming/outgoing data. I personally do these things much more often than having to figure out 'tricky algorithms'. In fact, I've found that those trivial examples tend to get students worked up on some small aspect of the algorithm, totally losing sight of what they need to be learning.
Some people think that computer science programs need to focus on abstraction (and I used to think this at first), but I in fact feel that they already focus too much on abstraction. Yes, abstraction of a problem is an ESSENTIAL skill for programmers, but let's not forget that one learns to abstract by noticing patterns in various examples over time. In other words, by performing a concrete task over and over, you notice patterns in how you performed the task and are able to generalize it to other problems. So, if you have someone program over and over and over, they will (in most cases) begin to realize patterns without a book or teacher to help them out. It's how I learned, and I bet since others are asking for abstraction to be taught more, it's how they learned too. =)
I really think the best way to teach programming is by mentoring, and an open source project is a great way to do that. Have them first take basic computer courses, and view (but not contribute to) the project and its code. Maybe even explain to them portions of the code in-class, focusing on why certain decisions were made. Then, ask them to make a few simple modifications, and submit them to project maintainers (which could be professors). The maintainers explain what is good/bad (i.e. security issues, compatibility issues, etc.) and the sample is sent back for revision. Then just step up the complexity of the modifications little by little.
This model just doesn't really fit with our current education system, unfortunately. =) In any case, I think the computer science programs in effect do need some major updating. Some of them have learned to be more modern, but I find I do more reading than programming, and when I do program oft times I'm not sure what the point of the example is.
I'm a comp. sci. student, and I'm constantly amazed that nearly all my fellow students find it boring to learn *anything* about how an operating system or low-level hardware works.
IMO, to have an imformed opinion about how your program works, regardless of the black box your coding for - be it a JVM or whatever, you need to know something general about how it works low-level.
I don't think that the black box way of thinking is the problem. I use black box programming often. But I'm also the one creating the black box code, so I know what it's doing and what to check if anything does go wrong. Having the source code helps alot. That's the drawback to third party software. Sure, it works. It might even work well. But one bug in there and you're stuck waiting for some pimple-faced youth to get up off his ass and implement the changes you've been emailing him about for the past 3 frickin' months.
This is exactly the argument the ABAP developers down the hall have been searching for! Down with OOP!
All your base are belong to us!
As a creaky old Fortran programmer, I vividly remember taking code and rewriting it to run efficiently on other systems. If you didn't know how the compiler "unrolled" multi-dimensional arrays in memory, you couldn't write an efficient routine that walked that array. Sometimes you could reduce execution time by over 95%, just by rearranging the order of the indices in nested loops. So even in Ancient Tymes it was critical for the programmer in the "high-level" language to understand what the "lower-level" components (in this case the compiler and the platform hardware) were doing. Think of what a "black box" is: For a specified input it must generate specific output. However we can't seem to create ANYTHING without bugs or "special case" conditions where the system returns unexpected results. So, just as the Olde Pharte Fortran programmer had to know what the lower-level components are doing to write efficient code, anyone who uses any kind of "black box" mustn't think of that system as a "black box" at all: They MUST be aware of the low-level behaviour of all the systems they interact with and how to deal with that behaviour. JS
Sometimes the "writing on the wall" is blood spatter...
Hello??? An aircraft's "Black box" isn't an interface to any of its control system. Rather, it simply LOGS data from as many systems as possible. It's called a black box because data goes in, but it doesn't come out.
In the event of a crash, they analyze the data recorded by the 'black box' to find out what went wrong. This is in no way related to the software-engineering concept of a "black box interface".
There's nothing to see here. Move along.
what things safely can be put into the black box, and when.
Later, a project might come along where something in the black box should be taken out, exposed and given attention.
It behooves you to learn about everything on your project as much as time permits so you can help define your own black boxes, because the people around you and in charge of you have only an inkling of what is assumable and what is not.
"Provided by the management for your protection."
Another elitest post without a real clue. A good programmer knows how to get a job done and should ALWAYS have a big picture view of how things work around them. It does not matter whether they are working on a web site or writing a backend database app or a game engine for the latest and greatest game. Somebody writing good PHP code could probably write good backend C++ code. You are associating the tasks people do with how capable they are. Languages and programs are TOOLS and a programmer should be able to quickly learn to use new tools whether it is a new language, interacting with a new API or using a performance profiler. A good programmer really should not care HOW they get things done- ONLY that they DO get them done.
I miss the Karma Whores.
I work for a rather large microelectronic firm. An engineer in our group went for a rotation in a circuit design team, and came back AMAZED at how many people on his team had absolutely NO IDEA what the little part of the circuit that the worked on was supposed to do. They just designed it to take input X and crank out an output Y. Personally, I'd have to know the bigger picture, but some people/engineers/coders just don't care to find out. They do their job, do it well (or not) and go home.
Programming Levels:
1. Big Picture Architect
This says it all.
As for a real culprit for our persistent software security and safety woes, try the following:
As a manager, probably not what you wanted to hear, though.
Like the mars lander, the challenger, the columbia... yeah if only the aerospace industry made cars - they could get 2mpg, cost a trillion bucks and run twice a year.
Hardware fails sometimes, through no fault of humans.
Ahh, but who designed and manufactured the hardware?
To recurse one level further: Deep Thought, perhaps?
It looks like you're trying to fly a craft with only one wing. Would you like to:
* Plummet to the ground
* Stick an arm out the window and flap vigorously
* Visit the Morton Thikol technical support website?
This would be dependent on the scope of the overall project, and the development methodology. In Structured programming I think it would be prudent to be mindful of the overall picture of the project as opposed to the "walls and windows" approach of OOAD (Object Oriented Analysis and Design). On a smaller project, of course you can be aware of the overall project, after all theres not much to look at, but on a large scale project its almost impossible to do that, thats why there are development teams assigned to a particular phase or portion of a project. I dont think the "black box" approach is the cause of poor security as much as poor programming. Its been a while since Ive read up on it, but Ive always been under the impression that the biggest problems in security and bugs are poor pointer handling and use of non-bounds checking string routines such as strcpy() as opposed to strncpy(). If theres security issues raised from the "black box" mentality, its due to poor management, design, the methodology, however if a developer is aware of the overall pictuer and can in some way increase security with that knowledge, I dont think it would be frowned upon.
The blackbox approach is always best if the system will allow the abstractions to do it; if not, then you gotta get down to the metal. Either way, it is still programming. There are good ones and there are bad ones.
Somewhere in the world there is the worst working programmer and he is probably writing code as we speak.
Jamey Kirby
Many programmers have strong opinion one way or the other about black box programming. However, the fact is that you are _always_ developing in a black box environment.
What I mean is, there is always some point at which you have to develop within a certain set of defined interfaces with some other system. If that happens to be the ANSI C standard on one end, then that's fine.
You don't need to develop in or out of the box, just ensure that you know where the boundaries of that box are.
I like your post, but would prefer the phrasing "who limits their thinking to a black box mentality..."
Like many of the posts on this thread, you emphasize the importance of an overall system view. However, there is also a need to abstract out concepts in any complex system (as the rest of the posts on this thread point out).
I know I'm preaching to the choir here, but the key is to figure out where to draw the line. How much abstraction is "enough"? How much detail is "too much"? While there's a lot of proven guidelines and methodologies out there, the fact is that each project has to draw that line for themselves.
And that, dear friends, is where the art lies in engineering.
P.S. This, of course, presumes that project management supports the chosen level of abstraction with policy (e.g. enforcing code reviews) and resources (having enough heads on board so the abstrations don't leave developers isolated - you need overlap). Without that support, you're straight out doomed despite any sound engineering planning.
"Prepare for the worst - hope for the best."
I have one rule: For every "if" you write, write the "else" too! :) It sounds trivial, but _this_ is the wisdom of the Programming Gods.
Most undergrad programs put out students that are really ill prepared to start working as professional commercial coders. Unfortunately, most programs stress theory more than practicality. Students graduating have no idea how to write system designs (some don't even know what it is), few know how to unit test, or to participate in design discussions. I remember a good prof I had tell us that something like 75% of the time we would spend working as programmers would be doing something else than programming. While I'm not sure about that 75% after working for 7 years out of school I do agree that a lot of my time is spent doing things I hadn't planned on or knew about. I've worked on systems that you simply cannot get a big picture understanding of it until you've spent at least a year working on it, mostly my experience was interfacing with a complex MF. Abstraction to a certain point is neccessary, few people other than a system architect really have a big picture understanding of every portion of the system. As a developer I think it's important to understand what you are developing on a low level and understand how it impacts the rest of the system at a higher level, which in itself takes time. This is just ranting here, I've mentored a few interns in the last two years who have had varying coding abilities. One in particular from a big name school knew all the latest languages and such but unfortunately wrote code that was uncommented (or worthless comments), hard to read, and in general acted almost as spaghetti code. He was very good in spouting buzzwords of course which is all management really cares about. Shortly after that I mentored someone from a community college who wrote excellent code that was at a professional level but was very quiet and withdrawn. I don't think any undergrad program will be able to graduate a professional developer as the on the job experience is really the second learning stage of being a developer. Undergrad programs try to prepare students for careers, not give it to them, so in that regard I don't mind if they don't get the big picture. I figure they don't know a thing anyhow so expectations for them to understand a complete system architecture are fairly limited.
In this time I learned that teaching is all about limiting complexity. Thats a mantra guys; Limit complexity!
The blackbox idea is a very good way to get there; it has (alas) been mistaken as the only solution.
Successful programmers learn to make a system that contains just enough complexity to do exactly what the problem demands.
The most common mistakes are:
- Designing for future use by using expensive design patterns when a simple solution would work just fine.
- Creating one system that is too big to oversee, instead of creating a number of smaller systems that are seperate black boxes (thus limiting complexity)
- Guessing about an APIs usage instead of reading the API docs.
- When a bug occurs in the system people just rewrite the method since understanding the code allready there is hard. (learn to read code!)
Desiging a system should be about connecting black boxes, and basically that is what ObjectOrientation was designed for..riiiiight
tell that to my current employer?
[I can picture a world without war, without hate. I can picture us attacking that world, because they'd never expect it]
The problem, of course, is that both sides are absolutely correct. I know I'm preaching to the choir here, but the key is to figure out where to draw the line. How much abstraction is "enough"? How much detail is "too much"? While there's a lot of proven guidelines and methodologies out there, the fact is that each project has to draw that line for themselves.
And that, IMHO, is where the art lies in engineering. You need a lot of expensive investigation and analysis by very experienced, talented people before they can get to the point where they can sit in a room and say "I have a hunch this is where we should draw that line." Even with all the science to back it up, it is still a very intangible thing. In the military, Von Clausewitz called it "the Fog of War." Everyone else calls it "Real Life." You can minimize uncertainty, but you can never eliminate it.
P.S. All this, of course, presumes that project management supports the chosen level of abstraction with policy (e.g. enforcing code reviews) and resources (having enough heads on board so the abstractions don't leave developers isolated - you need overlap). Without that support, it's like marching off to battle without ammunition, maps, or boots - no amount of strategy will save you.
"Prepare for the worst - hope for the best."
Black box mentality is actually a very good thing - to an extent. The trouble with most programmers who adopt this mentality is that they fail to understand that the black box has to harmonize with the design of the system as a whole.
One example that occurs to me is this guy who wrote a component to handle mail processing requests. Before we had a chance to work out the interfaces and semantics, he had hammered out the implemenation he chose and took that stance that it was a black box and thus code must conform to its design. Unfortunately, he missed some functionality and also used the entirely wrong programming model (we wanted a synchronous call interface, he made everything async job/queue).
"Deal with it" is the mantra of those who take this concept too far. The "Black box" is the protection of the *implementation* and not the *design*.
Not only errors: black box routines can be expensive on performance too. Take database programming as an example: black box programming teaches us that we must break the problem down into its smallest (easily) solveable (and reusable) parts, create a routine for each one and then work our way up. In a program that does disk reads to solve the problem this could mean many, many more reads from disk than what is necessary, if you're not careful.
IMHO part of what makes a good programmer an even better programmer, is to know which routines to black box, taking into consideration performance and resource availability, and which ones not to.
High level languages are no less than a library of black box routines. And there are often parts of a program that are best written at the lowest level, for efficiency's sake.
A good programmer will constanly be weighing up the pros and cons of his or her methodology in order to provide a system that is sufficiently practical for the underlying architechture while taking into consideration all the other constraints of the project such as budget and deadline.
Did you do you're embedded programming in C? Or Forth? I'm thinking C because you seem to put as an 'either or. ' I think the difference is how much you, have to do, yourself, to get to something the resembles a 'virtual machine.' More, "Who provides the virtual machine?" than "Is it a virtual machine?"
/have to/ deal with
One thing to remember is that the only thing some programmers learn from school
is how to misuse elements of CS to rationalize away the fact that they suck.
IGNORE sophistical arguments, instead of buying into the BS and getting the wrong idea
about whatever they're using as an excuse.
It sounds like you've heard 'blackbox coding' where I've heard 'implementation details'.
Like:
Me: What if?---
Them: I'm not worring about 'implementation details' [with a tone that suggest that they are above it all]
But for you, perhaps, it was like:
You: What happens when?
Them: We don' have to worry about that because we're coding to a virtual machine.
You: Yeah I know, but ---
Them: Haven't you taken object orientied programming and design?
You: Yeah
Them: Ah then you see... [start long winded lection about OO that isn't germane ].
There are two types of people: People who don't know everything, and people who don't admit that they don't know everything. If we had been dealing with the former. It would have gone like:
Us: What about?
Them: Well, if that happens we're fscked. That's something we'd
if we were doing embedded systems programming like in medical equipment, but for now for the purposes of CS201 we implement the 'ostrich algorithm'.
I studied Computer Science in College, and currently work as a Programmer/Analyst for a non-profit organization (Desktop, Web and Server-based systems). Yes, all of the above encourage a "black box" model to design and coding. Furthermore, I am guilty of perpetuating this to the people forced to listen to me blather, and will continue to do so until I see a better way.
I understand that it hides some bugs. I don't like this. On the other hand, we can never have enough staff to make sure we have people expert on not only each system used but on each interface between the system to do a good integrated system.
So what we do is take some premade components (eg. hardware, OS kernel, C library, certain widget libraries, web server, etc), and say "OK, assume these work according to these specifications, we're going to work on adding a piece that does this". When the premade component deviates from the specifications, we fix the component or update the specs.
As much as possible, we make use of open standards and free software so that if we need to, we can open up the black box and fix something. However, the more we can assume that a component is a black box that will just do what it's supposed to do, the faster we can develop the "interesting" bits.
The bottom line for us is to manage complexity. The more complexity that we can abstract away, the faster we can work on the custom stuff unique to our organization. A "black box" model works well for us, but yes, it does cause some bugs that need to get cleaned up after the fact. Most organizations I've seen make a similar design choice (or blunder into it blindly), and most schools teach their courses with a similar mindset.
If we were to develop a truly critical system, one that lives or big bucks depended on, we ought to take a different approach for that system, but we aren't likely to work on such a system for a while.
----
Open mind, insert foot.
The issue at hand is the interpreted disregard programmers have for hardware. This is largely created due the the enormous amount of hardware abstraction present in most modern operating systems. Windows developers couldt give to craps about hardware they are not specificially coding for thanks to the enormous volume of API and library calls available to provide a seamless interface for them.
To paraphrase; It's unfair to brand "black box" brand software engineers with some engineers' stigma because they dont code for, not care to code for each specific piece of hardware they may come in contact with.
I work for Software company that builds Point of Sale Systems, and as such I have have the opportunity to talk to both hardware and software interfaces, such as credit card units, pinpads, MSR readers, receipt printers, etc. Some are handled with Windows APIs, making it transparent what specific piece of hardware is on the other end of my commands.
The reason Black Boxes of any type, but particularly software black boxes exist is to modularize code. OOP centers on the idea that you can take a chunk of code, wrapper it with an API and make it a component someone else can slap into their program to save them time and effort.
Maybe purists will insist that you need to write your nlogn search algorithms from scratch each and every time you use them, but in production, and particularly consumer grade systems programming modularlizing code, using (gasp)ActiveX controls, and other such "Black Box" techniques save untold time and money. Programmers who hit deadlines are programmers who keep their jobs and maybe even get raises.
IMHO
Some managers think that you are doing nothing unless you can say, yup, wrote 3000 line of code today. They don't really seem to care that you have just created a mantenance problem.
I was just at a job interview where my comment that I have done several projects in which I mentioned that I have replced around 5000 lines of code with 500. Due to his obsession with lines of code, this got missinterpeted as, I have only done mantenance.
Note: the 500 line of code did the more than the 5000, with fewer bugs and was alot faster (ie, 1 scan though the data as opposed to multiple).
Every job that I have had, I have found areas of code bloat either done due to the pressures of meeting a deadline or though incompentence. In either case, the best thing to do is to just clean them up and go forward.
after readin da artikal I would say that IBM drops Itanic instead of Linux..
Linux sux anyway..
It is simply not possible to keep the entire system in mind in any way but by considering as a set of components with no characteristics externally (i.e., as "black box"). Anything else is incomprehensible beyond relatively small systems.
I think therefore I mess up the design.
I see preliminary designs for databases of objects that magically exist in pure object-land (i.e., they don't actually do anything) and yet somehow the work gets done.
By training and disposition, whenever I don't smell silicon, I become deeply suspicious, so my first reaction is that such designs are nonsense. Perhaps it will not always be this way -- for instance, perhaps the designers of those very systems will get around to saying who actually does something and how they do it.
But I've grown to realize that I must accept a certain amount of nonsense (subject always to good engineering judgment and a demonstration that some of these fanciful schemes can actually work) because the "how" absolutely must not enter into the design.
If I have to say to someone writing the software for communicating between commanders and various kinds of "things" (I'm going to apply some severe declassification here) that to talk to a big orange truck you have to stick a 32-bit word into a mailbox interrupt register at such-and-such and address, while to talk to a little red truck you have to send "HELLO, WORLD!" to port 80, they're going to say to me, "Just what the hell have you been doing in your software architecture group for the past six months?"
This is a gross example -- but the less obvious examples are nearly as bad, from my point of view.
For instance, since one of the requirements for this SoS is that communications not be of the form, "Let's tell the enemy what we're going to do", and since communications security is best done by people who know what they're doing, we will not train every engineer to manage communications security everywhere in his application, but rather layer the architecture so that, to the greatest extent possible, engineers will not even know it's happening.
Indeed, I expect the architecture our team develops to survive several iterations of "how"s. The first implementation better not work as well as the final implementation, or somebody's wasting money.
In short, we'll use elementary principles of engineering in order to define common objects that communicate with one another in precisely defined ways at a level of abstraction that's appropriate for the objects themselves. That some objects will have precise real-world counterparts (e.g., big orange truck) is merely evidence that the architecture is sane. And if some of those objects have functions associated with them, that's because in the real world functions aren't performed by spirits and demons, but by (now let's not always see the same hands) objects!
This ain't rocket science, people. If you've written an API that you can't jack up, haul out the Yugo that's underneath, and replace it with a Viper with no one the wiser except the customer who appreciates how fast he's going, you've screwed up. You've let the "how" creep into your "what".
(Hoping some people will return my phone calls and answer their email so I can stop talking about this and get back to doing it).
Yesterday I had to learn that a small utility I make (www.feyrer.de/g4u) is used (among others) by a company who is involved with the production of weapons of mass destruction of a very trigger-happy nation. :-(
- Hubert
I think the big picture belongs in the domain of the program manager and system engineer. Then the ability of the individual developer to deliver a correct product or implement a subsystem properly depend on passing down complete documentation of system requirements. It also doesn't hurt to include the customer in detailed design reviews. Without a "black box", you wouldn't be able to contract anything out!
I recommend reading MIL-STD-498, it gives a great overview of the type of documentation required by government contractors. I think IEEE 1074 is more up-to-date, though I haven't read it. If you're familiar with some of these life cycles, then you won't be shocked as much by the extensive documentation you'll be maintaining when you work for your first defense company.
After reading many of these posts, I think were getting a bit confused here. In part, the original poster is a bit confused, or at least imprecise in his wording.
When coding a module or piece of program for use in a wider system, such as a library or module, black box thinking is a good thing. In this context, a black box means something that does not expose its internals, and provides an abstraction to the user/programmer.
The STL or the standard c library is a black box. TCPIP and the sockets API is a black box. the java standard class library is a black box. If you were required to know all the interal details of all these systems to be able to use them, you wouldnt get very far.
Abstractions, which is what we are talkign about when we use the term black box as above, are absolutely required to write decent software. You simply cant reliably keep that many details inplay at once and expect to get it to work right.
As useful as the above black box libraries are, it is very possible to create a library that is unusable. Making the user of the library know all the impelentation details from inside is one way to do this, but the other is to use the wrong abstractions, make the wrong assuptions about usage, etc.
In short, what the original poster and many others are complaining about when they say they dont like black boxes is that they are using bad or incompatible boxes!
Having to fight with a library that makes assuptions that are invalid, provides the wrong level of abstraction, or is not implemented very well is not an indictment of black boxes, but of black boxes that suck. The fact that their internals are hidden is not the problem. the fact that their interface to these internals sucks is the problem.
So dont confuse the problems of using bad black boxes with a fundamental problem with black boxes. A bad impelemtation doesnt mean the concept is bad. A poorly designed system well may provide a black box that no one can use, or at least not in all situations. but making that box white, and exposing all its internals, isnt the solution. designing the module to work right is.
Further, one persons perfect abstraction is anothers miserable pile of junk. Just because the black box doesnt provide the interface you need/want doesnt mean it sucks, it could just mean your using the wrong tool for the job. Complaining about how hard threads make it to hammer in a screw doesnt make much sense. Use a nail.
I've always beleived that a proper system, from an oo perspective, has a top-down hierarchy with no links going upwards. That is never possible, but if you have tools T that are used by users U, and U are used by Senior users S, then T will never use U, and U will never use S. Heck, it works in companies...
Now I know that sounds elementary and naive, but I still beleive that if you are making [too many] links up, it might be possible to re-work your design.
-P
Mines Beige
The problem goes deeper than the original post suggests - The type of mind which is attracted to programming, and is good at it, is already entrenched in methods of decomposition and logical seperation. I don't think one could *be* a good low-level programmer without this sort of thinking. It isn't just methods of education - I think it's a fundamental dichotomy built into the subject matter of designing and building complex systems.
Is this a problem with using the "black box" mentality, or a failure to do defensive design and coding?
If your programmers code and design defensively, even the black box approach will succeed. No input can every truly be guaranteed to follow a certain spec. No user will always act like you expect. No system will ever be setup exactly like you planned.
Keep these things in mind, and your project will be robust!
An online Starcraft RPG? Only at
Online Starcraft RPG? At
Dietary fiber is like asynchronous IO-- Non-blocking!
The idea behind the black box approach in programming is to ensure the stability of separate modules. The rest of the system is irrelevant. That individual module has certain properties which are characterised during the design phase. Once the coding is done, the module is tested, not as part of the program, but against a test harness which tests the correctness of the module.
A programmer decides how they will code the module the best way they know. They do not change the specifications of the design. If design and code are not mutually exclusive, the program is sure to fail.
There is no 'need to know' how other modules are coded, only that they can be used exactly how the specifications say.
Well, are we talking about computer science or about software engineering. To my thinking, these are about as similar as physics and electrical engineering.
Systems Engineering is really just a superset of Software Engineering, but in the end both are primarily concerned with how *all* of the bits and pieces fit together properly, not on the details. The systems engineers are the ringleaders of the other engineers, they don't always know how to make each piece work, as they focus on making sure the completed pieces work together as they should.
Think of them more like Civil Engineers. You don't see any Civil Engineers out there pouring concrete or hammering a timber structure together, but they make sure that the tradesmen that do these things do them to the right specs, lest you end up with this.
Truth be told, there are entirely too few Systems Engineering and Software Engineering programs out there considering the demand for them.
Aren't software teams all about role playing? The architect or senior members of the team will have a system-wide view.
But I think it's fair for new starters on a team (either for the company or graduate engineers) to be given tasks where they shouldn't have to think too much "outside the box".
In some cases this is beyond their experience or ability at that point.
Anyway, I think that the issue of the GUI is a great example. Programmers got carried away with the GUI, and now applications and OSes are completely over-GUIed. The mouse is much, much slower than they keyboard when it comes to many tasks. I use graphic design programs on a regular basis, and I would give an arm and a leg to have a quick and easy command line interface in, say, Adobe Illustrator, for precise object manipulation. Same goes for Photoshop. AutoCAD and other programs have a decent implementation of the CLI, but it could get much better.
I would love to see programmers get out of the object-oriented point-and-click mode that they've been stuck in since the invention of the original Macintosh.
GUIs are great for representing data, and they are great for the visual manipulation of data. But visual manipulation is often imprecise. For precise data manipulation, the CLI is still necessary -- clicking through a menu and two dialog boxes to finally find a text box with the field to rotate an object by 20 degrees, or add a 2nd column to the page, or fix page margins; that's absolutely ludicrous. There should be a simple, (preferably standardized) command line that's accessible from all applications. Remember the ~ in the original Quake? That was a huge step forward. We need it in more applications. How much productivity has been lost by over-mousing? -Shylock0
Questions and comments welcome. Flames ignored. Post responsibly.
Statistically speaking, there's a 99.998% chance that my IQ is higher than yours. Get over it.
As one of those people who teaches programming, I think you're missing the point. The purpose of higher level languages is to abstract away the hardware and allow the software developer to create the abstract models of the real world without considering the underlying computer system. That may not apply in all case and maybe not in embeded systems. If anything there is still too much hardware involved, things like pointers and memory alocation should not be needed in a begining programming language. After students get some expertice then they can start conserning themselves with hardware issues.
Anyway, you seem to have learned something inspite of all those bad teachers. Have you considerd coming back to school and helping us out. We sure could use the help, were not proud, if you know a better way to teach, come and show us. We're willing to learn.
Not to forget the victims of some intervention in Palestine and the looming invasion of Irak.
Thanks.
Have the compiler add in runtime bounds checking for every type whenever an arg is passed. Severely spank anyone who tries to bypass it. Fuck optimization.
Maybe I'm a bit dumb, but happen to be one of those you address.
Could you specify some real example or a site for what you mean?
I had been under the impression that larger programs not only force to use 'black boxes', but encourage and thrive on 'black boxes'.
Isn't a subroutine or a class a very handy, useful and necessary 'black box'?
Do you want to tell me that embedded something (Windoze, Linux, etc.) can do without API or that you can follow your codes without, straight down from a kernel through all the drivers to the applications? Including a thorough understanding of what your ARM does internally and if I understand your post right, this is what you aspire.
Since this 'know-it-all' is impossible, should we not rather encourage *robust* 'black boxes??
*This* is actually something missed out by students (in my experience). They don't create 'Swiss-Army-Knife boxes' that could perform the task in any circumstances, environments and handle insane input in a sane way.
As far as the embedded security-box of Columbia is concerned, we seem to deal with a hard-coded encryption. At least this is what they want to convince us about. Encryption hard-coded in a single place; otherwise not such an urgent need to recover it. Any other encryption would make it easy to change the distributed keys, which leads us back to the necessity of 'robust' boxes.
Hope I got you right !?
First let me say that the overwhelming percentage of security problems are a result of admins not patching their system. Virtually all machines today used a RISC (as opposed to CISC) design because we realized that 90% of execution time is spent on only 10% of code. I don't understand why this same realization has not been made about security holes.
That being said, I certainly do believe that software is less secure because programmers don't completly understand all aspects of all systems. I also believe highways are less safe because not all drivers understand the physics of a vehicle in motion.
...is the view without the windows.
Strikes again!?
Keep your packets off my GNU/Girlfriend!
Relying on some function, class, library, component, what-have-you as a black box can reduce performance. It helps to have a little grey knowledge of how that function or class behaves, especially if that function takes anything more than constant time to do its thing.
For example, consider some function that gives you an answer, but always takes linear time. If you keep calling it repeatedly that means it now takes way more than linear time.
After a while, almost every C or C++ programmer has seen something like this:
strcat( buffer, "apple " );
strcat( buffer, "banana " );
strcat( buffer, "cherry " );
Each time strcat() is called, it traverses the entire buffer before it adds the second string. So, now those 3 functions calls are taking O(n^2) instead of just O(n). Put those 3 functions calls above into a loop, and they require O( n * n * m ) where m is the number of loop iterations. And then if that loop is inside a function that gets called repeatedly, the performance gets even worse. That is probably the most common example I see of people treating a function as a black box.
Many new programmers do things like that and unwittingly force other programmers into making expensive function calls. Just recently I fixed a cleanup routine that suddenly went from taking linear time to cubic time to finish. A coworker found a function that removed 1 item from a container, so she used it thinking of it as a black box. But, that function not only deleted 1 item at a time, but also sorted the remaining items using bubble sort - which takes O(n^2) time. After her change, the cleanup function was now much slower. No wonder it now took 10,000 times as long to cleanup a container of 100 items! The fix was to go through the container once, and not calling that other function - which needed some fixing of its own.
Have you ever wondered why some programs are slow no matter how fast your CPU gets? Ever wonder why some of the most complex software take so much time? My guess is that there are so many layers of inefficient functions calling other inefficient functions.
One reason I like the STL is that it has complexity guarantees. The definition of the STL tells you that some functions will take constant time always, while other take linear time at worst. Any implementation of the STL must match (or beat) those guarantees. So, you never have to worry about calling an STL function and wonder if its implementation will take 10,000 times as long as another implementation. I think of the complexity guarantees as some grey knowledge about the STL, so I can treat it as black box otherwise.
As a courtesy (or warning) to other programmers who have to develop on top of my code, I often say how complex a function is. Such as whether it takes constant time or logarithmic time for an input of a given size. That little piece of grey knowledge allows them to decide how they want to use the function.
Half of all programers are below average. And here the average is pretty low. Its amazing how I can barely pass calculus while the guy next to me gets an A, and yet I get an A+ in an OOP class where the average is a C
"The Big Picture" is something that hasn't been covered at all in my computer engineering bachelors program. Hell, they don't even teach us how to use CVS! How the hell can you work on a large project with more than one developer without using CVS (or some other source control software) religiously? Teamwork is discouraged througout the curriculum. Shit, you can get kicked out of school just for collaborating with another student on a homework asignment. Now for the first time in my last semester after 5 years they're trying to force us to work on teams to do our senior projects. I told them exactly where to stick it. Frankly I don't WANT to work with any of my classmates, not because I'm not a team player. I would LOVE some help on my project. Rather I know them well enough to know none of them would be very helpful. They were never taught the big picture, and were either non smart or not dedicted enough to pick it up on their own.
I'm rambling.
-73, de n1ywb
www.n1ywb.com
Managers are trained to deal with schedule and budget. Not with designing complex systems.
An organization is a complex human system. Executive management are charged with viewing the big picture and designing a large complex system -- but they can't use an engineering approach, because they're dealing with people, not objects.
It's arguably more complex than engineering for many reasons: you can't guarantee strengths in individuals - you can only observe and take an educated guess, people tend to dislike change and may actively work against any change, plus there are many fuzzy aspects to actually inacting change in an organization due to the complex & mixed nature of an organization's motivation.
The closest thing I can think of is that managing an organization at the executive level is like debugging a concurrent, distributed system with byzantine failures.
This is why good executives are so rare, and why so many companies do a poor job of organization and re-organization.
(One half-baked aside:
I find it amusing that the general solution in computing to byzantine failure -- quorum consensus -- looks amusingly like organizational bureaucracy. Both are woefully ineffecient as size increases, but work to eliminate corruption.)
-Stu
Folks are not taught that sometimes 1.0 + 1.0 != 2.0 (rounding errors)
:
That's a scary thought!
I remember when we were taught at school that computers aren't as accurate as you'd like to believe. The teacher wrote on the board something along the lines of
FOR v = 1 TO 2 STEP 0.1
PRINT v
NEXT v
(ie. on every loop, v is incremented by 0.1, from 1 until it equals 2)
and told us the loop would never end. I remember it was pretty hard to believe at the time. He didn't explain why; he just left us to try it and see for ourselves and fathom out why. Either that, or he'd just discovered it himself and he hadn't a clue what was going on...
I've never trusted floating point values since. I'm currently up to my eyeballs in physics simulations using single precision for speed. Trying to keep the rounding errors under control is half the battle!
Check out Michael Jackson's "Software requirements & specifications:a lexicon of practice, principles and prejudices." There is a chapter entitled "Deskilling." I think it addresses the issues you raise.
Another elitest post without a real clue.
I disagree. AFAI understand, the list is not about the tools per se, but the level of understanding required for each tool.
A good programmer really should not care HOW they get things done- ONLY that they DO get them done.
By your definition, a "good programmer" is one who is capable of functioning at a high level in the list. Well, QED. IMO the list doesn't imply a good programmer can't use Frontpage, it only implies that someone who can use Frontpage but is clueless about raw HTML is unlikely to be able to create C code that interacts with APIs or otherwise see the big picture of what he's doing.
My major point is that thinking with a black box mentality in most situations will allow somebody to stick it to the programmer when things go bad. So for your own job security and satisfaction, it is important to either deal effectively with a poorly structured environment or make sure you are in a place that has a good development methodology.
And no I have not written 100% solid code but I have worked in a lot of poorly organized shops and the programmers usually were to blame for accepting the poor docs given to them be not understanding their ultimate responsibility for giving the customer what they needed.
All security problems stem from programmers not knowing how to program!
Why did you write the log, was the analyst such a pain to deal with that you couldn't resolve the issues as you came accross them, or in the design phase?
The log smacks of 'I told you so', but did you tell him/her in the first place?
Ok, maybe this is a pipe dream, I usually moan about some crap, and pick up the mess though internal bug reports (which can only happen post release), it's more of a fault in the corporate system then most peoples progamming skills.
thank God the internet isn't a human right.
For many years I've seen this evolve. Most coders get caught up in there code and their target system. They usually overlook the intricate parts of where their software will run. Being a Linux user and looking for solutions that are cross platform I do look heavily at Virtual Machines such as Java VM. I really like to code my projects independently of the platform. If you only worry about independence of platform all the time you do give way to many security concerns. You have to look not only at the client machine but the network environment that your application will run on and think of other possible network configurations. Who's code (beside embedded systems) is not going to be implemented into a network environment?
The largest amount of security holes these days seem to be stemming from poor programming ethics. The amount of software coming out seems to be pushed more by financial needs of the vendor then the need for good software. If software was given the proper tests (i.e. Alpha and Beta) we may actually end up with more secure software. For you programmers out there look at the big picture as often as possible. I myself see myself falling into this dark hole of the Black box.
Happy Coding!!!!