Are Written Computer Science Exams a Fair Measure?

me! asks: "I seem to have this inability to write substantial chunks of code (500+) in exam conditions (for uni). I have been writing code for years for open source and commercial applications, so I know a thing or two. There is just something about exams and code that does not work for me. I find that I need to be sitting in front of a computer to get a problem out, to get in the 'vibe', have you will. I have done exams on computers (closed environment) that involve coding, and it work so much better for me. So what I am asking is...how do people tackle exams that involve solving problems on the fly, on paper, in exams?" I have this exact same problem, and I've never thought written tests were a fair way to measure someone's knowledge of coding. It's fine when you are asking questions about design and structure, but when you need to write code it falls way short. How do you feel about it?

Comments

Error checking by compiler

[Anonymous Coward]

Is the reason you don't do well on the tests because your hand gets tired writing so much code or because you don't know if you are doing things right because you can't compile and see where your errors are. For me it is easier to think of my code as communicating with other programmers my ideas. Writing in this style is easier on paper.

Re:Error checking by compiler

[heybrakywacky]

first of all your teacher does not want a compiler to check your work.. you are supposed to do it yourself...

You're supposed to write perfect code the first time around? I agree that it's an interesting test of your knowledge of a concept, but it's completely unrealistic from the standpoint of what you would do in real life (as much of academia is, I'm afraid). I can write code that is pretty close, debug through compile and runtime errors, unit test for missed functionality, etc., and have something working in half the time it would take me to attempt to do all that on paper, analyzing each statement "by hand." Is this not the way that 99% of all developers work?

and second, you should not be looking up stuff with man if you ever get a job... just wait untill the boss finds out that your spending your payed hours learning how to code when school should have taght it to you...

I use man as a reference, as I would imagine most people do. I don't keep every last detail of every function or program I'm going to have to use in my brain.

I know that both of these arguments are somewhat counter to the purpose of academia. I have an HP calculator that will do integrals for me too, but can obviously see where that wouldn't be allowed on a calculus test where you're demonstrating your ability to integrate. But as for writing code on paper, it just never made much sense to me. It's completely outside the medium for which it was intended, and completely counter to the process which most developers use to write code.

If you want to test someone's knowledge of a computer concept/algorithm/etc., have them write detailed flow charts. Get 'em used to it. Make UML requirements. These are the kinds of things that not only make for good development practices, but also will have a practical application when students actually get out of school and start working.

My $0.02

Re:Error checking by compiler

[Iffy Bonzoolie]

just wait untill the boss finds out that your spending your payed hours learning how to code when school should have taght it to you...

Well, I don't think that's fair at all... You are paid to do a job, and it's easy to be effective without memorizing tedious information. Compilers are better at finding missing semicolons or parentheses that you are. Why not let them do it? You might know what a library function does, and how to use it, but maybe you forgot the order of the arguments. Sure, a guess probably won't compile, but it doesn't mean you are a bad programmer. I'd rather work with someone who has a strong grasp of OOAD and can learn the specifics on the fly than someone who can write code that compiles every time out of the gate, but doesn't "get" the whole "objects" thing. (At least in my case, where we work in an OO environment.)

Part of being a good programmer is using the right tools to maximize your effectiveness. The compiler is a great syntactic and semantic validator. Let it do that for you. The manual is much better at retaining information about specifics. Let it do that for you. As long as you can use these tools to craft quality code, then you are being effective. There's insignificant cost to running the compiler twice as often, or looking something up in a manual or on the web. You tend to memorize the things you use really often, but what you use can change from project to project, or release to release. It's hard for an employer to find someone who already knows everything exactly that they want... and when they get them, they may not be as effective/productive as someone who has to learn part of it on the job.

If it's a class in Specific Language X Syntax, then maybe that makes sense, but that's a useless class. At my school, we did a lot of hand-writing code on tests, but usually pseudocode was sufficient to get full marks, and you could get good partial credit if you had the basic idea. The focus was not on details like syntax, but on the concepts, ideas, paradigms of the subject (compilers, operating systems, computer graphics, or whatever).

You are hurting yourself if you stop learning once you are out of school. Technology is always moving and changing, and what's available and how to use it might change on you, you have to adapt. You should always be learning, 1 year after school, 10 years after school... why stop? No one I know at anyplace I've worked ever expected anyone to know everything about what they were supposed to do. They care about results... providing those requires being able to learn effectively on the job, because who knows what you might be doing next release, next project, next company... Being versatile will get you exceeding expectations all the time. Success is rarely about what you know, but how quickly and how well you learn, and whether you can make use of it.

-If

Re:Error checking by compiler

[dakoda]

"Part of being a good programmer is using the right tools to maximize your effectiveness. The compiler is a great syntactic and semantic validator. Let it do that for you. The manual is much better at retaining information about specifics. Let it do that for you. As long as you can use these tools to craft quality code, then you are being effective. "

Exactly =) Programming, as a career, is _not_ memorizing all the man pages, but knowing how to use them quickly, and learn on the fly. Every task you do is likly to be different, so why bother memorizing everything? i, for one, use them for function argument order, which i never remember because all the libraries i use have slightly different orders.

I suppose it'd be better to think you get it right, only to find a logical error somewhere down the line when you provide the source and dest structs in the wrong order..? right ;)

I feel the same way

[scode]

I have no resect for any exam that involves writing actual code on paper. Asking about fundamental principles is fine; asking for pieces of code or specific API details is IMO rediculous.

Granted, you can never design a perfect test. But you can at least do away with the totally obvious imperfections like asking people to do things in ways they never do and never will do in the future (except on other tests).

The few times I was subjected to these tests, I sure felt primitive erasing entire chunks of code after having realized I needed to insert something in the beginning. And it's a hard thing to avoid; I don't think any reasonably experienced programmer writes code entirely from top to bottom. Just a simple thing as placing braces becomes a hurdle in writing unless you know the length of the code you need to put between them!

Re:I feel the same way

[MobiusKlein]

Jeeze, it's not so bad. In upper division and graduate math classes, they have you write proofs in exams.
You have to
1) Remember definitions.
2) String statements together in logical, consistent groups.
3) Reach the proper conclusion with the hypotheses given.
4) Justify interesting or tricky steps.

Sounds a lot like creating code in an exam situation. A reasonable professor (or TA more likely) will read the code/proof looking for the the correct ideas and flow more than perfect pedantic correctness. At least that's what I did when I graded for upper division classes.

rbb

ps - use scratch paper to avoid erasing.

Re:I feel the same way

[AJWM]

You miss the point.

Indenting is relevant to this for the same reason that Python uses only indenting, not braces or begin-end, to indicate block structure.

Yes, one school of coding in a text editor is to place the open/close braces first and then fill in the code. As you say, that's tough to do on paper. It's also a crutch.

Another school says indent your code properly and the braces are superfluous, just there for the compiler (for non-Python languages).

If you follow the latter train of thought, you just put in the closing brace when you're done filling in the intermediate, indented code.

I don't mean to sound mean..

[Fixer]

.. but you should know the language you are using well enough to be able to simulate what's going to happen in your head. Yes, it takes a lot longer, and you have to double and triple-check things, but it's a good skill to have.

This is A Big Problem...

[Mad Browser]

This was my least favorite thing about CS at school...

I am much better at solving a programming problem in front of a computer.

Also, they are nothing like real world conditions... Do you know any programmers without reference books on their desks? That have the whole Java API memorized?

No need for large portions

[jacobjyu]

I find that the CS tests that I have encountered don't require me to write a large portion of code, just snippets and tiny routines. Usually, a test problem leads us down a long list of sub-problems that, at the end, proves to the professors that you can indeed reason about programming, even though you haven't written a large portion of code.

Testing the ability to write large chunks of code is not reasonable on a written exam, and then, usually your grade is determined by your handwriting speed and neatness :)

Writing code (not typing it) is *hard*

[seanadams.com]

... because you don't have a compiler to tell you when you make mistakes.

This is why it's an outstanding way to test people's knowledge. Anyone can make a program work given a compiler and sufficient time. If you can do it with just pen and paper, and remember the syntax without having other code in front of you, then you know your shit.

This is why it's a great test for interviews. You'd be amazed how many "senior C developers" we interviewed at my last company who couldn't write push() and pop() for a linked list.

Re:Writing code (not typing it) is *hard*

[Sloppy]

in the last 6 years of programming (C++) I have never ONCE had to write a linked list.. [snip] ..Waaaaaay back when I was coding in C I STILL wouldn't have been able to do it without resorting to a reference book because I had a set of libraries which implemented this. [snip] ... It shows NOTHING about their ability to solve problems in real world conditions under their own recognisance.

Ugh. You've got everything backwards; I almost can't believe you're serious.

It's fine to have to resort to a reference book, but the thing you should be forgetting and having to look up, is how to use your library (e.g. the names of functions, the order of their parameters, etc), not how elementary data structures conceptually work.

If you can't implement a list or stack, it does show an inability to solve problems. Here's the problem: solve it! Can't? Then you're not a problem solver.

It's ok to forget libraries and syntax; seanadams is in the minority in thinking that matters. But stacks and linked list aren't things you should have to remember -- they're something you should be able to derive and re-invent on-the-spot. If you can't whip them up ex nihilo, then your problem solving skill is ..um.. frumious.

Learn this skill

[Arandir]

This is a skill that you need to learn. When you go out on a job interview, you will most likely be given a marker and white board and told to code something. If you're going to freeze up without a monitor and keyboard in front of you, the employer will be very hesitant.

It seems that you've associated syntax and semantics with the visual cues of a computer. That's not the same part of the brain that answers exams. So you need to retrain yourself. Here are some things you can do to relearn programming:

A) Stop using IDEs. Use the plainest text editor that you can find to write your code. Turn off any syntax highlighting and code completion.

B) Design your code before you write it. Use UML, flowcharts or whatever, but design it first. Then when it comes time to code you will know where you are going.

C) Write code away from a computer. Use a pencil and paper. I didn't have a computer when I was in college (very few people did). So I coded by hand on paper, then went to the lab to type it in.

Re:Learn this skill

[Arandir]

This would be like if an English prof told you to write a term paper and you did extensive research but lost points for bad grammar. It matters, but is it enough to dismiss the research done?

Bad grammer on an English test? Gee, that sounds like arithmetic errors in a calculus test!

Re:Learn this skill

[e40]

I've hired a lot of people over the years and I'm completely against this type of on the spot exam.

I find that people's performance over the long haul has nothing to do with a 30 minute interview coding session. Some people don't work their best when under pressure, especially when they are interviewing. A lot is riding on that interview, and the stress does very odd things to people.

Re:Learn this skill

[prockcore]

"When you go out on a job interview, you will most likely be given a marker and white board and told to code something."

Most likely? I doubt it. Just because you may have done it doesn't mean everyone does. I've never had to write code on a whiteboard in a job interview. I *have* had them give me a program spec and say "go home, make this work, and email it to me" That was even better in my opinion.. there wasn't a time limit, so they can judge you on how fast you work when there's no motivating deadlines, and the project can be a lot more complex than "write a 5 line routine on a whiteboard"

Now I'm the one doing the hiring, and that's exactly what I do.

Re:simple

[Anonymous Coward]

The greatest thing about college for me was NOT the CS, Engineering, and math courses that eventually got me the job I now enjoy...

It was the total mind-opening academic scene that surrounded me 24/7 for 5 years, the chance to really go out and party my brains out to relieve the stress, the hour long conversations with professors who are true geinuses in every sence of the word.

You might get paid a lot of money, but if you see college as simply a means to that end, I strongly suggest you take a year off and attend full time. Because you certainly do not get the point.

And, through so many experiences, I learned that your final career and salary doesn't define you, your worth, or how much you can get out of life.

but congrats on your job, though. Pretty sweet deal.

Good programmers use tools, not memorization

[Jack William Bell]

I have run into this same problem in job interviews. Usually it comes down to the fact that, on the job, I have resources at my command; books, web sites, code examples, etc. So, rather than memorizing every API (or even every keyword), I become very facile in knowing where to look up what I want very quickly.

But when I am asked questions about the same thing away from a computer and my books I start drawing blanks even about things I know in detail. This is why, IMHO, computer programming exams and even job interviews should be 'Open Book'. Memorizing details doesn't make you a good programmer -- knowing how to use the tools does.

Jack William Bell

Re:I also

[Arandir]

If the test is asking you to use obsure Motif function calls or NT sockets, then you have a point. But if you're being asked to write a linked list or balance a binary tree, then it is perfectly fair. These are skills you are expected to know as a software engineer.

Just because you have a calculator is no excuse not to learn how to add.

Re:simple

[Anonymous Coward]

DUDE!!!!!

sociology, and all those core classes totally sucked but those were the ONLY classes that i took that ever had hot chicks...

Re: code on exams

[Arandir]

When you're on the job, you WILL have those kinds of time pressures. One of the things the interviewer wants to know is how well you can cope with it.

It's Friday at 5:00 and you're leaving on a weekend trip at 7:00. The software is handed off to SQA on monday. There's one more bug in the queue and it belongs to you. Question: A) Do you bail on your trip? B) Do you sneak out the back door? C) Do you go fix that bug in 57 minutes?

Re:Hate it!

[Neon Spiral Injector]

So make it a timed test.

The only language I can write 100% on paper is COBOL, and I hate COBOL.

With C I need a compiler to catch the few typos or what ever I made. But I don't need all day, one or two compiles does it.

Luckly at my school the COBOL tests were on paper, and the C test was on a computer (and timed).

I liked my OS class. The UNIX tests were held in front of terminals. I couldn't remember the quite how to use the non-GNU `sort` command. So `man sort` saved me there. I told the teacher what I did after the test, he said, "excellent."

Re:Hate it!

[denial]

Exams that ask for written code aren't testing for syntax errors, generally they are marked with lenience in that area.

But written code tests something very important, which is whether you can design a solution and describe the solution in essentially correct code. That's something very different from just slowly crufting a solution together because you can build it up. I've seen so many programmers commercially who can't design, describe, or communicate their approach to solving a problem, and I think quality suffers directly.

Obviously there are other important things to test that require prac exams to test, but asking for written design, pseudo-code or even code in some exams is, imo, a good idea.

benefits of paper programming

[Preposterous Coward]

Writing code on paper may not be a "fair" measure of your actual sitting-in-front-of-a-PC programming skills, but that doesn't mean it's entirely useless either. Some reasons it's arguably a good exercise:

One of the things that universities are supposed to do is give you a chance to express your knowledge in new contexts. It's relatively easy to apply things you've learned in familiar contexts, but how good are you at applying them in unfamiliar environments?

Making you work with pencil and paper forces you to plan ahead and architect your solution, not just jump in and start writing code.

When you write by hand, the logic and aesthetics of your code are especially important -- since you (and the reviewers) can't run the code to see whether it works or not, you need to make an extra effort to make it transparent and comprehensible.

I might take issue with an exam that was nothing but one big handwritten coding assignment, but I don't think this is at all out of place as a part of an exam or course. It's a little like essay questions in a soft-science class; they're unrealistic because in the "real" world you have access to reference materials and time to spell-check and so on... but at the same time they are a useful gauge of your ability to articulate a subset of your knowledge, to think on your feet, etc.

Also, in courses I've graded where such exercises where required, we usually didn't worry about things like minor syntax errors that didn't obscure the intent of the author. The goal was to look for an understanding of and ability to solve the problem, not to be human compilers.

It depends what they ask and how they mark

[oliverthered]

Questions like, here is a recursive function, write the code to turn it into a loop.

Write some code to rotate/scale an image.

Fine an element in this sorted list (of say 10000)

Give this xxx write a class model for it.

Now as long as they don't expect perfect code, eg. forgeting to +1 because of a rounding error, or a binary search can result in a high or low result(for a non-perfect match), these seem fair enough questions to me because:-

They all test your abilty to write good code, not just code that works.

I know loads or 'programmers' who don't know how to turn a recursive function into a loop.

Would use sin/cos to rotate each point on the image, instead of using an interger line drawing algo to scan accross the image to rotate it (or even floating point)

Would write a n order search instead of a binary search.

And would use a poor or unflexable design pattern.

So long as they mark your programming skills and not how perfect your code is, i don't see a problem.

Debugging is also a key skill, but is very hard to test in an exam, a good debugger uses a hell of a lot of past experiance and insperation to locate, fix and check that there are no simila bugs.

Answer: Not fair. But we have to do it anyway.

[IvyMike]

I used to teach a class to engineering freshman, and part of the course was writing code in Matlab. There are in fact a bunch of matlab questions that you can ask that aren't just writing code, but you really don't get complete coverage unless you have the student put some code on the page.

Additionally, there probably was some cheating going on with the homeworks (this was a large class taught to all freshman, with relatively small programming assignments. It would be difficult to prove someone was cheating.) We needed a controlled environment where this couldn't happen.

However, we all realized how crappy this was for the student, and we graded the assignment more like it was psuedocode. If the code had obvious typos, we ignored those. If they were close on syntax, but a little unclear on the API, we usually let it slide, too. (We understand when online you can look that stuff up with the online help.) The thing we stressed was the overall algorithm itself, as well as a demonstration that the student knew how to use matlab. (After all, as a TA, I'm taking programming courses myself; I have some idea of what's reasonable to expect and what's not)

If you do have a written CS test, and the prof does mark you off for syntax errors, I would probably go in and (politely) complain. Say, "I made a typo which would have been caught by the compiler, but I got the gist correct." If you demonstrate that you know what you're talking about, and you're not a jerk about it, and the professor's not a total jerk, I bet you'll get most of the points back. (We would do this all the time, especially if the student could clearly explain what they were trying to do.)

If the professor or TA is a jerk, all bets are off, though. But if they're a jerk, they aren't interested in being fair anyway, so you were screwed from the start. We've all been there. Sorry.

Re:AP Computer Science

[DMDx86]

When I took the AP computer science test in high school we had to code on paper! I'm just like you...I get my vibe when I am in front of the computer coding, not with a pencil and paper.


Well, I will be taking AP Computer Scince next year. I have seen copies of the test, it seemed pretty resonable, IMHO. How did they grade your exam? Did they give you oppritunity to contest the grade if you felt it was graded incorrectly?

DESIGNING code (not typing it) is *hard*

[kwerle]

I half agree with this. Syntax doesn't matter. If they can't get the syntax right, who cares. That's like saying that people who can't spell can't think - those are unimportant details. The important thing is that the know HOW TO WRITE push and pop. Not that they be able to type it once and have it compile.

I can't write a java main() statement. Just can't remember how - don't do it very often. But my emacs expands when I type 'main', so it's not a problem :-).

I couldn't write push and pop and get it to compile, either. But I know how. And I can communicate the design.

If you're talking about "can't design push and pop", then my sympathies. If you're talking about "can't get the {}'s right", then I don't want to work with you anyway.

overrated?

[Anonymous Coward]

college? overrated? yeah...only if you're a geek who codes in a vi screen in his pitch-black dorm room all day...

Are written plumbing exams a fair measure....

[Lictor]

... of someone's knowledge of fluid mechanics?

Once again, Slashdot continues to perpetuate the (Computer Science == Programming) myth.

Programming IS NOT "computer science" any more than telescope design is Astronomy (with apologies to Dykstra).

Programming is a trade skill. Certainly, a challenging and rewarding one.. but it is not a science.

Being the first guy to figure out quicksort... thats science... thats research. Coding a quicksort in LISP is not.

I've ranted about this before, but it upsets me to see such a misleading headline.

I think *computer science* exams are quite fair... if you understand the data structures and algorithms associated with B-trees.... you shouldn't have any problem describing them on an exam. Likewise, if you understand the halting problem, reducing a simple contrived example should be doable on an exam.

On the other hand, I very much disagree with the "code this up in PL/I, using proper syntax" type of exam question. What an enormous waste of time...

The bottom line is that if you've been through a computer science program and understand the underlying principles, I should be able to give you a manual for 'trendy language x', a few examples, and 48 hours. After that, you should have no trouble coding in Jav^H^H^H 'trendy language x'. Will you screw up the syntax occasionally? You bet. Is it fair to dock points on a formal exam for being human and forgetting a semicolon? I sure don't think so; and I've *never* done that to my students.

But I stray from my point. Once more, for the record, since no matter how often I rant about it, no one seems to listen:

COMPUTER SCIENCE != PROGRAMMING.

well, of course...

[gyratedotorg]

I find that I need to be sitting in front of a computer...

of course it's easier to write code on a computer. you have the compiler/interpreter to immediately alert you to your mistakes.

still, i dont see the point in these exams where you have to write a program on paper. its infinately more important to understand the concepts rather than to memorize the syntax of a particular language. thus, i believe that pseudocode should be the official language for written exams that involve programming in (at least higher-level) computer science classes.

Whats up with these places...

[Pinball Wizard]

First we have math programs that won't let you use calculators, now CS tests that make you write code on paper. There's always that argument..."if you get stuck on a desert island and need to rebuild society you won't have the benefit of these modern tools." I've actually heard a variation of this, repeatedly.

The coolest professor(this was a EE class) I ever had did it this way - he made the tests extremely hard, but allowed open book, open notes, and calculators. His philosophy was that on the job you would use these tools, so you might as well use them on the tests. You were competent in the subject if you made it out with a C, and perhaps 2 people in a class of 20 would get an A.

Actually, I think CS tests are pointless, period. They have very little to do with your skill as a programmer or knowledge of the subject. Its much better to give hard assignments that require lots of work before they are due(i.e. a two week assignment that is virtually impossible to finish the weekend before it is due). I can see concepts on tests, and perhaps a few functions to demonstrate knowledge of certain algorithms, but a 500 line program is ridiculous.

Re:simple

[Anonymous Coward]

Step 1: Drop out of college
Step 2: ???
Step 3: Profit!!!

These tests are wrong.

[acidblood]

I feel the same way -- every written test I had to take, I was proficient on the underlying theory, and would be able to code it in front of a computer. Most of the time I miss the 100% grade due to an off-by-one error, or an `if' clause testing the exact opposite of what I intended to. These sort of errors are easily spotted if you have a computer to aid in debugging, and that's why I believe coding tests should be done on a computer, period. Especially with students unfamiliar with the mechanics of coding -- after a few years of daily coding, a person seldom commits these silly errors.

As a side note, having learned touch typing, it's really hard to write on paper. Thoughts flow much faster than handwriting, and you end up losing yourself sometimes. I'm positive that also affects other coders when doing written tests.

You need to be able to do this

[mmmmbeer]

I, too, find it easier to write code on a computer, but there are times when you need to be able to write something out longhand. If you have a lot of difficulty doing so, I suggest you practice it. It can be a very useful skill to have.

First, it's something that will come up during an interview, and you can look really bad if you can't do it. I have given more than one interview when an otherwise impressive candidate was turned down because they couldn't write a relatively simple algorithm. I've also been on the other side of the table, where it really gave me an opportunity to distinguish myself. (Btw, from an interviewer's perspective, what's more important there is the approach, not the end results. I wouldn't be surprised if many of your teachers feel the same way.)

Also, it's often necessary to be able to write out code when you're working in a team. It's not always enough to just describe your approach, especially if another programmer wants to go a different way. It's like any visual aid, it helps you get your point across.

Finally, it can be useful when working on your own. I sometimes find that it's helpful for writing complex programs if I sketch it out longhand first. It helps me guarantee I haven't left anything out, so I don't have to go back over my code as much.

So, is having students write code on an exam a good way to test their coding skills? I don't know. But maybe coding skills aren't all they're testing.

Paperboy..

[bryans]

I actually find it easier and better to do it on paper first. My solution is likely to be more thorough and with less logical errors, doing it on paper.

When you get to a certain maturity level at programming you should be able to do it in a methodical manner - paper and pen first, before coding and the keyboard.

Start coding with discipline. 'on-the-fly' coding might seem far quicker, but wait till you face large complex problems.

Programming is a necessary tool

[VP]

Unlike fluid mechanics, where you can learn, and do research without knowing plumbing, you cannot be a good computer scientist without being very good at programming. Exams, which require you to write correct code, are usually done in introductory classes where the goal is to teach you programming in a specific language.

Learning the exact syntax and idioms of a particular language in an introductory class is necessary, as it teaches you how to learn a new language as much as it teaches you the basics of programming.

Re:ABSOLUTE BULLSHIT!!!

[Tadrith]

So, what you're telling me is that there is only one way to store data, and that personal preference has absolutely nothing to do with it?

Wow, I'd really like to know what kind of programming you've done lately. Perhaps in procedural programming there is a set place to put things, but as far as OOP goes, I'd say there's quite a few ways you could arrange things. People have different reasons for coding things differently, both syntactically and in organization. One person may find it more convenient to store something in a simply ASCII text file, someone may find a small but efficient database system better.

I agree, you shouldn't use more resources than you have to, but there are multiple ways of accomplishing things, each with their own benefits and drawbacks. If you can't see that, I'd really hate to see what happens when you have to work together with someone on something.

Re:Hate it!

[lucifuge31337]

All it takes is someone to grade the test that actually knows what they are doing.

But if they knew enough about coding to do that, they'd have a real job and wouldn't be grading written tests.

Oh well...it was a nice idea.