"ExamSoft" Bar Exam Software Fails Law Grads

New submitter BobandMax writes ExamSoft, the management platform software that handles digital bar exam submissions for multiple states, experienced a severe technical meltdown on Tuesday, leaving many graduates temporarily unable to complete the exams needed to practice law. The snafu also left bar associations from nearly 20 states with no choice but to extend their submission deadlines. It's not the first time, either: a classmate of mine had to re-do a state bar exam after an ExamSoft glitch on the first go-'round. Besides handling the uploading of completed exam questions, ExamSoft locks down the computer on which it runs, so Wikipedia is not an option.

Comments

Really?

[nospam007]

"leaving many graduates temporarily unable to complete the exams needed to practice law."

And that's a bad thing, because ... ?

Re:Really?

[fiziko]

The business next door proctors these and similar exams. They are expensive and not available in every community, so the test takers have often paid a relatively large amount of money at this stage of their lives, not just several hundred to take the test, but also travel, accommodations, missing day(s) of work, etc. to be where the test is available. The proctoring company does not charge them for the second attempt, but all of the expenses needed to be there get doubled.

Re: Really?

[O('_')O_Bush]

Lawyers are not necessary to maintain law and order. They are useful only when law is written by and presided over by other lawyers, for lawyers.

That is, they are a solution to a problem they create.
  You can look back at just about every functionjng society for most of human history and neither find mob rule nor lawyers. You also find law that is comprehensible to a lay person.

Not saying our legal system is better or worse than old legal systems, just that the point you made about mob rule is certainly not necessarily true.

Re:Really?

[rossdee]

"Yeah, mob rule is so much better than law and order..."

I have never seen "Mob Rule" but "Law and Order" stopped being good after Jerry Orbach died.

Re:Really?

[nealric]

Whatever you feel about the necessity of lawyers in society, many of these graduates are out $150,000 of tuition and are $200,000+ in student loan debt. They are prohibited by law from working in their profession until they pass the bar, which is only offered twice a year. So yeah, it's a pretty huge deal to be sentenced to 6 months of unemployment when you are in deep debt because of a software glitch.

Re: Really?

[EvilJoker]

I still can't read a word of it though

Lockdown

[GigaplexNZ]

Besides handling the uploading of completed exam questions, ExamSoft locks down the computer on which it runs, so Wikipedia is not an option.

Yeah, that'll work, because nobody has internet capable cellphones, secondary machines or even Virtual Machines.

Re:Lockdown

[Anonymous Coward]

Besides handling the uploading of completed exam questions, ExamSoft locks down the computer on which it runs, so Wikipedia is not an option.

Yeah, that'll work, because nobody has internet capable cellphones, secondary machines or even Virtual Machines.

and exams are never held in controlled conditions on known hardware with invigilators..... but sure lets let law students BYOD to there exams....

Re:Lockdown

[Anonymous Coward]

For exam takers, a secondary machine and internet capable cellphone is not an option. The exam doesn't take place in your living room but at a monitored location. In my state, the exam proctors don't even let you bring in your own pencil. Pulling out a cellphone would be a great way to be kicked out and never be allowed to take the exam again.

I am not sure the current state of virtual machines and ExamSoft, but at least a few years ago the ExamSoft software would not run when a virtual environment was present. While I am sure people have found workarounds, the point of the software (and why most law schools and state bars use it) is to avoid that result.

Re: Lockdown

[pla]

Sure a bunch of geeks with no legal training could use Wikipedia and slashdot. To pass a law exam. Maybe yahoo news posts for the constitutional parts.

...Or maybe just use any of an hundred searchable online testbanks of past Bar exam questions?

TFS mentions Wiki, but you'll notice that TFA did not, nor did I. No doubt, if someone seriously tried to do this, Wikipedia would fall pretty far down the list of places to look for answers.

That said, what the GGP and I jokingly pointed out counts as a much more serious issue for those borderline folks actually studying law. Sure, I would probably have trouble even figuring out the intent of some of the questions, and even if I didn't get stuck on the impenetrable jargon, I probably couldn't realistically look up the answers fast enough to finish it in the time given. Someone who (barely) made it through law school, however, would no doubt have at least picked up enough of the core skills to successfully (and quickly) make use of online resources, given the chance.

Re:Lockdown

[sandytaru]

Last time I took a test (CAPM), the testing place gave me a temporary locker to put my stuff, and also requested that I turn my pockets inside out to show I didn't have a tiny cell phone or something hidden in them. They take it pretty seriously.

I noticed that the exam software we used was running on XP and appeared to have been originally programmed for Windows 98. I wonder if they ever upgraded those boxes to Win7...?

Re:Lockdown

[Chelloveck]

RTFA:

The digital system for the exam works on usersâ(TM) personal laptop, which they bring to the testing facility, where they download the company's application to the computers they use to take the tests. At the end of the exam, the file closes and locks. When the user is able to connect to the Internet, the file uploads. Users cannot make changes to the file after the conclusion of the test.

You have to drive to the exam site but you're expected to bring your own equipment? Who thought that up? Rather than trying to intrusively lock down everyone's machine it would be far better to simply issue everyone a cheap tablet or netbook on which to take the exam. Controlled hardware, no need to try to "lock down" innumerable variations of BYOD. The ExamSoft web site says the software runs on "any modern machine", defined as Windows, Mac, or iPad purchased in the past 3-4 years. But disable any anti-virus, and no VMs. They're basically trying to secure any random machine off the street to prevent cheating. That's a very fine example of "doing it wrong".

Lawsuit Just Begging to Happen

[Mr D from 63]

Sue the bastards... but they might need to hire a lawyer to do it.

Trust

[DoofusOfDeath]

If we can't trust these applicants to take the test honestly, how can we trust them to act as officers of a court?

Re:Trust

[flopsquad]

I took the bar exam this week. My experience was that there are lots of restrictions on what you're allowed to bring in with you (in a clear ziplock bag), but they are not cavity searching you for violations, or even really looking at your bag at all. I.e. to a large degree they are trusting us to take the test honestly.

Now of course the nerds among us (you, me, most everyone else here) can think of clever ways someone could conceivably cheat—a tiny scroll in 4pt font rolled up into your pen, rules of law written in uv ink on your shirt sleeves and wearing special glasses, etc etc. But apart from the fact that it's completely unethical (which matters more to lawyers than pretty much every other profession), getting caught will get you banned for life (still with six-digit loan debt) and it wouldn't be all that helpful anyway. There's like a thousand pages of testable material and time is extremely limited. Unless you broke into the NCBE offices and stole the answers, cheating just won't be a very effective strategy.

Same goes for the computer/essay portion. Yes it's your device to which you have root access, a third party can never 100% verify security. Of course a 1337 haXor could trick the ExamSoft program into running on a VM or, hell, even hard solder something into the keyboard circuitry that sent a paragraph worth of text at the press of a key.

But this isn't spycraft, all they need to do is detect something (anything) hinky and then banhammer. Are you willing to bet 3 years of tuition on your certainty that you beat the program and that it logged nothing? And, again, there's another thousand pages of material being tested, in essays where you get 45min and 6500 characters (maybe 1200 words) each. It's a hell of a lot of potential downside and very little upside unless you burgled the bar examiners' offices. In which case, why are you trying to become a lawyer? You could be planning Oceans 15 right now.

Re:Trust

[DoofusOfDeath]

Those are preparing to be lawyers, not judges or prosecutors.

I thought that even civil and defense lawyers are considered officers of the court.

I also think that even they are given certain powers not available to regular citizens, such as issuing subpoenas. I thought that was one of the reasons for requiring even them to be of good character.

perspective from one of the victims

[Anonymous Coward]

So I took the NY Bar yesterday and can validate this is all true.

This thread has seen a lot of jokes, but just to put things in perspective:

This software costs $100 for a one time use accounting for two 3 hour sessions. Furthermore, it is bound to your computer, so if you need to transfer it, you need to pay for an additional license.

The software is not complex. It has exactly 3 main functions:
1. Provide limited word processing functionality
2. Lock the user out of other programs
3. Automatically upload the answer files to Examsoft servers upon closing.

At my administration of the exam, there were perhaps 1000 people. Mine was in Albany, but there were also administrations in NYC (the largest), Buffalo, and I think one other location. So say 5000 people. Several other states were offering their bar exam with Examsoft on the same day, so lets be generous and say there were 50,000 students who needed to upload files over the course of an evening. The files uploaded consisted of 2 250k files in their proprietary format. So we are talking 25 whopping Gigabytes TOTAL being thrown at them. And they were paid roughly $5M for one day of testing. Also, bear in mind, that there is a winter administration as well, albeit with fewer candidates, and that Examsoft is used for many other types of test as well, so their yearly gross is probably well into the 8 figures.

Yet somehow, they didn't have the bandwidth/and/or the server capacity to handle the connections or puny amount of data being thrown at them? This is unacceptable by any business standard. And they have been the exclusive provider of this service for years, so it's not like they had no notice of what kind of volume they should be prepared for!

I personally paid $300 just to take this exam ($200 to NY for the exam itself, easily one of the cheapest fees among the states, + $100 for Examsoft). I also spent about $1000 in travel expenses. After many tries, I apparently managed to upload the files (my software said they failed, but I received confirmation emails saying they had succeeded). Examsoft can't confirm either way whether data corruption has occurred. If they did not manage to upload in one piece, my exam will have automatically failed by the Bar's standards, a decision which is unappealable. So I will be out my financial investment, close to two months of study time, as well as 6 months in lowered earning potential SINCE I WON'T BE LICENSED.

This was a massive, massive failure, and I will frankly be shocked if multiple lawsuits aren't filed against Examsoft over this.

Re:perspective from one of the victims

[nealric]

You don't need to BE a lawyer to sue and you don't even need to hire one. You can sue on your own behalf without a license.

Let Me Get This Straight

[Anonymous Coward]

... So the end result is fewer lawyers in the world?

That's not a bug. It's a feature.