Slashdot Mirror
Is The DOJ Using Obsolete Software To Subvert FOIA Requests? (theguardian.com)
"A new lawsuit alleges that the U.S. Department of Justice intentionally conducts inadequate searches of its records using a decades-old computer system when queried by citizens looking for records that should be available to the public," reports The Guardian. Slashdot reader Bruce66423 writes:
An MIT PhD student has filed a suit in Federal court alleging that the use of a 21-year-old, IBM green screen controlled search software to search the Department of Justice databases...constitutes a deliberate failure to provide the data that should be being produced.
Ryan Shapiro's lawsuit alleges "failure by design," saying that the Justice Department records are inadequately indexed -- and that they fail to search the full text of their records when responding to requests "When few or no records are returned, Shapiro said, the FBI effectively responds 'sorry, we tried' without making use of the much more sophisticated search tools at the disposal of internal requestors." The FBI has a $425 million software system to handle FOIA requests, but refuses to use it, saying that would be "needlessly duplicative...and wasteful of Bureau resources."
Ryan Shapiro's lawsuit alleges "failure by design," saying that the Justice Department records are inadequately indexed -- and that they fail to search the full text of their records when responding to requests "When few or no records are returned, Shapiro said, the FBI effectively responds 'sorry, we tried' without making use of the much more sophisticated search tools at the disposal of internal requestors." The FBI has a $425 million software system to handle FOIA requests, but refuses to use it, saying that would be "needlessly duplicative...and wasteful of Bureau resources."
Green screen........mellenials.
When the Federal government is invloved, don't blame on intentional malice that which can be explained by consequences of Republican budget cuts.
No. This is wrong. They already have the tools to do better. They simply refuse to use them. If they had to buy some new software, you'd have a point. They don't, so you don't.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
... intentionally...
When the Federal government is invloved, don't blame on intentional malice that which can be explained by consequences of Republican budget cuts.
It's too bad we don't have another party, a group of people who could come together and vote as a bloc to prevent these sorts of outrages.
How quaint. You don't understand that most major politicians want a population of toadies and people they can punish. They want to destroy and humiliate. Wars and taxation are just window dressing in their sick minds.
Contribute to civilization: ari.aynrand.org/donate
Political flamebait works both ways. The other side of the coin is that Democrats set up overly complicated systems that can't work without an ever-increasing price tag, then complain (loudly) that they just aren't getting the support they need.
Let's move on.
When the Federal government is [involved], don't blame on intentional malice that which can be explained by...
...anything else.
Bureaucracy in general is a breeding ground for unintentional malice. There are literally thousands of people in the federal government with the ability to influence programs like this, and they often have conflicting priorities. Some are mostly concerned about the economic cost, thinking that a strong economy is the clearest path to "general welfare". Others want social support services, being of the opinion that minimizing hardship makes everyone's lives better. Some think that government should do as little as possible, allowing individuals to decide for themselves how to pursue happiness, while still others believe that a life led according to religious principles leads to a better eternity.
Those are only a few examples, and not terribly nuanced, either. People can have multiple opinions, conflicting opinions, and even different opinions for different subjects derived from the same principles. The representative government reflects the opinions of the people, and in a country of over 300 million people, it is perfectly reasonable to have a very complicated set of opinions in government.
The most that we as individuals can hope for is that occasionally, enough people agree on an issue that they'll do something matching one of our strong opinions.
You do not have a moral or legal right to do absolutely anything you want.
More recently, the FBI’s own investigation into the September 11 attacks found that “[o]n September 11, 2001, the Bureau’s information technology was inadequate to support its counterterrorism mission”, noting further that “[t]he FBI’s legacy investigative information system, the Automated Case Support (ACS), was not very effective in identifying information or supporting investigations”.
kinda bad when even you are on record as saying the system sucks.
Anons need not reply. Questions end with a question mark.
Not only have the state of the art in searching normal databases not changed in the last 21 years (specialized cases like web searches excluded), the use of "green screens" rather than a web interface have nothing to do with the quality of searches - the _real_ complaint is that the full capabilities of the existing system isn't used!
Having previously worked for a large corporation, I found that the costs are always limited to the ticketed price. There are others including employees to run and manage the system; process change; new audits; infrastructure; politics; expertise; training; software limitations leading to longer execution time.
Sometimes saving money leads to spending more money and creating more headaches. Not always, but you do need to understand the nature of the environment you are dealing with.
In our particular case we had a lab where we believed we would be doing good for the company with the $200/month network connection vs the $1000/month connection. Boy were we wrong. The $1000/month connection would have saved so much trouble with internal resources, even if the $200/month connection would have worked just as well.
Jumpstart the tartan drive.
The DoJ is one of many departments these days that are run for the benefit of the of the administration, not to serve justice or even the American people.
Alcohol, Tobacco and Firearms should be the name of a store, not a government agency.
You're kidding right??? They have plenty of money to conduct their obscenely detailed internal searches... This has nothing to do with the two fully inbred political parties. It has everything to do with the protection of the hegemony of the State.
Dear Microlimp: I give you 2 valid product keys for win7 and you reject both of them. Piss off you wankers!!!
We know what you need. You typed the evidence yourself.
I don't think they can do that kind of surgery yet.
By requesting a search that should have netted his own earlier FOIA requests, and didn't, Ryan Shapiro was able to demonstrate the inadequacy of the index-term search the FBI is using. Clever.
But, but, but, they don't intend to break the law, so it is okay.
if it ain't broke don't fix it
Is it just me or is anyone else getting pissed off with this excuse for shitty ancient systems still in use today?
Sure, by all means do not try to 'fix' something already working well.
But there's the keyword of the day: well.
This system is beyond well, another universe of not well.
A lot of these older systems are also usually maintained by awful sysadmins that are lazy, and even geared the whole system in weird exotic ways only they can understand, on purpose, so that they can use this to secure their job basically for life.
Any question of upgrade is hastily shit on with extreme prejudice.
Not everything new is bad.
Not everything is made by a group of wizard-wielding Indians that were certified in a summer course.
There are loads of people out there that do amazing work, well-documented work.
Political flamebait works both ways. The other side of the coin is that Democrats set up overly complicated systems that can't work without an ever-increasing price tag, then complain (loudly) that they just aren't getting the support they need.
The problem is that that assertion doesn't line up with reality. Go down to your DMV some time, and observe the kinds of systems that they're using. They're using databases built in the 80s and 90s on top of DOS, running on ancient computers with CRT monitors (at least around here).
What reasonable business do you know of that hasn't upgraded their systems since that time to allow for more efficiency savings, faster processing, reduced staff costs etc?
There's a lack of investment in this kind of system, plain and simple, being disguised as "government efficiency" by the republicans.
Why yes, I do!
Just another day in Paradise
Just because it looks old, doesnt necessarily mean it is. The mainframe paradigm is still alive and utilized everyday effectively. My doctor's terminal in the rooms are just thin-clients.
Good-bye
It's always a matter of priorities. The FBI would rather chase down 'pirates' than complete FOIA requests.
Faster! Faster! Faster would be better!
One point that keeps getting lost on the whole Clinton email fiasco is FOIA requests. Shouldn't she be in jail for violating that law? She was running a private server that contained official government records subject to this Act. She controlled the information that could be "lost" or "not found" whenever she wanted. That was legal but this isnt?
Frequent posts from The Guardian and BBC cover important events in the US that local media fail to report. This should make us wonder why American news media aren't on top of these stories.
Of course tight budgets constrain many traditional news outlets and restrict the ability to really investigate anything. The de-funding of Public Broadcasting was a disaster in American history, forcing a dependence on advertising and fund raising.
But those of us who entertain conspiracy theories may suspect that the 'free press' in the US is heavily influenced by various pressures from government and advertisers. For instance, many media are now forbidden at Trump rallies because they have offended The Donald by asking serious questions. Some media are unwelcome at White House briefings. Your local city/state politicians also have preferred, cooperative, outlets for their announcements. Cooperation with big advertisers is also important for American media to survive financially. Evil Monsanto stories go on page 3 or nowhere at all.
Most US publishers share with their readers the political posturing of government officials and the promotional 'news' of advertisers but fail to investigate anything. The remainder of US news is crime, weather, celebrities, a smattering of drama about terrorist activity, and no mention of large parts of the world like Latin America.
So, thanks Guardian and BBC, for a fresh look at the world and my own country.
...omphaloskepsis often...
The allegation is that only searches of an incomplete index are ever performed for FOIA purposes, and such searches are (1) archaic and unusual nowadays, (2) rarely find the requested material. That's more likely misfeasance than innocent.
The DOJ is an agency that ought to find compliance with law of primary interest, and arguments of 'needlessly duplicative' ring false. The plaintiff's test was amusing: he asked DOJ to find his previous FOIA requests, and was told there was no record...
This isn't about the hardware. The most important aspect is that the FOI searches only look at an index hand-built by whatever person entered the record. There appears to be no full text search for these external requests. Internal requests take advantage of modern techniques. Public requests are limited, by design. Kudos for exposing this by well crafted requests. "We tried" is not sufficient.
It's a nice simple argument to make to a judge as well:
"Your Honor, if you approved a subpoena for records and the response was 'We searched and found nothing responsive,' would you accept that response if you knew that the search consisted of nothing but looking at a list of filenames? After all, that's a search - a very poor one, but a search nonetheless."
fencepost
just a little off
This is the same Federal government that uses a mainframe system from the Kennedy administration in the 1960s to manage tax collection. If the USG can't upgrade a system that is responsible for its own core revenue generation, what makes you think they'll prioritize the FOIA system?
I've worked with government IT and it's behind the time everywhere. It's not a conspiracy. It's a constantly budget conflict.
When you have budget X and it's near impossible to get funding. This kind of stuff happens on a regular basis. IT winds up just making it work.. good enough for as long as they possibly can. Like a poor person or minimalist trying to get 50 years out of their old pickup.
Bullshit!
They'd just prefer spending our money that they confiscate under threat of deadly force on spying on the private communications of regular US citizens (particularly those who oppose the current regime and/or their policies/actions) and not on systems to keep them accountable to those citizens.
Strat
Progressivism (aka US 'Liberalism'): Ideas so good they need a police/surveillance-state to enforce.
That's an interesting theory. It has one fatal flaw. every time one of those "tax cuts for the rich" got passed, the percentage of federal revenue paid by the top 1% went up.
The truth is that all men having power ought to be mistrusted. James Madison
The problem is that that assertion doesn't line up with reality. Go down to your DMV some time, and observe the kinds of systems that they're using. They're using databases built in the 80s and 90s on top of DOS, running on ancient computers with CRT monitors (at least around here).
...And is that a problem? Does the thickness of the monitor really impact how legibly they can print your drivers' license?
What reasonable business do you know of that hasn't upgraded their systems since that time to allow for more efficiency savings, faster processing, reduced staff costs etc?
As one example, in the mid-2000s, I worked at a company whose main computer was built in 1988, with only minor upgrades (disk capacity, and a modem that was occasionally plugged in so it could be maintained remotely) since its construction. It had survived the obsolescence of its product line, the rise of DOS and Windows, and had only a minor stumble for Y2K. For a system whose primary purpose was tracking orders moving through departments, and tracking employees' time cards, it did the job perfectly well. That particular company was in the top 10% of the industry by order volume and profits, so it seems to have done just fine by most standards of "reasonable".
There's a lack of investment in this kind of system, plain and simple, being disguised as "government efficiency" by the republicans.
Again, to show the other perspective, there is grossly excessive spending in other kinds of systems, being disguised as "upgrades" by the Democrats.
I'm not promoting any particular political party here. Rather, my point is to illustrate that every partisan criticism in this thread has an equally-valid counterpoint that is too-often glossed over. When the Republicans shout about "spending", the Democrats shout "obsolescence". Nobody ever seems to want "get what's useful and nothing more", or "review the cost/benefit analysis for every component in the system".
I've worked for the federal government before, notably on one particular system whose lifespan was about 20 years. The system was designed and built to be state-of-the-art, using top-of-the-line COTS hardware available at the time (as a cost-saving measure, naturally). Ten years into the system life, those original components were obsolete, and being replaced with new top-of-the-line hardware, with the promises you mentioned: efficiency savings, faster processing, reduced costs, et cetera.
However, the basic workflow hadn't changed at all, and the software hadn't been rewritten (as that'd be prohibitively expensive), but only ported up to newer technologies. Even though each part of the process was indeed faster, the system as a whole hadn't changed significantly. It could run perfectly fine on modern (for the day) mid-grade or even low-end hardware, but because "upgrades" were seen as desirable, the system continued to be built with top-of-the-line parts, for about triple the cost.
Towards the end of the project lifespan, there was an effort to re-engineer it using minimal hardware, but by that point the idea had grown into something of a legend. The managers (and bureaucrats) who had seen the system's early versions and knew its original cost couldn't believe the system could actually run on such a low hardware budget. Every actual test was successful, but the mantra that "you get what you pay for" had become such an integral part of common sense that actually getting approval for a cost-efficient system was impossible. Eventually, my team ended up inflating our quoted costs to get approval, then delivering a working system under budget and getting extra praise.
That tale doesn't meet my idea of "reasonable", but it was definitely the reality that I saw.
You do not have a moral or legal right to do absolutely anything you want.
...And is that a problem? Does the thickness of the monitor really impact how legibly they can print your drivers' license?
Yes, it is. It indicates that their systems are so old as to require special purpose hardware. The only reason that places use this kind of monitor and computer any more is because they don't have enough money to replace everything outright. In the long run, this causes higher costs, not lower, because they end up needing to seek out compatible hardware to keep things running, rather than being able to wholesale upgrade. If you go and look at companies, you'll find that almost every company has a 3 year rolling refresh of hardware, not because they think it's fun to upgrade hardware, but because it's cheaper to keep rolling it along than it is to end up with the costs of maintaining outdated systems, and then a giant all-around upgrade.
As one example, in the mid-2000s, I worked at a company whose main computer was built in 1988, with only minor upgrades (disk capacity, and a modem that was occasionally plugged in so it could be maintained remotely) since its construction. It had survived the obsolescence of its product line, the rise of DOS and Windows, and had only a minor stumble for Y2K. For a system whose primary purpose was tracking orders moving through departments, and tracking employees' time cards, it did the job perfectly well. That particular company was in the top 10% of the industry by order volume and profits, so it seems to have done just fine by most standards of "reasonable".
Sure, one single system in the back of one company did not get upgraded. I'd be willing to bet that the reason it didn't get upgraded was simple - it had got so old that it was at this point a major pain, and a major cost to upgrade. Furthermore, how many of the systems sat on the desks of average employees were that old? Care to take a guess at the reason?
When I look at the DMV, and I look at the DVLA in the UK (where I'm from originally), I see this... ...
The act of applying for a provisional driving license takes about 3 hours for the customer, and about 20 minutes of the time of various employees in a DMV building. It takes a bunch of literal paper pushing, and probably a bunch more employee time in the back office. In the UK, this is 5 minutes of the customer's time to fill on a form on the internet, and no time spent by employees at all (bar the amortised cost of the guys running the IT system and database).
The act of booking a test - again, a bunch of time for the customer, a bunch of time for the DMV.
The act of carrying out a theory test - more time for the customer hanging about in queues at the DMV building, a paper test, which is handed out by employees, and marked buy an employees. More time wasted! In the UK, the theory test is taken on a computer system, and marked automatically. No need for anyone there other than one single receptionist. All booked online, no queueing.
The act of
The list continues. Every single interaction with the DMV involves 3 hours of the customers time, 20-30 minutes of the time of various employees filling out and stamping forms, and all of this has to happen in a pretty large building which has to be maintained. Those buildings have to be regularly spread out all over the place, because the amount of time taken is huge.
Meanwhile, the DVLA manages to process all this, with far far fewer employees, because they actually had some investment in setting up database systems and web pages so that most of the job can be automated. Basically, the whole DMV is hugely inefficient because no one has ever invested in the system. All because the republicans busy trying to find cost savings that will look good on paper in the next 4 years.
I'm not promoting any particular political party here. Rather, my point is to illustrate that every partisa
Please try reading.
It indicates that their systems are so old as to require special purpose hardware.
A VGA monitor is now considered "special-purpose hardware"?
Having a CRT monitor indicates only that the system is compatible with a CRT monitor. If you're making further assumptions about the system's capabilities based on the age of a peripheral device, that's your fault, not the system's.
As one example, in the mid-2000s, I worked at a company whose main computer was built in 1988...
Sure, one single system in the back of one company did not get upgraded.
No, that was the main system running the whole industry-leading company.
I'd be willing to bet that the reason it didn't get upgraded was simple - it had got so old that it was at this point a major pain, and a major cost to upgrade.
That's only half of it. The other half was that it wouldn't bring any benefit. The company's production was limited by physical processes and market demand, not the computer's record-keeping.
Furthermore, how many of the systems sat on the desks of average employees were that old?
Outside of the customer service area (who had shiny new Windows XP boxes, with DSL Internet access!), there were three other new computers in the company, all for special-purpose workstations that needed to do processing-intensive tasks. Most desks had VT terminals (ranging from VT300s to VT520s) to connect to the mainframe.
Care to take a guess at the reason?
I'll go with "the cost/benefit analysis did not support an upgrade", since that was the CEO's answer when I asked. Each department did one thing, and one thing only. The system already existed, and was known to work well for the necessary tasks. The company had the source to the software, and made software changes when necessary to support improved workflows, but for the most part the process was mature.
It takes a bunch of literal paper pushing, and probably a bunch more employee time in the back office.
So it's not actually related to the CRT monitors?
In the UK, this is 5 minutes of the customer's time to fill on a form on the internet, and no time spent by employees at all (bar the amortised cost of the guys running the IT system and database).
...that you know of. Realistically, there could be a herd of paper-pushers in the back end that you'd never know about, because you're getting distracted by the shiny interface.
The act of ...
Let me just interrupt this rant with "your mileage may vary". The last time I went to the DMV, it was for a full re-issue of a driver's license after a relocation, and required a test. The whole process, from entering the building to walking out, took about an hour.
After the queue, the agent scanned my old license to read the data, checked it for accuracy, and sent it to the back for processing while I waited for an available test machine. The tests were administered on kiosks built around CRT touchscreens, that looked like they had been operating since I was using that aforementioned mainframe. One test machine was being serviced, and I noticed that the kiosk was just a commodity desktop PC running Windows 7. The PC had a small form factor case, sitting in a cabinet just the right size for a full tower. Clearly, the machine had been upgraded, but the cabinet and interface was original.
By the time I had finished the test, my forms had been processed, and the agent handled the registration of my vehicle while my license was being printed. The agent submitted the vehicle paperwork to be processed while retrieving the license and handling payment. Once the vehicle processing was finished, I was handed new vehicle plates and wished a pleasant day.
You do not have a moral or legal right to do absolutely anything you want.
Hardly the green screen era for NEW software. Maybe the DOJ got their dinoware in 1997 but the sw is not from that year.
this is why I wish Slashdot still had an edit function. Still not the green screen era though
I think you may be looking at this the wrong way. Most US states have invested in online DMV activities so that you don't actually have to go there and wait. The only time I've had to go there is when importing a vehicle from out of state. It doesn't make sense to update a system that is seeing less and less usage. The investment is going into the web-based channel. At some point it won't even make sense to have physical DMV locations so why put money into upgrading the facilities or equipment?
It's always a matter of priorities. The FBI would rather work for their corporate overlords than do something for the public.
FTFY
I wasnt defending aging tech, i was defending useful tech. Nothing wrong with green screen and mainframes if its useful and not a burden. I get what you are saying, but its exactly the reason i commented, too many people want change for change's sake.
Good-bye
As if Democrats don't cut budgets? Why would you think that Republicans cut the DOJ's budget so that they can't afford to run the better system that is already paid for?
APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?