Slashdot Mirror
US Monitoring Database Reaches Limit, Quits Tracking Felons and Parolees
An anonymous reader writes "Thousands of US sex offenders, prisoners on parole and other convicts were left unmonitored after an electronic tagging system shut down because of data overload. BI Incorporated, which runs the system, reached its data threshold — more than two billion records — on Tuesday. This left authorities across 49 states unaware of offenders' movement for about 12 hours."
As the astonished submitter asks, "2 billion records?"
They just need to upgrade it so they can track the other 4 billion properly.
Damn sick criminals! ALL OF THEM.
MS Access can't possibly handle 2 billion records, no matter how much hardware you throw at it.
Did you know that "FTW" ("for the win") is a direct translation of "Sieg Heil"?
BI increased its data storage capacity to avoid a repeat of the problem.
ONCE AND FOR ALL.
Ice Cream has no bones.
Seems like it took them a few hours to change the key column from unsigned +/- 2^31 to signed 0-2^32-1
Assuming that's a normal "US" billion, and assuming it's a journal of historical data going back a few years, I don't think it's unreasonable to think there could be information in there on a couple of hundred thousand people each of whom has been track for an average of at least 6 months. So, approximately and with some guesses, that's around 55 records per prisoner per day. 1 update every 30 minutes? That sounds about right, maybe a little on the low side if anything.
What is surprising is that they were running some sort of database process that maxxed out at 2 billion records, and that it just stopped once it hit that limit rather than failing over to a backup process. But then, this is a government IT contract, so maybe it's not too surprising.
http://twitter.com/onion2k
Prisons and other corrections agencies were blocked from getting notifications on about 16,000 people, BI Incorporated spokesman Jock Waldo said on Wednesday.
- interesting number. Anyway, it's not about the number of people in the database, it's about some number of records associated with each person presenting their location, so probably GPS coordinates taken at some time intervals.
Also note that they are still logging the data, they just can't read it, so it's an application for displaying the coordinates that is failing. Quite possible that the actual problem is in filtering the data, maybe they are just trying to view data for an entire time period per person rather than looking at latest records, something like: 'last month only'. But this is, in the words of infamous W, 'speculaaation'.
You can't handle the truth.
"According to the U.S. Bureau of Justice Statistics (BJS): "In 2008, over 7.3 million people were on probation, in jail or prison, or on parole at year-end — 3.2% of all U.S. adult residents or 1 in every 31 adults."
This doesn't make me feel safe.
The actual data was only about 500K. The rest was XML markup.
One "Aw, Shit!" is worth 100 "Ata boys!"
2 billion? That's awkwardly close to 2147483647... This is why your ID field should be BIGINT and not INT.... They where probably logging coordinates etc.
I'm not sure any data has been lost. Say they have a table with the following columns:
id (auto increment) ...
felonid
gps
timestamp
If the 2 billion number is simply id that has run over, there's still enough data in the database to recreate the felons whereabouts using the gps and timestamp columns. Might be a problem in the system pulling data (based on id), but probably no data has been lost.
There seems to have been a period, roughly when hard drive capacity was rising more rapidly than application demands for data, when nobody cared too much. Before that, backing store was limited and we had to worry about data size. Now, application data sets are growing enormous even for quite trivial applications, and we need to worry about keeping data storage in bounds again.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
Anyone remember when Slashdot hit 16,777,215 comments, and overflowed MEDIUMINT? The ALTER TABLE statement that fixed it took hours to run. I shudder to think how long it'll take to fix this, even with the problem diagnosed.
There's no failure quite as dissatisfying as a complete and total solution to the wrong problem.
And you are clearly completely unaware of the accounting world.
I have yet to meet an accountant that knows much of anything about access or any other database system. On the other hand the majority of them have complained about the 65000 line limit in excel.
They ALL do this. You're telling thousands of accountants to change how they do things, and honestly, not for the better. They know how to use excel and know how to make things balance with excel.
A large portion of them took accounting because it was supposed to make them a lot of money, these people don't even use 1/10th of the functionality provided in excel, lets not try to make them learn another entirely different software skill set, ok?
Even if you're currently working in IT and are like "Oh, no, our accountants have access to all this stuff in our system and they would never do that". Trust me, they do. It all ends up in an excel sheet somewhere eventually.
Sharding? Partitioning? But most importantly, using 64bit int types (or bigger) rather than 32-bit ints for primary indexes? I mean, what the hell they were using to store that data anyways? A Visicalc spreadsheet running on a TRS-80?
It seems to be the crap database of choice these days, especially for consulting companies. Friend of mine got a job not long ago as a consultant for a consultant. Yes really, he consults for a consulting firm. Not like he is someone they hire out, he is a consultant they hire to work on jobs they've been hired to work on. The thing that got him the job was his Quickbase experience. This company loves them some Quickbase for some reason. However they are always bashing in to limits it has. Had they used MSSQL or Oracle they'd be fine, but they didn't. So a major thing he does is work around those limits in various creative ways. Retarded, but that's what they want and they'll pay for it.
Maybe the answer isn't better software, but fewer criminals to fill up the database with.
I keep seeing articles here and there how the U.S. has more people imprisoned than China. A large chunk of the prison population are inmates convicted of drug crimes and a large portion of that set of people were convicted on marijuana laws.
I don't smoke, but as a tax payer I would rather see the government make marijuana into a tax revenue generator instead of a huge expense to paid for with taxes.
And you are clearly completely unaware of the contracting world.
I have yet to meet a contractor that knows much of anything about screwdrivers or any other tool than a hammer. On the other hand the majority of them have complained about how hard it is to drive screws with the hammer.
They ALL do this. You're telling thousands of contractors to change how they do things, and honestly, not for the better. They know how to use a hammer and know how to drive nails.
A large portion of them took contracting because it was supposed to make them a lot of money, these people don't even use 1/10th of the functionality provided by a hammer, lets not try to make them learn another entirely different tool skill set, ok?
Even if you're currently working in contracting supply and are like "Oh, no, our contractors have access to all this stuff and they would never do that". Trust me, they do. It all ends up pounded by a hammer somewhere eventually.
2 billion offenders tracked should be fine, as there are only about 300 million people in the US. But 2 billion locations? Someone needs a real database. Or a chron job to archive these puppies.
I'm going to go out on a limb here and say that if you think it's spelled "chron", you probably shouldn't be making suggestions on this subject.
There is no authorization in the constitution for laws that control what you do personally or consensually. The criminals, as Mark Twain told us, are in the legislature.
And as long as the government is out of compliance with the constitution, the government is a criminal organization. Law-breakers and oath-breakers, both.
I've fallen off your lawn, and I can't get up.
The limit has been raised to about a million for a few versions now: http://office.microsoft.com/en-us/excel-help/excel-specifications-and-limits-HP010073849.aspx
The ghost of Alan Turing wants a word with you.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
Excel + ODBC + Oracle/Postgres/MySQL/whatever = warehouseable accounting data with no change in user experience.
No OS on the planet can protect itself from a user with the admin password. - Yvan256
You obviously don't know the meaning of physical castration, the effects of chemical castration, and the varieties of violent sex offences.
Physical castration is simple removal of the testicles. You can still gain and maintain an erection after such a procedure on a normal male. Rape is still possible, especially since it's less about sex, and more about power, thus a sex drive reduction is immaterial to the process.
Chemical castration prevents erections, as long as you're still taking the drugs. Miss a dose, and you're operational again fairly quickly. However, it doesn't stop sex drive at all, nor does it curb aggressive behaviour, so foreign object rape is still possible, which is usually much more damaging to the victim. Also, the chemicals required are *really* fucking expensive.
And your plan doesn't cover female sex offenders in any way, shape, or form. Please, before you spout off idiocy, make sure it's actually idiocy that stands some hope in hell of actually working, instead of just inflicting it on those of use who use our brains as more than a way to keep our ears separated.
Canada: The US's more awesome sibling.
I don't see those people as criminals, at least not with a capital 'C'. I'm straightedge but I don't see smoking pot as being any worse than alcohol. I would rather have my crime fighting dollars go to jailing thieves, murderers, rapists, *narcotic* dealers, etc. Not someone doing something the equivalent of having or selling a drink.
You're missing the point. They don't want a solution - they want to quickly look at some data, make a decision based on it, and move on. No-one's going to pay for/install an app; deploy stuff to a website etc. I can do the sql query, export to csv, import into Excel, spend 1 min on it and email it on. Done - on to my next task. Access is a piece of shit, and getting data from Oracle/SQL Server and putting it into Access is just bizarre, even if I could guarantee all end users would have it installed.
> Also, chances are, if you think any typical business data set is best represented by a spreadsheet, you are probably not qualified to make the call.
You're not qualified to make this call if you can't understand that I'm giving the users what they want, quickly and cheaply; they'd tell me if it wasn't any use. They want the numbers, not a "solution".