Slashdot Mirror
US Monitoring Database Reaches Limit, Quits Tracking Felons and Parolees
An anonymous reader writes "Thousands of US sex offenders, prisoners on parole and other convicts were left unmonitored after an electronic tagging system shut down because of data overload. BI Incorporated, which runs the system, reached its data threshold — more than two billion records — on Tuesday. This left authorities across 49 states unaware of offenders' movement for about 12 hours."
As the astonished submitter asks, "2 billion records?"
They just need to upgrade it so they can track the other 4 billion properly.
Damn sick criminals! ALL OF THEM.
Visual Basic for the FAIL, signed int? They really should know better.
MS Access can't possibly handle 2 billion records, no matter how much hardware you throw at it.
Did you know that "FTW" ("for the win") is a direct translation of "Sieg Heil"?
BI increased its data storage capacity to avoid a repeat of the problem.
ONCE AND FOR ALL.
Ice Cream has no bones.
Is it that hard to think of having multiple rows per offender perhaps?
Seems like it took them a few hours to change the key column from unsigned +/- 2^31 to signed 0-2^32-1
2 billion records ought to be enough for anybody.
Assuming that's a normal "US" billion, and assuming it's a journal of historical data going back a few years, I don't think it's unreasonable to think there could be information in there on a couple of hundred thousand people each of whom has been track for an average of at least 6 months. So, approximately and with some guesses, that's around 55 records per prisoner per day. 1 update every 30 minutes? That sounds about right, maybe a little on the low side if anything.
What is surprising is that they were running some sort of database process that maxxed out at 2 billion records, and that it just stopped once it hit that limit rather than failing over to a backup process. But then, this is a government IT contract, so maybe it's not too surprising.
http://twitter.com/onion2k
Sounds like an int overflow ;)
Prisons and other corrections agencies were blocked from getting notifications on about 16,000 people, BI Incorporated spokesman Jock Waldo said on Wednesday.
- interesting number. Anyway, it's not about the number of people in the database, it's about some number of records associated with each person presenting their location, so probably GPS coordinates taken at some time intervals.
Also note that they are still logging the data, they just can't read it, so it's an application for displaying the coordinates that is failing. Quite possible that the actual problem is in filtering the data, maybe they are just trying to view data for an entire time period per person rather than looking at latest records, something like: 'last month only'. But this is, in the words of infamous W, 'speculaaation'.
You can't handle the truth.
"According to the U.S. Bureau of Justice Statistics (BJS): "In 2008, over 7.3 million people were on probation, in jail or prison, or on parole at year-end — 3.2% of all U.S. adult residents or 1 in every 31 adults."
This doesn't make me feel safe.
Is it odd that I feel secure in their incompetence?
The actual data was only about 500K. The rest was XML markup.
One "Aw, Shit!" is worth 100 "Ata boys!"
2 billion sounds like the records are numbered with signed long's as they have a range of -2 147 483 648 to 2 147 483 647. It would explain why the system just fell over rather than slowing grinding to a halt, anyway.
2 billion? That's awkwardly close to 2147483647... This is why your ID field should be BIGINT and not INT.... They where probably logging coordinates etc.
obviously, this is false! the system must have crashed "accidental" when some criminals were in the middle of a heist!
I'm not sure any data has been lost. Say they have a table with the following columns:
id (auto increment) ...
felonid
gps
timestamp
If the 2 billion number is simply id that has run over, there's still enough data in the database to recreate the felons whereabouts using the gps and timestamp columns. Might be a problem in the system pulling data (based on id), but probably no data has been lost.
Looks like the ID type of the row in the database was a signed integer, overflowed from ~2 billions.
In this case the problem would be a simple database structure design problem, using the wrong type (with heavy consequences).
On MySQL instead of int, using bigint would have allowed up to the theoretical value of 2^63-1 records.
Slashdot, fix the reply notifications... You won't get away with it...
Also, chances are, if you think any typical business data set is best represented by a spreadsheet, you are probably not qualified to make the call.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
There seems to have been a period, roughly when hard drive capacity was rising more rapidly than application demands for data, when nobody cared too much. Before that, backing store was limited and we had to worry about data size. Now, application data sets are growing enormous even for quite trivial applications, and we need to worry about keeping data storage in bounds again.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
With her traditional keg stand.
Anyone remember when Slashdot hit 16,777,215 comments, and overflowed MEDIUMINT? The ALTER TABLE statement that fixed it took hours to run. I shudder to think how long it'll take to fix this, even with the problem diagnosed.
There's no failure quite as dissatisfying as a complete and total solution to the wrong problem.
Sharding? Partitioning? But most importantly, using 64bit int types (or bigger) rather than 32-bit ints for primary indexes? I mean, what the hell they were using to store that data anyways? A Visicalc spreadsheet running on a TRS-80?
It seems to be the crap database of choice these days, especially for consulting companies. Friend of mine got a job not long ago as a consultant for a consultant. Yes really, he consults for a consulting firm. Not like he is someone they hire out, he is a consultant they hire to work on jobs they've been hired to work on. The thing that got him the job was his Quickbase experience. This company loves them some Quickbase for some reason. However they are always bashing in to limits it has. Had they used MSSQL or Oracle they'd be fine, but they didn't. So a major thing he does is work around those limits in various creative ways. Retarded, but that's what they want and they'll pay for it.
Maybe the answer isn't better software, but fewer criminals to fill up the database with.
I keep seeing articles here and there how the U.S. has more people imprisoned than China. A large chunk of the prison population are inmates convicted of drug crimes and a large portion of that set of people were convicted on marijuana laws.
I don't smoke, but as a tax payer I would rather see the government make marijuana into a tax revenue generator instead of a huge expense to paid for with taxes.
We don't know!
(check the name of the BI Inc spokesman)
For repeat or first-time violent sex offenders, castration seems like a much better solution overall. Either physical or chemical, the positive results for both the offender and society are well proven alternatives to lifelong incarceration, tracking, monitoring, etc. In addition to helping protect society, it would also be less expensive, a nice additional bonus for our society.
2 Billion. That is a lot. It simply boggles the mind that many governments today think it is OK to rigidly restrict mostly harmless activities of law abiding citizens, yet are so resistant to taking effective steps to prevent or moderate one of the worst sorts of human crimes. In the US, politicians find it convenient to ignore constitutional issues when it comes to increasing police powers and controlling basic citizen activities, but they shirk their responsibilities and make all sorts of claims why they can't do something to actually protect society from our worst criminals.
As far as I can tell the population of the USA (http://www.wolframalpha.com/input/?i=population+of+USA) is around 312 million.
Which is not even close to 2 billion....
So how can the US be monitoring over 6 times it population?
Troll is not a replacement for I disagree.
maybe the update got delayed like that fly attendant system tracking / scheduling system that crashed when it hit to many changes for a month it has placed to be replaced / updated but due to budgets cuts that was pushed back.
One record per second for the year, tracking your location, 31 million records per day, per offender. A decade of records, a dozen offenders, 3 billion records. Oops ... they're not even tracking them that accurately. Maybe they're only tracking minute by minute, but for a couple of thousand offenders. I sure hope it's not hour by hour, it'd be too easy to get to a school, do something horrible, and get back before the system would notice.
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking
There is no authorization in the constitution for laws that control what you do personally or consensually. The criminals, as Mark Twain told us, are in the legislature.
And as long as the government is out of compliance with the constitution, the government is a criminal organization. Law-breakers and oath-breakers, both.
I've fallen off your lawn, and I can't get up.
In another note the person tasked with running Nagios against the datastore suggested someone comes up with an OS/2 client quick!
Access *is* limited to 2 billion bytes.
I assume they are mistakenly equating bytes with records.
The ghost of Alan Turing wants a word with you.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
Look at that. 25% of federal inmates are in there for drug possession. I bet you a good amount of these people wouldn't rob you at gunpoint.
Well, the drug laws are wrong and ineffective. However, that tells you that drugs are not the primary reason for the high US incarceration rate, because even 75% of 3.2% is still high.
So, what do those numbers mean? If you look at the statistics, there is actually more crime overall in Europe than in the US, and the rates are increasing in Europe and decreasing in the US. Furthermore, much of the US crime is concentrated in specific populations--30% of it alone among illegal aliens. That means that if you're a middle class American--the kind likely to vote--your risk of being a crime victim is even lower compared to Europe. Conversely, Europe has higher crime statistics despite having much lower illegal alien populations, better social services, less wealth disparity, supposedly better in-prison rehabilitation, and fewer immigrants.
The primary reason for higher incarceration rates in the US are longer sentences. Harsher drug laws also contribute. I bet more effective policing also does contribute. But given the crime statistics in the US and Europe, European arguments that the US should reduce its sentences are not actually all that convincing.
Personally, I hope the US will decriminalize many drugs and rely more on long-term out-of-prison monitoring for non-violent offenders; that should save money and allow people to re-integrate into society. I don't think the simplistic European approach of lighter sentences is going to be politically feasible, and it certainly has not shown to be effective.
If this was pushed back or delayed, that speaks volumes about the engineering management involved here that also doesn't care about their customers or is doing anything on software maintenance. Companies that ignore the need to at least keep a developer on existing projects fixing bugs and staying fresh on the code base are going to find in the long run that their software will simply suck. It is also a good indication that you need to flee a company selling such software that has been abandoned.
I've had managers who thought that owning a software company implied that they had a license to print money. In other words, once the software was written that you could fire the developers and keep copying the software laughing all of the way to the bank for each copy you sold. I'm suggesting that is never true except for very trivial pieces of software that aren't worth much in the first place.
But most importantly, using 64bit int types (or bigger) rather than 32-bit ints for primary indexes?
Some languages still have no 64-bit integer type. For example, in PHP compiled for a 32-bit architecture, anything bigger than a 32-bit signed integer is a float. Or in standard C++ compiled for a 32-bit architecture, anything bigger than a 32-bit unsigned integer is a float. (C++98, the current standard, came out before long long was added to C99, so if you see long long in a C++ program, it's a compiler-specific extension.)
There's no constitutional authorization to forbid private citizens from committing murder...
"No State shall make or enforce any law which shall abridge the privileges or immunities of citizens of the United States"
"No State shall" doesn't appear to apply to federal statutes. Marijuana prohibition in the United States is a federal statute, and U.S. courts have ruled it a constitutionally acceptable "regulat[ion of] commerce ... among the several States".
if you give in like that then the criminals win
every anarchist is a baffled dictator. Benito_Mussolini
But most importantly, using 64bit int types (or bigger) rather than 32-bit ints for primary indexes?
Some languages still have no 64-bit integer type. For example, in PHP compiled for a 32-bit architecture, anything bigger than a 32-bit signed integer is a float. Or in standard C++ compiled for a 32-bit architecture, anything bigger than a 32-bit unsigned integer is a float. (C++98, the current standard, came out before long long was added to C99, so if you see long long in a C++ program, it's a compiler-specific extension.)
In cases where you don't have a 64-bit int data type (or you know that your target architecture or compiled format does not support one), simply use two 32-bit (high and low halves) as an aggregate data type, detect when an overflow of the low half is to occur and carry the overflow bit up to the high half. This is sooooo extremely easy to do in C or C++, that it is not even funny. Similarly, if a DB does not have a 64-bit type, then use the same approach just described and use a compound key (high+low) as the primary key.
How many architectures out there support 128-bit ints? And yet there is a shitload of systems that support 128-bit "big int" arithmetic in the financial world.
This is a classic solution for implementing a N*m bit counter when the max you have is a N-bit type. It is a classic, textbook CompSci 101 problem that has been successfully and efficiently solved again and again and again using virtually the exact pattern I just described above.
Having an underlying bit-type architectural constrain has never been an impediment to implement arithmetic on much, much larger bit-types. It is not insurmountable, and in fact, one could argue that it is trivial (when done right, the later which is not that difficult either.)
It's true that China executes more prisoners than any other country, but the numbers are still relatively small - 14th place per capita when factoring in documented executions. The U.S.A. is 20th, per capita - 7th in raw numbers.
http://www.nationmaster.com/graph/cri_exe-crime-executions
China's 470 executions in a recent year wouldn't even make a dent in the U.S. prison population. Per capita, the U.S. would need about another 120 executions per year to match China's record, and that wouldn't be enough to uncrowd even one typical U.S. prison.
There are several factors here: the war on drugs is one significant contributor. If one includes both inmates who are being held for possession and inmates whose real crimes were only committed as a consequence of the drug war (e.g. they shot a competitor, or robbed someone to pay the inflated price for their fix, or robbed someone after they lost their job because they tested positive for marijuana on a drug test, etc.) then over half the U.S. prison population is there because of the drug war. If we include those who committed other victimless crimes or petty offences we are talking about an overwhelming majority that should never have gone to prison.
Unfortunately, they have gone to prison, and prison makes more prisoners. Once imprisoned, a person is more likely to commit more crimes. Children of prisoners are also much more likely to commit crimes and become prisoners themselves.
That said, most criminals are young men - and most of them age out of criminality. A few do not. They need to be locked up or otherwise prevented from committing more crimes. They are more likely to commit theft than other crimes - these are the people who have long rap sheets with low level offenses, but they are ultimately one of the two groups most damaging to society - the other group being those with poor anger management abilities.
Any attempt at reform will fail if it does not focus on these two most detrimental forms of criminality: violence and theft (including fraud). Even so, we will need to get away from zero tolerance policies - we should not consider a touch as an assault, nor should anyone ever have to feel concerned about the legality of defending themselves or others from attack. Mostly we need a little common sense, which is always in short supply.
most of them age out of criminality
I'm supposing that, in general, people tend to calm down as they mature; curtailing garden-variety criminal tendencies is a subset of this.
Likewise, some firebrand rappers or punks from the old(er) school seem more controlled now.
I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
WTF ? thought US had around 200 million total. That means an average 10 records / citizen ...