Slashdot Mirror

← Back to Stories (view on slashdot.org)

US Monitoring Database Reaches Limit, Quits Tracking Felons and Parolees

Posted by timothy on from the paid-by-the-row dept.

An anonymous reader writes "Thousands of US sex offenders, prisoners on parole and other convicts were left unmonitored after an electronic tagging system shut down because of data overload. BI Incorporated, which runs the system, reached its data threshold — more than two billion records — on Tuesday. This left authorities across 49 states unaware of offenders' movement for about 12 hours." As the astonished submitter asks, "2 billion records?"

15 of 270 comments (clear)

  1. 2 billion... by onion2k · · Score: 4, Interesting

    Assuming that's a normal "US" billion, and assuming it's a journal of historical data going back a few years, I don't think it's unreasonable to think there could be information in there on a couple of hundred thousand people each of whom has been track for an average of at least 6 months. So, approximately and with some guesses, that's around 55 records per prisoner per day. 1 update every 30 minutes? That sounds about right, maybe a little on the low side if anything.

    What is surprising is that they were running some sort of database process that maxxed out at 2 billion records, and that it just stopped once it hit that limit rather than failing over to a backup process. But then, this is a government IT contract, so maybe it's not too surprising.

    --
    http://twitter.com/onion2k
    1. Re:2 billion... by Statecraftsman · · Score: 4, Interesting

      If you track 16000 people and store their location once per second, you'll only need 1.55 days to reach 2^31 records. Once per minute only gives you 90 days. Once every 10 minutes, less than 3 years... I wonder if anyone is on the user end of this system that can comment.

    2. Re:2 billion... by Teancum · · Score: 2, Interesting

      Still, the complaint about how intelligently the software architecture was put together is seriously put into question as those who designed the system really didn't think through how long their software would last or what kinds of records were being put into the system. I understand how IPv4 had unanticipated problems with billions of computers on a network originally designed to handle merely hundreds and when v4 came out it was still in the mere thousands of computers being connected. In this design, it sounds like it was almost by design going to eat up a whole bunch of records.

      Besides, people have been bitching about IPv4 running out for decades and have anticipated the problem by introducing IPv6 quite some time ago. Any competent software engineer should have seen something like this coming years ago, so when I see something like "running out of space" I can only assert either:

      • The software developers on the software were incredibly incompetent and deserve to be fired.
      • The management of the company involved doesn't know jack about what it is that they are doing, likely hiring the developers on a short-term contract or they fired the competent engineers somehow along the way.

      Either way, it certainly doesn't inspire confidence in this company, and they certainly seem to be in way over their head here. If you hire a bunch of developers from Waziristan because they low bid on the development contract, you get what you pay for. This certainly isn't going to be the only problem with the software coming from this company as rookie mistakes like this are likely to be the tip of the iceberg.

      Yes, this is a rookie mistake I would expect out of a freshman CS student, not somebody trying to sell a supposed professional service.

    3. Re:2 billion... by hedwards · · Score: 2, Interesting

      That depends upon how they set it up. More likely they're storing the actual record information for the person in that DB with the location data for the person's record in a separate DB. The reason I say that is that, space issues aside, a DB being run like you suggest would be slower than hell, and be a PITA to keep optimized.

      Whereas you'd have plenty of room to store the data if the person has their own DB for the location. You'd have something like 68 years to work with. That's fairly close to an entire lifetime.

    4. Re:2 billion... by Grave · · Score: 2, Interesting

      Absolutely correct. However, 16,000 offenders being tracked.. 2 billion records.. Approximately 125,000 records per offender? I suppose it depends on the sort of data they are recording, and over what duration it needs to remain in an active (and not archived) state, but that just seems like an awful lot.

      I guess the upside to this is that we know the US government (and it's contractors) can't actually track all ~300 million citizens with any sort of accuracy or utility then. There simply isn't enough brainpower working in government IT to build a usable system that could track us like that. Google or Facebook, though..

  2. about 16000 by roman_mir · · Score: 4, Interesting

    Prisons and other corrections agencies were blocked from getting notifications on about 16,000 people, BI Incorporated spokesman Jock Waldo said on Wednesday.

    - interesting number. Anyway, it's not about the number of people in the database, it's about some number of records associated with each person presenting their location, so probably GPS coordinates taken at some time intervals.

    Also note that they are still logging the data, they just can't read it, so it's an application for displaying the coordinates that is failing. Quite possible that the actual problem is in filtering the data, maybe they are just trying to view data for an entire time period per person rather than looking at latest records, something like: 'last month only'. But this is, in the words of infamous W, 'speculaaation'.

    --
    You can't handle the truth.
  3. 32 bit signed integer strikes again by Co0Ps · · Score: 4, Interesting

    2 billion? That's awkwardly close to 2147483647... This is why your ID field should be BIGINT and not INT.... They where probably logging coordinates etc.

  4. Data loss is not guaranteed by Ebbesen · · Score: 2, Interesting

    I'm not sure any data has been lost. Say they have a table with the following columns:

    id (auto increment)
    felonid
    gps
    timestamp ...

    If the 2 billion number is simply id that has run over, there's still enough data in the database to recreate the felons whereabouts using the gps and timestamp columns. Might be a problem in the system pulling data (based on id), but probably no data has been lost.

  5. Lemming database design by Kupfernigk · · Score: 2, Interesting
    We consistently see this in monitoring systems designed by other companies. In our own monitoring systems we make extensive use of sparse appends (i.e. data only gets added to when there is a significant change, and we maintain a timestamp for the last update for each entity being monitored so we know that monitoring is actually taking place.) Of course this puts a lot more up front effort into actual system design.

    There seems to have been a period, roughly when hard drive capacity was rising more rapidly than application demands for data, when nobody cared too much. Before that, backing store was limited and we had to worry about data size. Now, application data sets are growing enormous even for quite trivial applications, and we need to worry about keeping data storage in bounds again.

    --
    From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
    1. Re:Lemming database design by MichaelSmith · · Score: 2, Interesting

      And maybe they don't need every GPS position in the database. It could be there just to cover for legal requirements in which case they could append it to a binary file and open a new file every day. Compress the old files with bz2 and archive all data more than a year old.

      --
      http://michaelsmith.id.au
  6. Hmmmm by luis_a_espinal · · Score: 3, Interesting

    Sharding? Partitioning? But most importantly, using 64bit int types (or bigger) rather than 32-bit ints for primary indexes? I mean, what the hell they were using to store that data anyways? A Visicalc spreadsheet running on a TRS-80?

  7. Quickbase would be my guess by Sycraft-fu · · Score: 3, Interesting

    It seems to be the crap database of choice these days, especially for consulting companies. Friend of mine got a job not long ago as a consultant for a consultant. Yes really, he consults for a consulting firm. Not like he is someone they hire out, he is a consultant they hire to work on jobs they've been hired to work on. The thing that got him the job was his Quickbase experience. This company loves them some Quickbase for some reason. However they are always bashing in to limits it has. Had they used MSSQL or Oracle they'd be fine, but they didn't. So a major thing he does is work around those limits in various creative ways. Retarded, but that's what they want and they'll pay for it.

  8. Re:Well no wonder by maxume · · Score: 3, Interesting

    Yes, that was the joke. See, GP poster is implying that even though the system should have been using something designed for the load, since it is a government contract, they used Access.

    --
    Nerd rage is the funniest rage.
  9. Re:Well no wonder by assertation · · Score: 2, Interesting

    If you haven't, rent a copy of the documentary "Hacking Democracy".

    Diebold chose to use MS Access as the backend for voting machines

  10. Re:then the criminals win by assertation · · Score: 2, Interesting

    I don't see those people as criminals, at least not with a capital 'C'. I'm straightedge but I don't see smoking pot as being any worse than alcohol. I would rather have my crime fighting dollars go to jailing thieves, murderers, rapists, *narcotic* dealers, etc. Not someone doing something the equivalent of having or selling a drink.