Slashdot Mirror

No, the problem is that the public sector does not operate anything at all like the private sector all the while trying to emulate it under the overhead and red tape that comes along with requiring the public's input.

In addition to the issues seen with how the public sector operates, we have the requirement of outsourcing to the private sector to do the bulk of work through private/public partnerships which the public sector cannot and will not effectively manage,

The competing interests of these partnerships leans heavily on the private sector to make loads of money while the public sector expects them to operate within the bounds of the red tape the private sector is not accustomed or willing to accept as part of their business model.

If the government took this upon themselves to do anything in its entirety, it would likely be done slowly but correctly. Unfortunately, we end up with the result we did: a quickly cobbled together, expensive, and poorly implemented product which would never have seen the light of day in the private sector.

This happens ALL THE TIME with public/private partnerships. Take a look at the website redesign for the City of Apple Valley, Minnesota which was originally budgeted at $76,000 but later reduced to a much more reasonable, although still incredibly expensive $30,000. The resulting site is basically unusable, slow, horrendous to update, and slightly more useless than its predecessor (lipstick on a pig).

No, the problem is that the public sector does not operate anything at all like the private sector all the while trying to emulate it under the overhead and red tape that comes along with requiring the public's input.

In addition to the issues seen with how the public sector operates, we have the requirement of outsourcing to the private sector to do the bulk of work through private/public partnerships which the public sector cannot and will not effectively manage,

The competing interests of these partnerships leans heavily on the private sector to make loads of money while the public sector expects them to operate within the bounds of the red tape the private sector is not accustomed or willing to accept as part of their business model.

If the government took this upon themselves to do anything in its entirety, it would likely be done slowly but correctly. Unfortunately, we end up with the result we did: a quickly cobbled together, expensive, and poorly implemented product which would never have seen the light of day in the private sector.

This happens ALL THE TIME with public/private partnerships. Take a look at the website redesign for the City of Apple Valley, Minnesota which was originally budgeted at $76,000 but later reduced to a much more reasonable, although still incredibly expensive $30,000. The resulting site is basically unusable, slow, horrendous to update, and slightly more useless than its predecessor (lipstick on a pig).

I have repeatedly requested camera views from publicly owned but privately operated buses in the southern suburbs of the Minneapolis/St Paul metro area.

These cameras exist both inside and outside of the buses but whenever an issue arises which negatively impact the bus drivers or the system itself, the camera feeds are unavailable, usually due to some sort of unknown malfunction: http://www.lazylightning.org/bus-2-0-directs-mvta-driver-onto-dirt-shoulder

However, when they are not at fault, the videos are available to me right away and without question: http://www.lazylightning.org/mvta-rider-alleges-racism-over-bus-incident

I have repeatedly requested camera views from publicly owned but privately operated buses in the southern suburbs of the Minneapolis/St Paul metro area.

These cameras exist both inside and outside of the buses but whenever an issue arises which negatively impact the bus drivers or the system itself, the camera feeds are unavailable, usually due to some sort of unknown malfunction: http://www.lazylightning.org/bus-2-0-directs-mvta-driver-onto-dirt-shoulder

However, when they are not at fault, the videos are available to me right away and without question: http://www.lazylightning.org/mvta-rider-alleges-racism-over-bus-incident

Back in late 2009 and early 2010 I was scraping jail inmate registry records for Scott and Dakota County, MN. This was simply a script which incremented the ID numbers by one several times a day and put them out into a CSV. I uploaded these to Google Docs and had Docs Widgets build simple charts based on those data for a rolling ~6 month window of inmates.

As I started looking deeper into the data I started noticing I had ages lower than 18. Odd I thought but sure enough, Scott County was including their juvenile records in the data mixed with the adults even though it wasn't shown on their public website.

I contacted the County and they fixed the bug (you can read about that here: http://www.lazylightning.org/scott-county-quickly-fixes-juvenile-jail-roster-issue) but I was still surprised at the relative lack of security for juvenile records:

Within mere minutes of my e-mail they were on the phone with me and informed me they closed the hole. After mentioning that the only way someone may have been able to retrieve a juvenile record is if they âoeguessedâ the booking number, I replied that the booking numbers are sequential and thus âoeguessingâ is as simple as incrementing by 1. After our short discussion they asked me to let them know immediately if I noticed anything else with their data and the call was ended.

It's surprising how lax security is anywhere and to the poster elsewhere in this thread that said this is what you get when you outsource to India, this particular web stuff was not performed with outsourced talent so that comment was nothing short of asinine.

Minnesota Valley Transit Authority buses are all equipped with multiple cameras both internally and externally. When I have either witnessed issues which could be considered safety hazards to the buses and/or the people riding, I have requested these feeds. Most of the time I am told the camera was not functioning at the time and they'd investigate why: http://www.lazylightning.org/mvtas-multimillion-dollar-bus-2-0-is-a-failure

When they were in the right they were quick to release the tape: http://www.lazylightning.org/mvta-rider-alleges-racism-over-bus-incident

This is exactly what will happen here.

Minnesota Valley Transit Authority buses are all equipped with multiple cameras both internally and externally. When I have either witnessed issues which could be considered safety hazards to the buses and/or the people riding, I have requested these feeds. Most of the time I am told the camera was not functioning at the time and they'd investigate why: http://www.lazylightning.org/mvtas-multimillion-dollar-bus-2-0-is-a-failure

When they were in the right they were quick to release the tape: http://www.lazylightning.org/mvta-rider-alleges-racism-over-bus-incident

This is exactly what will happen here.

I scour publicly available records for fun stuff all the time. I not only find it online but I also request it from government agencies (not Federal usually but local/county/etc).

In Minnesota data must be, "easily accessible for convenient use." While that has specific wording related to historical records, it basically means that on recent data it must be in some sort of electronic format or otherwise easily found and presented, free of charge as long as you do it in person, to anyone who asks--even anonymously. Now. This is great in theory. Unfortunately just because it's easy for the agency to use it doesn't mean it's easy for you to use or interpret.

Let's take for instance data on bus ridership data. It's not well organized for outsiders to read it and due to collection methodologies (not explained to the general person who had to pay $50 to get the data in the first place) is basically useless.

They have the data and after months of fighting with them for how much they claimed it cost (they wanted to charge me more than $300 IIRC) I got it down to $50 and got what you see above even though they already pulled it (and summarized it) for the mass media but wouldn't release it in a raw format.

So. It's in a format which isn't standard. It's methodology is questionable and it's expensive. So no matter the mandates, the promises, etc, the data is not terribly useful across agencies or to the public without some intermediate steps which costs the taxpayers more than doing it right the first time around.

I operate a website which has a local, loyal, knowledgeable, and (many time) contentious userbase. These people routinely post comments which get to the heart of the matter and because of the content of the site (restaurant reviews and local politics (county/city level)) can cause business owners and politicians to become upset.

One particular business, which isn't local yet but plans to be in the next year+, had some representatives post comments on the site and engage my readership. While I always track down new commenters to the best of my ability, especially to out astrotufing, these reps took the time to clearly announce who they were before I had to do any real work (thanks!)

However, after they posted some comments they realized two things:

1. They didn't sound very good.

2. They didn't like what my readers had to say.

As a result of this several exchanges went back and forth with them trying to get me to allow their comments and the comments of others to be edited/deleted.

Obviously the repeated answer I gave was "No."

---

People are learning about the Internet, especially PR, but for the most part they're very naive. I continually catch business owners or their reps trying to post astroturf comments in favor of the business and I happily out them. This happens on a continual basis and really brings into question sites such as Urbanspoon (which I actively support on my site) and Yelp.

If you're interested you can see one of my discussions about this here: http://www.lazylightning.org/astroturfing-poor-attempts-at-changing-opinion

However, if I am told by a court order to remove the comments, I will. I will go to bat for as long as I can before I have to put my financial stake on the line for a bunch of people on the Internet. I do my best to keep them anonymous (no long-term logging, allow them to use any non-bouncing e-mail address that they actually check) but I will only go so far.

Because the public sector has very little time to handle FOIA requests and they sometimes cost more money to complete than I'm willing to pay (usually because they don't do much of their own data work in-house and have to call on a contractor to do it for me), I use their websites to glean the data I want.

Last week I gave a talk about using SAS to do screen scraping and then perform analysis on the data of jail inmate registries and level 3 sex offenders in MN. I have dashboards of the data available on my website and as I mentioned in my presentation it has even been used to help one county avoid what could have been a serious privacy issue.

So while there are any number of pitfalls to screen scraping (not understanding the meaning of the data and trends, being fed incomplete or purposefully incorrect data, or even being banned outright) screen scraping can be great for learning about and reporting on the public sector when they are physically or financially incapable or simply unwilling to do it themselves.

I eat foods laced with the ghost pepper often wings and Caribbean to name two of my most favorite) simply because I enjoy eating spicy foods.

The problem with spicy foods is that you apparently gain tolerance to the effects. The first few times I had ghost wings I got an endorphin rush (light headed, arms went numb, etc) but subsequent tastings have provided much less response.

While I know there are plenty of extracts available which will permit me to add up to 15+ million Scoville units, I prefer to keep the fresh pepper taste (or at the very least an all natural experience) when eating my food.

Hot foods aren't for everyone. In fact, I don't recommend you jump right to the ghost pepper (or higher) if you cannot handle anything more than a jalapeno. But being that I really do enjoy eating pickled habaneros raw and adding ghost peppers to most foods, I personally believe that furthering natural Scoville ratings is a great thing.

I cannot wait to try something 50% hotter than the ghost chili. I'm sure it'll bring me right back to that high I experienced the very first time I tried them.

YMMV.

I eat foods laced with the ghost pepper often wings and Caribbean to name two of my most favorite) simply because I enjoy eating spicy foods.

The problem with spicy foods is that you apparently gain tolerance to the effects. The first few times I had ghost wings I got an endorphin rush (light headed, arms went numb, etc) but subsequent tastings have provided much less response.

While I know there are plenty of extracts available which will permit me to add up to 15+ million Scoville units, I prefer to keep the fresh pepper taste (or at the very least an all natural experience) when eating my food.

Hot foods aren't for everyone. In fact, I don't recommend you jump right to the ghost pepper (or higher) if you cannot handle anything more than a jalapeno. But being that I really do enjoy eating pickled habaneros raw and adding ghost peppers to most foods, I personally believe that furthering natural Scoville ratings is a great thing.

I cannot wait to try something 50% hotter than the ghost chili. I'm sure it'll bring me right back to that high I experienced the very first time I tried them.

YMMV.

Or how about newspapers? http://www.lazylightning.org/thisweek-ignores-creative-commons-licensing

Or rag mags? http://www.lazylightning.org/citypages-feels-copyright-infringement-deserves-a-one-time-pass

Or Esquire? http://www.flickr.com/photos/bill_roehl/5173042830/

Or restaurants? http://www.lazylightning.org/pardon-my-french-uses-photo-without-permission

This is simply the fact that people think that everything on the Internet is fair game for them to copy and use without the author's consent. If a bunch of people working at a newspaper, a respected magazine, and rag which have been around for 20+ years each don't know the laws in this country how are those from outside our boundaries to know?

Basically it sucks all around.

Or how about newspapers? http://www.lazylightning.org/thisweek-ignores-creative-commons-licensing

Or rag mags? http://www.lazylightning.org/citypages-feels-copyright-infringement-deserves-a-one-time-pass

Or Esquire? http://www.flickr.com/photos/bill_roehl/5173042830/

Or restaurants? http://www.lazylightning.org/pardon-my-french-uses-photo-without-permission

This is simply the fact that people think that everything on the Internet is fair game for them to copy and use without the author's consent. If a bunch of people working at a newspaper, a respected magazine, and rag which have been around for 20+ years each don't know the laws in this country how are those from outside our boundaries to know?

Basically it sucks all around.

Or how about newspapers? http://www.lazylightning.org/thisweek-ignores-creative-commons-licensing

Or rag mags? http://www.lazylightning.org/citypages-feels-copyright-infringement-deserves-a-one-time-pass

Or Esquire? http://www.flickr.com/photos/bill_roehl/5173042830/

Or restaurants? http://www.lazylightning.org/pardon-my-french-uses-photo-without-permission

This is simply the fact that people think that everything on the Internet is fair game for them to copy and use without the author's consent. If a bunch of people working at a newspaper, a respected magazine, and rag which have been around for 20+ years each don't know the laws in this country how are those from outside our boundaries to know?

Basically it sucks all around.

You're right private cameras are there. The cops have to jump through additional hoops to get the footage. If a business owner was aware of his (and his fellow citizens' rights) he'd tell the cops to get a warrant. Being that even public video footage isn't even worth watching to prosecute most crime because there is so much of it to go through, I'm guessing they'd ignore it for most crime unless it was super serious.

But hey, if the drone is flying they probably have someone watching the thing during its 40 minute flights and finding the crime going on. All this is going to do is pack our prisons with more unnecessary prisons while claiming it's protecting society.

You're right private cameras are there. The cops have to jump through additional hoops to get the footage. If a business owner was aware of his (and his fellow citizens' rights) he'd tell the cops to get a warrant. Being that even public video footage isn't even worth watching to prosecute most crime because there is so much of it to go through, I'm guessing they'd ignore it for most crime unless it was super serious.

But hey, if the drone is flying they probably have someone watching the thing during its 40 minute flights and finding the crime going on. All this is going to do is pack our prisons with more unnecessary prisons while claiming it's protecting society.

Because people commit crimes from outside the county but are included in the database. I track the addresses of criminals with complaints in my county and while the majority reside within the boundaries, there are the outliers who hail from all over the State of Minnesota (this is a rolling 30 day picture and is purposefully limited to only the MSP metro area for clarity's sake): http://www.lazylightning.org/dakota-county-criminal-complaints-mapped-again

I have mod points and I considered moderating your post but I couldn't decide to mark it down or up (it could easily work both ways) so I decided to respond instead:

Yes, ESRI shapefiles are prevalent in the public sector and the software is expensive and difficult to use. Thankfully you can now easily convert SHP files into KML and display a lot of the work done in the public sector for use everywhere else.

I have a small archive of stuff I have converted from SHP to KML in Minnesota available here which are used mainly for geocachers wanting to know what areas are off-limits but as you can see there are MN county boundaries in there as well. Why the county boundaries? Because the company I worked at prior to my current place for employment (a publicly funded educational institution) couldn't afford actual GIS software so I had to use what was freely available to do our analysis of marketing ROI.

So, yeah, ESRI sucks and they need to die but at least there are options now available which are affordable (read: "free") for the general public to get involved and do their own work with what has been made available from their tax dollars being hard at work.

They do it on blogs too. Unfortunately for them they can't capitalize on my site's high ranking for their restaurants because I track them down and expose them for what they are. See here: http://www.lazylightning.org/astroturfing-poor-attempts-at-changing-opinion

If other sites were smart they'd be doing the same things. The actual userbase is far more pleased with the service you provide when you don't allow that sort of bullshit.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with PostgreSQL.

2. Yes, I could use PostgreSQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in PostgreSQL I would have to make the CSV, pipe it into the PostgreSQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

There, fixed that for 'ya.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with PostgreSQL.

2. Yes, I could use PostgreSQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in PostgreSQL I would have to make the CSV, pipe it into the PostgreSQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

There, fixed that for 'ya.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with PostgreSQL.

2. Yes, I could use PostgreSQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in PostgreSQL I would have to make the CSV, pipe it into the PostgreSQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

There, fixed that for 'ya.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with PostgreSQL.

2. Yes, I could use PostgreSQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in PostgreSQL I would have to make the CSV, pipe it into the PostgreSQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

There, fixed that for 'ya.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with MySQL.

2. Yes, I could use MySQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in MySQL I would have to make the CSV, pipe it into the MySQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with MySQL.

2. Yes, I could use MySQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in MySQL I would have to make the CSV, pipe it into the MySQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with MySQL.

2. Yes, I could use MySQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in MySQL I would have to make the CSV, pipe it into the MySQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

If you have something more complex than a flat file, then use relational databases. Even Access databases are better than a collection of text files.

That really depends on what your intended use for them is. I mean I don't know this particular fellow's situation for data collection or what tools he uses for reporting and visualization but perhaps, for him, it's a much better idea to store them in flat files. Me? I have been using flat files for all my data collection about local crime (see here, here, here, and here) for several reasons:

1. I script it all with awk/sed to scrape the data and then put it in a CSV for summary with MySQL.

2. Yes, I could use MySQL for it all but I like to easily see it in its raw format on another remote machine. I also like to use Excel to do ad-hoc pivots and this is the easiest way for me to do that.

3. I upload the data to Google Docs and use their gadgets to make charts for my dashboards and maps. If I were to store it solely in MySQL I would have to make the CSV, pipe it into the MySQL, convert it back out to CSV and then upload it. An additional step for nothing.

Hey, no method is perfect for everyone and every project is a little different and while it's hard for me, based on the information provided, to give this guy any help, automatically suggesting that he needs a relational database to do his data storage might be just a little shortsighted.

YMMV.

Google updated Google Docs recently and I found that the new version didn't support some of the Gadgets that the old version did. I became quite concerned that the old and hackalicious python scripts I was using to upload CSV files that power my website's crime dashboards--something which would suck to have to recode.

I'm going to have to check this out and see if it works much in the same way like allowing me to just replace a Google Docs spreadsheet that already exists something which I require to keep my old code working.

Nice to see that they are continuing to make their commandline tools easier to use as I have to admit I was having some problems getting the ones I currently use to work but now that they are I certainly don't want to loose that functionality (I am good at using those scripts, just not coding new ones so any of the troll comments which say I should do it myself are not necessary, thanks :))

What a bewildering summary.

It's only bewildering if you don't understand how public administrators work. Being that I deal with these people, on a much lower level than the Fed as I am extremely interested in hyperlocal politics and news, on a daily basis I have to say that, "shedding sunshine on the sort of things that 'sunshine laws' may make legally accessible, but that often are not practically accessible," puts it perfectly.

I regularly have to make repeated requests for information that should be publicly accessible. Unfortunately for the general public the politicos do not want this information to be made available, even if it has to be, so they put up every last roadblock they can invent to keep people like me from releasing it to the public.

Let's take for example local transit boarding data for 2007 to 2009. I wanted the number of people who ride the buses in our local transit co-op broken down at the lowest level. A simple task one would assume right? It was clear, based on their reporting, that they had the data at some sort of granular level as they can easily roll it up to yymon, quarter, etc. I also watched as bus drivers hand recorded the number of boardings and wrote them on sheets, by departure time, every single day for more than 2.5 years.

Well when I requested this information here was the exchange which occurred over 7 months:

1. We don't have that data.

2. We don't have that data in an easily accessible format (which would be in violation of Minnesota Statute).

3. We have the data but it would take a very long time to procure. Hundreds of man hours (again in violation of Statute). It will cost at least $250. Pay first, we'll provide it later.

4. We have the data and it will take considerably less time than we first thought. $50 for the data. Pay first.

5. Here's the data you paid $50 to receive. If you want more explanation you need to pay more (in violation of Statute).

---

Now, I turned around and did exactly what they didn't want. I released it to the public and thus to the other state agencies who were originally told this data didn't exist in the way they wanted it. You can see the archive here (don't download the 7MB CSV unless you are really interested in the raw data as I host my site myself and I don't need my cable modem smoking all day long).

So why did they go through so much trouble, wasted man hours of their staff (including their counsel) just to keep this data out of my hands? Because they want to be the ones in control, even though they are mandated by law to provide it to the public, and they certainly want to make compliance with sunshine laws as difficult as possible to keep people from doing this time and time again.

So, unless you deal with that particular instance day in and day out for years, like I do on any variety of topics from any variety of local government entities, then you wouldn't have the faintest idea what that blurb meant. But to me it made perfect sense. I just hope that bringing this data to light and placing it out there for the public to interpret themselves isn't limited to skewed infographics and a couple of PDFs on Deep Water Horizon documents.

What's the difference, plenty of states already collect DNA samples from birth: http://www.lazylightning.org/minnesotas-unnecessary-newborn-dnablood-bank

Just so you know businesses care about all websites that have a high Google Rank. I, almost daily, deal with astroturfing except I have no problems researching and outing these people when I catch them. Nothing better than teaching businesses not to be fucking stupid. Here's today's example: http://www.lazylightning.org/rack-shack-bbq-burnsville-mn#comment-49534

Yelp was helping them be stupid while being stupid themselves. Double negatives are a no-no.

Replacing ARCView. Honestly I use a couple of OSS applications to convert SHP files to KML and then display them in Google Earth. Here are a few examples of Minnesota DNR/county data: http://www.lazylightning.org/boundaries/.

Awesome indeed.

School board members are elected but they aren't necessarily as dumb when you see them on the ballot (remember, most people do no research into these people). Hell, a local school board here recently wanted to ban their members from using social networking to discuss board business because it might go counter to the groupthink: http://www.lazylightning.org/isd-192-school-board-hates-twittering-members

Seriously, watch the video. It's classic.

Why didn't he just email them and ask them either not to use his pictures, or to pay him for them?

Because sometimes even when you do both of those things, you get nowhere fast.

Who cares about "sometimes", there's due process. Outing someone for infringing your copyright is akin to vigilante justice. While that works in comic books, in real life you just get violations of the law to enforce the law.

If it does turn out it's similar but not his texture then he's opened himself up for law suits for defamation.

Why didn't he just email them and ask them either not to use his pictures, or to pay him for them?

Because sometimes even when you do both of those things, you get nowhere fast. Twitter is something that a lot of people utilize and it's a good way to go about expressing your frustration and getting the word out to a lot of people (including the offender) quickly.

Do you really want to choose the tyranny of Comcast or AT&T over that of a local city or county meeting?

I have no idea how private companies run their business meetings or make decisions but I do know how my local cities do and honestly, based on how they choose to spend MY money to support the various overreaching services they already do ($5 million on a new LEED certified municipal liquor store or $20 million on an empty performing arts center which is in danger of losing over $1 million this year), I have to say that I'd prefer that these ventures remain at the private level where my tax dollar input is minimal and generally only if I choose to subscribe to the service myself.

We really don't need yet another venture owned and operated by the city (in one way or another, whether that be through a third party management company or not) that loses money because they are operating a service that they really don't know how to.

Do you really want to choose the tyranny of Comcast or AT&T over that of a local city or county meeting?

I have no idea how private companies run their business meetings or make decisions but I do know how my local cities do and honestly, based on how they choose to spend MY money to support the various overreaching services they already do ($5 million on a new LEED certified municipal liquor store or $20 million on an empty performing arts center which is in danger of losing over $1 million this year), I have to say that I'd prefer that these ventures remain at the private level where my tax dollar input is minimal and generally only if I choose to subscribe to the service myself.

We really don't need yet another venture owned and operated by the city (in one way or another, whether that be through a third party management company or not) that loses money because they are operating a service that they really don't know how to.

On my website I directly attack those in the public eye--especially local politicians--using my standard colorful language choice, I link to a video of an interview where I appeared on a local public access TV show and was accused of being too harsh in my tone and language when I directly attack these people for wasting millions of taxpayer dollars.

Being that this proposed law mentions nothing of the right of the people to directly attack public officials (protected speech, not harassment regardless of how many times I choose to talk about their poor choices at my expense) means that I could be held liable for this and imprisoned. How cute.

You specifically mention that this type of behavior towards the general public (not public officials) is already defined by preexisting harassment law. IOW, there is absolutely no need for this bill to be presented, discussed or passed. Why must politicians create unnecessary laws? Hey you fucking douchebags, stop wasting your time and our tax dollars formulating and discussing unnecessary legislation just so you can look better in the public eye. Assholes.

Yes, I purposefully used the language I did to make a point--I would be arrested and imprisoned under that proposed legislation for posting what I just did. I hope I made my point.

I am impressed with the data made available there in one location. I have done a lot of work with county criminal complaints mapping; SHP to KML for state, county, and local parks; and restaurant inspection reports for the general area around my home because it's difficult for the average person to get that information in a package that's useful to them.

It's absolutely fucking awesome to see that other areas are taking the time, effort, and dollars to make data available to those who pay for its creation in the first place. If anything, the Federal Government should mandate that all states funnel that data to them for display to the citizens in one place. There is absolutely no reason why data shouldn't be in one consistent format and in one place for us to pull down to aggregate ourselves if we so choose.

I for one welcome our data providing overlords.

I am impressed with the data made available there in one location. I have done a lot of work with county criminal complaints mapping; SHP to KML for state, county, and local parks; and restaurant inspection reports for the general area around my home because it's difficult for the average person to get that information in a package that's useful to them.

It's absolutely fucking awesome to see that other areas are taking the time, effort, and dollars to make data available to those who pay for its creation in the first place. If anything, the Federal Government should mandate that all states funnel that data to them for display to the citizens in one place. There is absolutely no reason why data shouldn't be in one consistent format and in one place for us to pull down to aggregate ourselves if we so choose.

I for one welcome our data providing overlords.

I am impressed with the data made available there in one location. I have done a lot of work with county criminal complaints mapping; SHP to KML for state, county, and local parks; and restaurant inspection reports for the general area around my home because it's difficult for the average person to get that information in a package that's useful to them.

It's absolutely fucking awesome to see that other areas are taking the time, effort, and dollars to make data available to those who pay for its creation in the first place. If anything, the Federal Government should mandate that all states funnel that data to them for display to the citizens in one place. There is absolutely no reason why data shouldn't be in one consistent format and in one place for us to pull down to aggregate ourselves if we so choose.

I for one welcome our data providing overlords.

I am impressed with the data made available there in one location. I have done a lot of work with county criminal complaints mapping; SHP to KML for state, county, and local parks; and restaurant inspection reports for the general area around my home because it's difficult for the average person to get that information in a package that's useful to them.

It's absolutely fucking awesome to see that other areas are taking the time, effort, and dollars to make data available to those who pay for its creation in the first place. If anything, the Federal Government should mandate that all states funnel that data to them for display to the citizens in one place. There is absolutely no reason why data shouldn't be in one consistent format and in one place for us to pull down to aggregate ourselves if we so choose.

I for one welcome our data providing overlords.

I have yet to hear/see a rational reason why marijuana is still illegal.

Please. While I have nothing against people getting as baked as hell and staring at their ceilings all night long while listening to Pink Floyd, I do realize that it is an intoxicating substance that is difficult to detect the level of intoxication.

So, you get pulled over and they decide you're baked. They can't really prove it because there is no "immediate intoxication test". They can tell if you have THC in your body but you could have ingested marijuana in some form or another weeks ago. Yeah, the cops have such great tests as green tongues and glassy eyes but a lawyer would destroy those in court.

So, while I have no problems with it becoming legal, I do see one of the reasons why the government doesn't want it to -- aside from the trillions they've put into the pointless "Drug War".

When the metro newspapers finally figure out that a lot of folks actually like non-national stories again, they may be able to save themselves. Uniqueness and specialization are the drivers of everything online.

My website is basically an aggregation of local news sources from all over our area and encompasses some 15 different local news sources. These newspapers have relatively small distribution areas and a lot of fluff. Very little of what comes out of them are "news" by any means and while they may break their news stories before the metro newspapers do, they are usually limited in scope and depth. Just because you have a paragraph blurb about some local happening doesn't mean it was worth beating the Pioneer Press or Star Tribune to the punch.

While I don't have print readership (one of the local papers sends out a weekly copy free of charge to your doorstep) I do have more regular reader than these papers do and I more or less just print blurbs of what they already covered and give my own opinion. While I have plans to do a little more than that, it really gives people something other than the fluffy horseshit that these papers provide (they are usually the "official newspaper of foo city" which is apparently mandated for public notices and thus their stories are fucking bullshit and always pro-city). People seem to want that and while I wish I wasn't leeching I just don't have the budget, time or staff (I'm one person doing this in 1.5 hours a day) to "report" on stuff.

Traditional media needs to get around to doing more editorial that's obviously non-biased. People don't give a shit if the official newspaper of the local school district believes that the renewal of the superintendent's contract was a great idea, in fact, most people disagree with the decision but here in Minnesota are too "nice" to admit it publicly.

Some of these local papers are trying to reach out the modern age with RSS support but refuse to move to full feeds because it would impact their measly ad revenue (of which anyone with an RSS reader probably isn't going to click anyway). I guarantee that I make 10x what they do in their online ads but trying to explain that to them is like speaking to the wall. These people have no idea how to function in the modern news world and I doubt that they ever will.

It's truly unfortunate because, as you said, they are doing a lot of legwork (even if it's more or less pointless) to get a blurb and beat out the bigger outlets to the punch.

I'd point people to take it up with their representatives and other relevant politicians or even picketing to bring attention their cause. Unfortunately the politicians are in on it and the picketing is now only permitted in "Free Speech Zones" and may end you up in jail after crooked judges who still sit on the bench after multiple infractions eliminating due process agree with the government that you are a terrorist.

So, just suck it up and let the assholes win while we all fucking suffer. Global Warming is a fucking threat? Please.

...as I listen to a local band cover the Grateful Dead.

You know, for years I touted the Grateful Dead as a band that was a pioneer in allowing the free taping and trading of their shows. Jerry was wise beyond his years saying that once they were done with a show it didn't matter what the fans did with it. In the years since his death (it seems like yesterday when I got word on IRC) the remaining members of the band have tightened the restrictions on the trading of their shows and now places like archive.org no longer distribute their shows in SBD format.

It's really disappointing that one of the leaders in the free music world and a band that shaped a movement in free access to some of their best music regressed towards a closed format. My favorite quote of the article linked above was, "Technically and policy-wise, it has been invigorating as you can probably appreciate. We have made changes in the past and we will make changes again."

While I have yet made it to the point of no return and have not yet stopped listening entirely, I have begun to support other bands that have room to grow upwards and don't seem to have plans to regress to the draconian limits imposed by the typical mainstream bands. If the Grateful Dead's remaining members continue to stomp on the traditions started by Jerry all those years ago, I will be forced to change my username, my personal domain, and my listening habits much to my great disappointment.

As for the fight going on with the RIAA. While I applaud people who are standing up to them for the rest of us, I really wonder if it will change anything. We are seeing a slow change in the tide (just like we did with other bullies like SCO) but it's really unfortunate that while the record companies are finally getting stepped on, the bands themselves -- especially bands who used to allow and encourage nearly unlimited use of their live material, are starting to bend to the commercial pressures that shouldn't exist.

I still go to live shows of bands that adhere to archaic distribution methods in the hopes that their growing fanbase might be able to change their closed stance (hey, it's happened!) but I mainly support only those bands that allow the free trading of at least some of their music (They Might Be Giants for example -- who are coming to Minneapolis in September and playing at First Ave for those of you interested, Dark Star Orchestra, etc).

If we all keep up the pressure, from all angles, everyone -- including the bands that seem so hellbent on profits -- may come around, someday.

Of course, once the spam bots start leaving ads in my voicemail, then I'm getting violent.

You mean like Justice A Clothing Store for Girls already does? They aren't alone however, most of the time I have a message from some carpet cleaner or other douchebag company that leaves no contact information except a reminder to get my fucking carpets cleaned.

I am seriously considering unplugging the fucking answering machine now too.

And remember that it doesn't mean that anything will get done, especially Justice just for Girls likes to call repeatedly and then when you complain, claim that you put their number there or that someone else did.

My other favorite are the automated carpet cleaning calls that you get, number unavailable, that don't leave their number, name or otherwise and I'm not sure what purpose they serve other than to annoy.

On my website I have a lot of "reviews" of restaurants in my area. Many of these I write about before they even open their doors to obtain a good Google search ranking so that when they do open I can capture many of the first searchers for them. When I do visit the places (or someone else does and posts a negative comment) I get a flood of astroturfing from the owners or employees (or other interested parties) which I usually can pick out immediately by the tone and message of the e-mail (corporate-speak laced comments) or the fact that they have 6 posts with different names and e-mail addresses from the same IP.

Raising Cane's Chicken Fingers had an interested financial party posting about it near the beginning, Kami Japanese Steakhouse had several attempts at this and most recently I had several comments from the same IP with different names and fake e-mail addresses for Bucky's Soup Sandwiches and Salads.

So it's not just limited to major corporations paying people to do this kind of shit. It happens on a much smaller scale everywhere. Thankfully I am a local website operator that takes the time to moderate comments and pay attention to their games. Who knows what more professional sites have to deal with as far as astroturfing skew.