Slashdot Mirror


America's Cities Are Running on Software From the '80s (bloomberg.com)

Even San Francisco's tech chops can't save it from relying on computers that belong in a museum. From a report: The only place in San Francisco still pricing real estate like it's the 1980s is the city assessor's office. Its property tax system dates back to the dawn of the floppy disk. City employees appraising the market work with software that runs on a dead programming language and can't be used with a mouse. Assessors are prone to make mistakes when using the vintage software because it can't display all the basic information for a given property on one screen. The staffers have to open and exit several menus to input stuff as simple as addresses. To put it mildly, the setup "doesn't reflect business needs now," says the city's assessor, Carmen Chu.

San Francisco rarely conjures images of creaky, decades-old technology, but that's what's running a key swath of its government, as well as those of cities across the U.S. Politicians can often score relatively easy wins with constituents by borrowing money to pay for new roads and bridges, but the digital equivalents of such infrastructure projects generally don't draw the same enthusiasm. "Modernizing technology is not a top issue that typically comes to mind when you talk to taxpayers and constituents on the street," Chu says. It took her office almost four years to secure $36 million for updated assessors' hardware and software that can, among other things, give priority to cases in which delays may prove costly. The design requirements are due to be finalized this summer.

40 of 211 comments (clear)

  1. Dead Programming Language? by Medievalist · · Score: 5, Funny

    It's Java, right? Tell me it's Java!

    1. Re:Dead Programming Language? by juniorkindergarten · · Score: 5, Insightful

      Cobol.

      --
      "Every security scheme that is based on secrets eventually fails." - Steve Jobs
    2. Re:Dead Programming Language? by Anonymous Coward · · Score: 4, Insightful

      There are probably millions of retired COBOL programmers out there who wouldn't mine working again for a little while. But they don't have "current" experience.

    3. Re:Dead Programming Language? by spudnic · · Score: 4, Insightful

      That still doesn't mean you couldn't put a web based front end that talks back to the cobol running on the AS/400 (iSeries). We do this all of the time. The cobol is probably very fast and efficient. It just needs a better user interface.

      --
      load "linux",8,1
    4. Re:Dead Programming Language? by bferrell · · Score: 2, Insightful

      Yes, but that needs to have monew, SCARCE TAX DOLLARS, allocated. And it's next to impossible to OBTAIN enough to do the basics... Like public transit, house the homeless, pay teachers etc, etc, etc.

      The comment reminds of when my three year old sister was told there was no money to buy what she wanted. She told our parent to just "write some" (write a check). We all laughed.

      How about if we make Apple, Google, Facebook et al pay what they should have instead of stashing it overseas accounts?

    5. Re:Dead Programming Language? by Tablizer · · Score: 3, Funny

      retired COBOL programmers out there who wouldn't mine working again for a little while. But they don't have "current" experience.

      Knowing how HR "works", they require "100 years or more of production COBOL experience."

  2. That's because their programmers were skilled by Anonymous Coward · · Score: 5, Insightful

    You can read the headline as a denigration of governments (which is always valid, because they get paid regardless of their performance), but you can also read it as proving that the programmers of the 1980s produces some pretty solid work.

    1. Re:That's because their programmers were skilled by thereddaikon · · Score: 5, Insightful

      The average professional developer of the 80's was probably better at their job than the average one today. For one, they actually had a working understand of the hardware on a conceptual level and could relate how their code would interact with the system. Ask your average JS coder how a computer works and grab some popcorn. Due to the limitations of distribution methods and storage media of the time software was also far less bloated and much more stable at launch. You couldn't easily patch something post launch like you can now so you had to get it mostly right the first time. There is no such thing as bug free code but there is code with some bugs and then there is code that is mostly bugs.

    2. Re:That's because their programmers were skilled by sfcat · · Score: 4, Interesting

      For one, they actually had a working understand of the hardware on a conceptual level and could relate how their code would interact with the system. Ask your average JS coder how a computer works and grab some popcorn.

      There's definitely a lot more going on nowadays with both the OS and application layer libraries. I doubt anyone can understand the entire stack the way people did back then.

      In fact, I'll bet you can't figure out what machine code a simple line of JS code turns into either. And even if you did, that answer would be wrong in a few weeks when the next release of V8 comes out.

      Due to the limitations of distribution methods and storage media of the time software was also far less bloated and much more stable at launch. You couldn't easily patch something post launch like you can now so you had to get it mostly right the first time. There is no such thing as bug free code but there is code with some bugs and then there is code that is mostly bugs.

      I seem to recall a lot of older software that had a final version named something like 1.0.4. And those were only made available months after the initial release. In other words, you'd have software with glaring flaws that you'd have to put up with for months. Versus now, any major bug would be quickly patched up post launch and it's a matter of a few days, or even just a few hours of waiting.

      Hahahahaha, modern software that is quickly patched is also broken about 99% of the time.

      I recall products and code actually working as designed in the 80s. They had computers you could literally pull a running CPU out of the board and it would still continue to work correctly. If you think today's software is anywhere near as well written or reliable as software of the past, you are a fool. Today's software business is built on cheap coders pushing out crap as fast as possible. In the past we did actual engineering and testing to make sure things worked before we gave them to users. Today, we let the users's beta test software live and 'let it crash' is the moto of the ops folks.

      --
      "Those that start by burning books, will end by burning men."
    3. Re:That's because their programmers were skilled by MBGMorden · · Score: 4, Informative

      Depends. I work in such a government office. Our property tax billing software is written in COBOL, was first written in the very early 1980's, and does generally work well.

      HOWEVER - whew boy is that some weird code. Nothing is driven by configuration tables. Pretty much every behavior is hard coded into the program. Even for security permissions like who can access which screens, there's literally a user ID HARDCODED into the source.

      And god help you if you need to something like widen a text field (all of which are notoriously small due to being defined back when disk space was much more expensive). You'll have to recompile dozens of programs that hit that file, and often times when it's absolutely necessary we have to cobble together fields to make a bigger one (ie, positions 1 through 30 of a field might be in the middle of a line, 30 through 40 towards the end, and then 40 through 50 in another field at the very end of each line.

      Rather than being an example of how good the code is, I think it's more an example of the fact that even for bad (or even terrible) code, if you've had 30 years to debug it it'll still function fine. It's an unmanageable mess, but it does indeed do exactly what its supposed to.

      --
      "People who think they know everything are very annoying to those of us who do."-Mark Twain
    4. Re:That's because their programmers were skilled by Darinbob · · Score: 3, Insightful

      You can read that as some people just are dumb, criticizing governments for spending money in the first place and then simultaneously criticizing governments for not spending money to upgrade old systems. This is an age old problem that if you get the budget once you can spend it to buy useful stuff and then you have to deal with the same thing for years or decades before you'll get budget to properly improve or maintain it. The fact that slashdotters are still surprised and amazed that governments use outdated equipment and software just proves that they've been stuck too long in the basement and need to get out more.

    5. Re:That's because their programmers were skilled by Livius · · Score: 3, Insightful

      A lot of software reach a level of being essentially feature-complete and mostly bug-free, and then could only go downhill from there. Some software was at that point in the '80s, and most of the rest reached that level in the '90s.

      If you have software that
      1) works, and
      2) physically cannot connect to the Internet (which seems to be the only way to genuinely guarantee something is secure)
      then replacing it would have costs risks that outweigh the benefits.

    6. Re:That's because their programmers were skilled by phantomfive · · Score: 2

      . Due to the limitations of distribution methods and storage media of the time software was also far less bloated and much more stable at launch. You couldn't easily patch something post launch like you can now so you had to get it mostly right the first time

      The amazing thing is that they are able to do it without at least three different noSQL databases.

      --
      "First they came for the slanderers and i said nothing."
  3. Because it works... by TWX · · Score: 4, Insightful

    ...and is optimized to run on extremely slow hardware. It was often written by people that were extremely talented at optimizing their code because hardware limitations forced them to do this.

    I won't deny that advances in computer language have made it possible to write programs better than they used to be written when machines were extremely procedural and single threaded, but at the same time, the amount of bloat in modern programming afforded by modern hardware has more than made up for it.

    I've seen the progression of software for simple things like workorder systems and asset management and audit get worse over time. The only 'improvement' is access, in that going from an 80x25 text console on a remote system with terminal emulation, to a a full-fledged program running on a specific architecture in a text mode, to a GUI program on a specific architecture, to 'applet' type programs using runtime libraries cross-platform, to web-based access that theoretically are entirely platform independent presuming a minimum browser version, and in just about all cases the further they've gone, the slower clunkier for actual experienced users that frequently use the system. It might not be any better for inexperienced users either if the vendor hasn't taken the time to look at workflow from an outside point of view.

    --
    Do not look into laser with remaining eye.
    1. Re:Because it works... by SurenEnfiajyan · · Score: 4, Interesting

      Yeah, today's bloat is incredible, and this era of shitty programmers with their toy programming languages is very disappointing. One can only imagine how many CPU cycles and electricity is wasted. Just a fun fact, Windows 95 required less ram (sometimes in order of a magnitude) than a basic calculator in any modern OS.

    2. Re:Because it works... by Anonymous Coward · · Score: 4, Interesting

      It blew my mind when I ran the Windows 10 calc.exe for the first time on a HDD system. It had a measurable loading time.

      A program function as old as windows which was snappy on a 386 now has a human noticeable loading time-and a link to a privacy statement! But that's another rant.

    3. Re:Because it works... by I+kan+Spl · · Score: 2

      I would hazard a guess that a significant part of this cost is upgrading the buildings to have modern networking, as opposed to token ring or whatever the old machines were using.

      Costs get pretty high quickly when your upgrade involves bashing out walls to replace data cables that were installed before the era when ducting became a thing.

      --
      My UID is prime and so is this number: 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0.
  4. Awesome! by DogDude · · Score: 5, Insightful

    I think this is awesome. It was good software that hasn't needed to be "updated" every other day like modern software is.

    --
    I don't respond to AC's.
    1. Re:Awesome! by Anonymous Coward · · Score: 2, Insightful

      Oh noes! The software has be running since (some_random_date_in_the_past). We must update now!

      One second there, pardner. This isn't a case of wooden sewer pipes or gas still operating streetlights. Hell, this isn't even a case of two cylinder cars running on super-highways. This stuff has been running since the 1980s, and as old as that sounds to many of you youngsters out there, that was only 25 to 30 years ago.

      My point here is that this is the very first iteration of the modern software cycle. We really have no concrete guide post on how long this stuff can run.

      Is the hardware failing? Replace it.
      Is the software inherently insecure? Upgrade it.
      Are we running out of people with IQs above room temperature so these old languages would be lost? Get more H1-Bs.

      Otherwise I'd say leave it alone.

  5. Was it ever acceptable? by rickb928 · · Score: 3, Insightful

    'can't be used with a mouse'

    Which describes an entire era of software. Was ti useful before a mouse was considered so vital, actually before it was even used, or existed? Well, back then it was specified, purchased, and used. Same as now, this is a specious argument.

    'Assessors are prone to make mistakes when using the vintage software because it can't display all the basic information for a given property on one screen'

    Dear, it seems as if either this software was NEVER usable, or are users able to take the necessary care to do their work accurately...?

    'The staffers have to open and exit several menus to input stuff as simple as addresses'

    Ah, the slings and arrows.

    'To put it mildly, the setup "doesn't reflect business needs now'

    As in ease of use, etc, sure. As in it has always worked like this, why do I seem to read this as 'it's old and clunky, and it's the fault of the software that I make so many mistakes'. Where I work, we do have a lot of this. Because we care, and work in private industry, we understand the software, make the necessary adjustments to our habits, and take the time to do it right.

    'It took her office almost four years to secure $36 million for updated assessors' hardware and software...'
    'The design requirements are due to be finalized this summer.'

    What comes first, the chicken, the egg, the funding, or the requirements?

    After all that, has no one in San Francisco government made a CBA case for replacing it? I'm betting they are leaving revenue on the table by not having accurate data. And I'm betting they need to build reasonable, achievable requirements. So many government IT projects fail because the project was designed so poorly from the start.. Examples abound.

    --
    deleting the extra space after periods so i can stay relevant, yeah.
    1. Re:Was it ever acceptable? by angryargus · · Score: 2

      Speaking as someone who has actually used SF’s assessor software in SF city hall (as a public citizen to lookup property records) I think they’ve making the common mistake of not realizing that introducing a mouse always hinders productivity (people are always faster keeping their hands on a keyboard and navigating via menu).

      The software is 80x25 and does look like a DOS app though behaves more like a unix or mainframe terminal app. The multi-user aspect might contribute to the replacement cost estimate but there’s gotta be lots of other software available that was already written for other counties.

      The “prone to mistakes” is a hiring issue (gov’t at its best), not a software issue. They’ve got a lot of errors in how docs are recorded and indexed that reveal sloppy and lazy work.

  6. A mouse? by Hugh+Jorgen · · Score: 5, Insightful

    A mouse would not increase efficiencies, look at the horrid inefficiencies of web applications. The lost productivity when IBM forced one of their large financial clients to move from a 3270 green screen app for change management to web app caused outrage. Most UI/UX people do not use the tools they design. Is there "technical debt" here, likely. Does the new app need to support a mouse or be web based? Likely not.

  7. nothing wrong with old tech - ransomware resistant by Anonymous Coward · · Score: 2, Funny

    The only reason for San Francisco to upgrade is if they want to be like Atlanta and get to try out some ransomware code on their production systems

  8. Re:Hint, they mean weather real esate... by rickb928 · · Score: 2

    Since it was designed that way, it always made the users pay attention and be inefficient.

    So it was a result of the limitations of the industry at the time.

    --
    deleting the extra space after periods so i can stay relevant, yeah.
  9. Quote by Anonymous Coward · · Score: 2, Insightful

    "We’re dealing with an irrational public who wants greater and greater service delivery at the same time they want their taxes to be lower." That's not irrational. That's optimistic. Improved service should decrease costs. If the cost of improved traffic management is less than the savings of improved road maintenance than the solution works. City planners work on long term solutions with tech that likely won't change for decades. Saying something was made in the 80s makes it sound old, but doesn't mean it is obsolete.

  10. Everything is running on software from the 80s by plague911 · · Score: 2

    Finance, Cities, Airlines, Pharma, the whole defense industry. What kind of news article is this for a software developer biased news aggregation portal when even this most basic fact is stated as some kind of surprise.

  11. Not just Minnesota by slipped_bit · · Score: 2

    ... Minnesota spent about a decade and $100 million to replace its ancient vehicle-licensing and registration software, but the new version arrived with so many glitches in 2017 that Governor Tim Walz has asked for an additional $16 million to fix it. ...

    Must have bought the same software that my state purchased. For over a decade they collected a "modernization" fee while using the old system of IBM 3270 terminals, and when they hired the company to develop the new system it was a colossal failure. Wait times went up significantly. Years later and the system still hasn't improved. Oh, and they still keep collecting that modernization fee. The only improvement is that you can now reserve a spot in line via a text message and they will then send you a text message when you're near the front of the queue, so you can go to work or whatever and don't have to sit in the waiting room for hours.

    I pay the "convenience" fee to handle as much as I can on-line and via mail, but some times that can't be done such as when purchasing a vehicle.

    I bought a utility trailer a while back. They had to fill out my name and address on no less than four different pieces of paper. Seriously? Even if they need to use paper for some strange reason, they can't just type it in once and print out the four different forms they need?

  12. Good by transformania281 · · Score: 2

    Leave it that way. Likely not directly connected to the Internet so it's secure. Or, if it is, not running anything that anyone remembers how to hack or are targeting these days.

  13. Beware the creeeeeep..... by magarity · · Score: 2

    The design requirements are due to be finalized this summer.

    I can almost guarantee the project will be years over schedule and billions over budget because, no, the design requirements were not actually "finalized" and city managers with enough clout constantly interrupt development with additional feature requests.

  14. Re:Not necessarily more secure by Seven+Spirals · · Score: 2, Funny

    Buwahahaha. Speaking as a guy who maintains old Unix and OpenVMS boxes for a living I can tell you that most people would have zero idea what to do if they did "get in" to a mainframe or even a mini. Those skills are very rare and that's why people pay good money for folks who have skills beyond the basics that lots of schmucks learn from using Bash on their Ubuntu machine. I watch newbies with MVS, MPE, and VMS and they can't even change directories, much less do any damage. The mother fuckers couldn't even set their terminals up for proper emulation. I've gone as far as to give people access to honeypot environments for various mainframe and minicomputer operating systems from the 1980's (long story on how I set that up) but it was hilarious watching children from China, Romania, Israel, Bulgaria, Russia, Ukraine, and even a few Americans flop around like fish once they dropped to an environment that doesn't have command line completion and on screen help to walk their ignorant asses through everything. These little tykes are too dumb to do much damage, even though all they'd have to do is READ for about 10 minutes to get familiar with things (so in other words - forget it). Hell there are even Youtube videos on how to get around on a various mainframes. Nonetheless, only the true Scotsmen seem to have the attention span to pull it off and they are too busy making money and living well to worry about breaking into old city mainframes.

  15. Its a trap by TRRosen · · Score: 2

    if you wait to long you get trapped by technology.
    The big problem here is while the data on these systems is trivial today the pushed the limits of 8-16k miniframes. Thus there were a lot of sneaky calls directly accessing memory or even processor registers. The software was integrated into the hardware with bits of assembly and what not. This makes it pretty much impossible for a young modern programer to figure out. The old ones are dead or want $500/hr to touch a COBOL system. and lets face it the guys that bid on gov contracts wont hire them anyway. So these systems become black boxes... they still work but no one knows how.
    There is a local business here that until being bought out still depended on an 80's wang system running software built for a 70's wang system. There only source of parts was trading with the air force and they only used theirs to train people to what tech may be hooked up to old soviet bloc weapon systems.

  16. AS400 isn't dead by Deathlizard · · Score: 2

    I never will understand this obsessive need to try and kill systems that just work.

    AS/400 based system are some of the most reliable systems money can buy. They can handle insane amounts of workloads and can scale from small systems to complete mainframes. The only real issue with them is terminals (which is a minor issue with modern terminal emulation) and the hardware and software maintenance costs, but you do get what you pay for.

    I'll almost guarantee that they will switch to some "modern" system running either a Java or Web based backend that will either get hacked, crash due to excessive load, or both, and probably pay twice as much as they're currently paying now to switch vs upgrading their current setup.

    1. Re:AS400 isn't dead by smoot123 · · Score: 3, Interesting

      AS/400 based system are some of the most reliable systems money can buy. They can handle insane amounts of workloads and can scale from small systems to complete mainframes.

      OK, so I was actually looking into this a few days ago. I'm working on modernizing our build system, part of which includes running Jenkins jobs on AIX servers (to build an AIX client module. Whatever.) We fire off around 10 parallel jobs to build on AIX, Solaris/Sparc, Solaris/x86, Windows, Linux/x86, Linux/x64, and so on. I wanted to understand which ones take the longest because that's what I have to optimize.

      Funny, the AIX, PA-RISC, and SPARC systems get absolutely crushed by any x86 system. They may get a lot of work done per CPU cycle but the newer systems have just so many cycles it doesn't matter. AIX, HP-UX, and Solaris might be heroes at getting lots of transactions done on a 100 MHz processor, yay for Big Blue, HP and SunSoft, but if I care about actual absolute throughput, just nope.

  17. Re:It kind of works by cayenne8 · · Score: 3, Insightful

    ...so a huge amount of training is required and employees clearly aren't getting it.

    Have you seen the average worker at the DMV? They're not exactly the sharpest knives in the drawer.

    --
    Light travels faster than sound. This is why some people appear bright until you hear them speak.........
  18. Re:$38 Million upgrade? by DarkOx · · Score: 2

    It probably is a bespoke solution though. I am sure the original software was modeled on an existing paper process. Said paper process probably had a lot embedded rules; which are defined by local statue.

    This is one of those cases where there is probably quite a bit of nity-girty that most people don't realize. Its one of those cases where yes there is probably some off the shelf animal that might meet there needs but it probably also requires a army of consultants to customize and develop rules for. Not cheap.

    The alternative is to realize that the data volume is only so big. There are only so many residents, only so many land parcels, etc... Is a big volume sure but probably not something that a good DBA and handful of develops can't implement from scratch + copy of MSSQL server or Postgres almost as easily as deploying some giant all encompassing solution from SAP or Oracle.

    --
    Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
  19. OMG! IT DOESN'T USE A MOUSE! by rnturn · · Score: 2

    It's COBOL. Running on an AS/400. Big effin' deal. I'm guessing that the county assessor's opinion of computer technology is "If I haven't used it, it's got to go." [1] COBOL is not exactly dead, but it isn't the programming language du jour. (I've seen recent job ads looking for people with COBOL experience. I could forward some of them to her.) I suspect she'd much happier if the whole system consisted of a single, shared Excel spreadsheet. At least she'd have her mouse to move around.

    I predict we'll be seeing a future post about the huge cost overruns, inoperable software, and the crisis in the assessor's office when the conversion to a trendy "non-dead" language on new, overpriced hardware flounders and the lawsuits begin.

    [1] - She seems like she might be a kindred spirit with the guy I once worked with whose title of "Director of Computing Technology" would never have prepared you for his daily criticism of each and every computer technology he came in contact with. It was really quite amazing. Nobody--and I mean nobody--in the department could figure out what technology would be graced with his approval.

    --
    CUR ALLOC 20195.....5804M
  20. Re:Not necessarily more secure by deKernel · · Score: 3, Interesting

    So what you are saying indirectly is that you are the brain-damaged one since you speak of things you have zero knowledge of. VMS was one of the best designed OS's ever. It was light years ahead of all the Unix's flavors at the time.

  21. Re:$38 Million upgrade? by painandgreed · · Score: 2

    While San Francisco is a big city, that just feels like a big number to me.

    Not really to me. My department uses three systems and just to upgrade one of them with the same vendor from one version to the next was around 12-15 million. That's for new servers, deployment team, trainers, vendor support, etc. Any changes needed to the actual server room because of need for better HVAC, user computers, or something like that would be extra. Now, that's for a pretty large system of servers handling the main cluster, reporting, back ups, web serving, file server and archives, etc. Start talking about going from an ancient system to a new one, the project is probably going to be a year plus and have to start with figuring out decades old workflow that isn't documented, data models and conversion, and other stuff before the servers even come up.

  22. Sounds good, but.. by Sqreater · · Score: 2

    You have to double the cost from 36 million to 72 million and expect a long delay in implementation and it won't work the way they said it would or at all and they will have to scrap it in the end and go back to paper and pencils because they tossed out the old system. Real world.

    --
    E Proelio Veritas.
  23. apocryphal by astrofurter · · Score: 2

    Here's a perhaps apocryphal story I heard from an old friend. At the time he was with a consultancy that did work for the City.

    About 15 years ago the City of SF felt that their old mainframe-based financial software was showing its age. This was 100% bespoke software, iirc written in COBOL. Lots of lawful-corruption programmed into it, of course. So the City asked Oracle for an estimate on what it would cost to "upgrade" to Oracle Financials.

    So Oracle sends in a pair of consultants to examine the old software. After a month they make their report: If the City wants to migrate to Oracle Financials, they are going to need to pay Oracle $10 million. FOR A PROJECT PLAN AND COST ESTIMATE. The actual migration was expected to cost much more.

    Needless to say, the City kept their old software.

    How accurate is this story? No idea - it came to me third hand. How plausible is it? Very plausible.