Slashdot Mirror
Anniversary of the First Computer Bug
aheath writes "According to the
US Naval Historical Center the first computer bug was logged on September 9, 1945 at 15:45: "Moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested at Harvard University, 9 September 1945. The operators affixed the moth to the computer log, with the entry: "First actual case of bug being found". They put out the word that they had "debugged" the machine, thus introducing the term "debugging a computer program".
The Wikipedia has a "computer bug" entry that lists some other "famous bugs" including the fictional HAL 9000 bug. What is your favorite computer bug story?"
September 9, 1945 at 15:45: "Moth found trapped between points at Relay # 70, Panel F [..] "
September 10, 1945 at 08:02: "Darl McBride Sr. claims he owns the moth."
September 10, 1945 at 23:53: "We snuck into Darl's room and put his hand in a bucket of warm water."
September 11, 1945 at 09:46: "Darl gets to work late but is proud to show us 'his' new bucket. We all hate him."
Trolling is a art,
Somehow, saying "First actual case of bug being found" seems fake to me. It's like finding cavalry sword from the first world war with the inscription, "Corporal James Smith, Third Mounted Infantry, World War One." You'd know that even if the sword was real, the inscription was years after WWII, making it less valuable, and lessening it's voracity.
Or is this the first actual case because they suspected before there were actual bugs in the system but never found them?
Then again maybe it was just prophetic. Like NASA when the STS missions launch(ed): "3...2...1...Liftoff! [message about this mission and it's 'first' for space here]"
A programmer is a machine for converting coffee into code.
Those things really multiply don't they?
First you find ONE in a computer relay. Then, almost sixty years later, they've multiplied so that there's one in every program I write.
Like cockroaches.
You just can't get rid of them. They're hard to find. And when you squash one, three more come from nowhere!
Slashdot Syndrome: the sudden, extreme urge to correct someone in order to validate one's self.
"Moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator
:-)
Cool. I always wondered about the etymology of "computer bug", and now I know the etymology is truly related to entymology.
Visit Jonesblog and say hello.
Win98 crashing on Bill Gates in front of millions of viewers.
Will code a sig generator for food
according to opera...
"The origin of the word "bug" has wrongly been associated with an incident where a moth was pulled out of a Mark II computer. Apparently, the term was used prior to modern computers to mean an industrial or electrical defect."
Why do I h8 apple?
By the way they logged the bug, "first actual case of bug being found" the term was already in use and they were pointing out the irony that the bug in this case was a real bug
Free cell phone tracking
My favorite bug was in an existing product that had been on the shelves for a while and went through numerous patches to fix many bugs. Going through the testing, I found the UI could not be moved around the screen with a left handed mouse configuration! Immediately, I dropped the bong and decided a cup of coffee and looking on a few other machines were in order. Did those and the bug was legit. Sent it to development and they scrubbed it "as designed". Silly bug, but I can't believe no one tested or complained about it.
...was that an update to Mark II Aiken Relay Calculator (a small screen over the air intake) was developed on 1 September 1945, but the navy was too slow in installing the patch.
When I was doing inhouse tech support for a large company that makes green tractors, I got a ticket about a system that was having random lockups. After investigating, I found that the lockups were indeed random, so set out to try swapping the RAM first. Judge of my surprise to find a tiny spider caught against the base of a SIMM, blackened and crispy. If someone had told me that there's enough juice flowing through a RAM chip to fry even a spider, I wouldn't've believed it, but there the little critter was. I couldn't believe that little bug alone would be causing a problem, but on a whim I left the chip in, sans spider, and behold, the system worked perfectly.
Odd, that.
And although it's not a bug, I have had someone bring a computer into my shop for locking up, and found a live mouse in it. It escaped into the shop and I believe it lives here on Dorito crumbs to this very day.
End of lesson. You may press the button.
It's a new and exciting feature!
Stop over-analyzing your analizations
The Schrodenbug...named after the Theroy of Schrodinger's cat...where by if you put a cat in a box, its not truely dead until you look at it again...
This is a bug which while in existance in your code has no effect until you happen to notice it, in the code. Then suddenly the effect of having this bug begins to appear. While until you noticed it, the effect never appeared and the program ran as intended.
Power Corrupts,Absolute Power Corrupts Absolutely, leaving one person(group)in charge is absolutely corrupt.
way back, my first job... only 2 programmers, me and another guy who worked from home over a 9600baud modem. We had no CVS or anything like it(we were noob).
The "bug" in question was merely him and me modifying the same file every other day. I used i,j,k,z for iterator variables. He had the habit of using i,j,k,m. The file had 2 functions, one with a parameter z, the other with a parameter m.
I guess you can figure out how horrible such things can get. It took weeks before we figured out it was a naming issue.
When will I end this grieving ? When will my future begin ?
Windows ME
Could we please stop hearing about it?
The The Jargon File covers this and includes a picture of the bug in the entry on "bug" and states:
John.To be specific, that first bug was recorded by future Admiral "Amazing" Grace Hopper, a (rare female) Line Navy officer (as opposed to a WAVE or Naval Reserve officer.) Her name has gone on to one of the most modern guided missile destroyers. She was quite a remarkable woman, read up on her career if you get the chance.
"I am an Adept of Tantric VAX."
I still think the bug in converting between metric and imperial units causing a billion dollar Mars probe to crash is the top one.
Regards,
--
*Art
Doing some tech work in Brooklyn, NY. I got a call from a small company (3 machines in a business run out of a apartment).
Well one of the machines was making funny sounds. I heard the machine when I arived and it sounded like a wire was caught in the fan. I opened the case and about 10 very large and nasty roaches ran out, there were about 20 dead ones inside the case.
It seems the 80mm fan in the back got pushed in an left a nice hole in the case, which the 2 childern in the house used to put food they didn't want to eat.
I refused to clean the machine out, and told them what they had to do, I got outta their as soon as I could, trying not to vomit thinking about the roach guts on the CPU fan.
"The word "genius" isn't applicable in football. A genius is a guy like Norman Einstein," - Joe Theisman
I don't know if this counts, but here goes:
I worked as student help at a college that had a PDP-11 based mainframe. One night it went down. Computer techs were called out but could find nothing wrong. This continued night after night at about the same time each night. So the techs hung around after hours to keep an eye on it.
Around 6:30pm, the cleaning woman came in with her vaccuum cleaner. She promptly went over to the wall socket, unplugged the mainframe, plugged in her vaccuum cleaner and started vaccuuming the floor.
IAALS.
It depends. If he was trying to show that the SCO jokes are as old and tired as the Al Gore jokes then he is very witty. Otherwise he is just a moron.
Perhaps the most influential bugs of all time were those that allowed the Morris worm to propogate. Sendmail, fingerd, rsh/rexec.. all to blame. The worm led to the formation of CERT. Quite influential.
-molo
Using your sig line to advertise for friends is lame.
Seen on the license plate of a VW Beetle: FEATURE
Arbitrary sig
While working on an embedded printer driver board, I had just burned new firmware and installed it, tested it, and, because we had had an incident where the internals of another printer had melted together, left it off and unplugged. Five minutes later one of the applications programmers came storming into my office claiming that my new firmware was crap. I calmly walked back out to the lab, looked over the machine, and commented "it works better if you plug it in..."
Antz is my favorite Computer Bug Story
they didn't find a rabbit in there. Then we'd all be referring to "derabbiting" or "derabbitizing" the program.
They say the first thing to go is your penis. Well, it's either that or your brain. I forget which...
The term bug when referring to a flaw in a mechanism does NOT originate in the coputer machinery of 1945. In fact, it is much older, and is traceable to as far back as Tom Edison:
On November 18, 1878, Edison wrote to Theodore Puskas, "It has been just so in all my inventions. The first step is an intuition--and comes with a burst, then difficulties arise. This thing gives out and then that--"Bugs"--as such little faults and difficulties are called--show themselves and mo nths of anxious watching, study and labor are requisite before commercial success--or failure--is certainly reached" (Matthew Josephson, Edison: A Biography, John Wiley & Sons, 1992, page 198).
Okay, back in the day(tm), I worked in technical support at Spry, makers of Internet in a Box(tm). One of my duties was to write up bug reports for the internal support system for the tech support reps.
:)
Turns out we had a bug in Spry Mosaic that, when it hit an empty IMG tag (as in, nothing else in the tag but the letters IMG), it would instantly crash. When I wrote up the document, I forgot to escape the less-than and greater-than marks, so it put the actual tag in the tech support document.
The upshot - when the tech support reps searched the database for 'crash in browser', one of the hits that would come up was the document I made - when they loaded it to see the details on 'crash in browser', that's exactly what they got. Ooops.
I can laugh about it now.
Actually, I laughed about it then, too.
A Heisenbug is a bug that goes away when you look for it and reappears when you stop looking.
Re-read parent - as he says, the way in which they wrote the log entry wouldn't make sense if that were true. They were being sarcastic when they affixed the moth to the log book, writing "First actual case of bug being found." This strongly implies that things were called "bugs" previously, but that they weren't literally insects. These guys had a sense of humor.
So the term was in use before these guys found the insect - this is simply the first incidence of the insect as in the urban legend, which postdates the original usage.
-Looking for a job as a materials chemist or multivariat
The log entry with the moth is from September 9, 1947, not 1945.
From the OED:
b A defect or fault in a machine, plan, or the like. orig. U.S.
When I worked as a technicion for UPS I was often called upon to visit customers at their businesses or homes. I visited this guy near Pt. Charlotte, FL (and that's another horror story in itself) who had a PC damaged during shipping. I should have known before I entered his house that it would be BAD -- there were shopping carts, old engines, tree branches all around his property. When I finally navigated through his living room into his (horrors) bedroom where the PC sat, I was already getting nauseous.
"What's wrong with it?" I asked, since there didn't seem to be any damage.
"It won't turn on," said he.
OK, no problem. As a technician we were allowed to pop open the PC to check if it was simply a cable or card that came loose during shipping. No problem. I pulled out my screwdriver and started undoing the case. Soon as popped the top a bunch of massive roaches scampered out.... followed by dozens of little miniature ones. Now, I HATE ROACHES. I can play with grasshoppers, earthworms, beetles, and other critters but roaches just give me the willies. The guy just looked at them marching around as if they were some little pets. With supreme effort I put everything back together and turned on the PC. It booted! The only sickenging thing was this flick-flick noise coming from the fan. I think there's a roach still lodged in the fan to this day, its little antennae wiggling, its nasty little legs twitching back and forth. flick-flick-flick...
(true story)
I call bullshit on this one. A Google search of this phrase yields nothing.
"byte under guard" indeed. Who moderated this +5 ? We need "moderation under guard" (MUG(tm)) here!
What am I missing?
(The linked articles didn't give any hints either.)
Your Servant, B. Baggins
I used to work on debuggers. The hardest bugs to find were bugs in the debugger. Why? You have to debug the debugger.
The absolute hardest bug I ever tracked down was actually a kernel bug. When single-stepping in assembly over a branch-shadow instruction, the application state was corrupted. It only happened on one particular model of RISC chip and only with a certain version of the kernel. Bleh!
"No matter where you go, there you are." -- Buckaroo Banzai
Separately, Tenner points out that 'bug' was used by telegraphers as the name for hidden faults in circuits, and that it also had a literal meaning for operators, since Western Union offices were notoriously dirty and insect-infested. In 1868, Thomas Edison, who started as a telegraph operator, invented an early version of an electrical zapper to debug his desktop.
However, it suddenly stopped working and it had turned out that during an upgrade the number had been changed slightly. The leading 9 (to dial for an outside line) had been removed. Therefore rather than hitting an outside line, it would dial 0 (getting the receptionist) and then try to negotiate with her before hanging up. Three minutes later, it would try again and again - until it had retried and failed 10 times.
The poor receptionist hadn't reported it to anyone and it was only after about a week did they find the problem. She'd put up with 10 calls a day for 5 days without saying a word. She thought it was some prank caller.
Avantslash - View Slashdot cleanly on your mobile phone.
Couldn't resist the "favorite computer bug" temptation...
In college, around 1982, a friend had a micro by a company called Ohio Scientific, a Challenger something-or-other (I think that's right). The machine was running a BASIC interpreter, and had a character set that supported some simple games. Among the special characters supported were "tanks" in various orientations, so one could write a simple tank hunting game. Which he did.
We noticed when we started playing that we could move the tank offscreen and back, since he hadn't put any bounds checking to constrain the tank movement. When we toured too far offscreen, however, the program crashed.
We typed LIST to have a look at where bounds checking might be added to the code, and we found the runaway tank. Leaving a swath of blank spaces behind it, there was the tank character embedded in a line of BASIC source code...
The actuall real bug was taped into the book because it was an actual *real* bug. The Pun was intended back then aswell. The term debugging had been used earlier when debugging ENIAC (real bugs too) and finding unusual and nerving errors.
We suffer more in our imagination than in reality. - Seneca
b. A defect or fault in a machine, plan, or the like. orig. U.S.
1889 Pall Mall Gaz. 11 Mar 1/1 Mr. Edison, I was informed, had been up the two previous nights discovering 'a bug' in his phonograph-an expression for solving a difficulty, and implying that some imaginary insect has secreted itself inside and is causing all the trouble.
Oxford English Dictionary, 2nd Edition
Quoted from Chapter 5 of The Practice of Programming, by Brian W. Kernighan and Rob Pike.
Fellowship 9/11
An ATM at a branch of the bank that I worked at developed a fault wherein it incorrectly reported its location. A guy used the machine while on a business trip. When his statement arrived it looked as if he had been somewhere else at the time. His wife accused him of lying and having an affair.
Eventually the bank was contacted and the fault confirmed, but by then the statement data had been archived to tape. We had to patch the archive statement suite to check each and every archive statement transaction request for the erroneous one and then modify it on-the fly to return the corrected information. The patch stayed in for many years until the data was migrated to new archive devices, when the data was permanently corrected.
Of course that was just one single transaction. There were probably hundreds that were wrong and who knows what happened in all those other cases.
Political Correctness is doubleplusungood.
I heard this story first (or possibly second) hand in Sweden; specifically Goteborg.
So, this is for admissions for dental school, about 5 years ago. Some bug causes the students with the _lowest_ test scores to be admitted.
The error is discovered but then the admission decisions have already been sent out. The school finds it inhumane to retract the offers those who have been admitted in error. However, they also find it unfair for the most qualified students: it is decided to admit both groups.
The funniest part is that rumor has it that there was no significant difference in performance between the two groups.
Tor
I test space station software for a living,
and I do in fact have to do a written log sheet
when we run a test formally.
The test is also observed by a government observer, we verify the hardware and software configuration is per drawing before starting a set of formal tests, and I print and attach the test results to the log sheet. Then it gets reviewed by a number of people here, and sent to NASA, where it gets reviewed by some more people.
By the way, we have our share of insect problems, too. We occasionally get ant infestations under the raised floor in the computer room. It's most likely due to the break area in the basement being right underneath us (fridges & microwaves)
Daniel
Apollos were well networked, and it was possible to manipulate the parameters of the windowing system on one machine from another machine (like you can with X Window system, given sufficient permissions).
The Apollos had a command to change the mouse speed (similar to the X "xset m" command). It took a numeric value specifying the pointer distance to travel per unit time. The bug was that if you specified a negative value, the mouse pointer would travel backwards. No big surprise really, and not very interesting.
When this bug was discovered but not yet fixed or widely known, someone decided to play a practical joke, and walked into a fellow hacker's office and sat at his workstation and started playing with his mouse. A few seconds later (with the help of a hidden assistant in another office), the hacker says, hey look, there's something wrong with your mouse, it's all backwards. Sure enough, the mouse is acting all upside-down. The prankster then says, hey, I know what's wrong, have you cleaned your mouse lately? You must have put your mouse ball in upside down. He then pops the mouse ball out and pops it back into the mouse, and sure enough (with hidden assistance), the mouse works normally again. The victim of the practical joke was, of course, entirely puzzled.
It school in the late '70s, they purchased a second PDP 11-34, and the sys admins thought "wouldn't it be cool if we could get the two machines to communicate!" So they connected a serial port on one to a serial port on the other. Tried to send a packet... Boom! Both machines immediately crashed. Rebooted, reconnected the serial port, started a send, crashed again. Finally, it dawned on them... they hadn't disabled terminal echo. When the first character was sent, it was immediately echoed by the second machine, then echoed by the first, etc. Comm interrupts were high priority and a lot of overhead on the PDP, so the machines never left the interrupt handler, and essentially were hung.
"Freedom means freedom for everybody" -- Dick Cheney
Well, he said, there was this funky wired up typewriter type thing that was the data input. Once you entered your problem and the computer finished calculating it, you had to open a door and WALK INSIDE THE COMPUTER and count the lightbulbs. On = 1, off = zero. There were banks of bulbs...
He said the computer itself was huge and took up most of a warehouse in Northern Virginia. This was all in the late 1940s, and he worked for the Pentagon. He said the pocket calculator he got for opening a bank account was more powerful than the humungus machine he had to deal with in the 40s.
Sorta puts things in perspective.
I asked him about reliability of components, and he said they had a problem with mice for a while, but due to the large voltages this thing ran on, it was usually a self correcting pest problem, and one easily detected: the smell of burning fur is rather distinctive...
RS
Shoes for Industry. Shoes for the Dead.
>The Wikipedia has a "computer bug" entry that lists some other "famous bugs" including the fictional HAL 9000 bug.
.36 seconds during the 4-day continuous siege, the error increasing with elapsed time since the system was turned on. This software flaw prevented real-time tracking. The specifications called for aircraft speeds, not Mach 6 missiles, for 14-hour continuous performance, not 100. Patched software arrived via air one day later.
Yeah, it lists them, but doesn't really link to good stories -- so...
An error in a single FORTRAN statement resulted in the loss of the first American probe to Venus.
Software reboot during the Apollo 11 landing forced Armstrong to manually land the lunar lander.
An Iraqi Scud missile hit Dhahran barracks, leaving 28 dead and 98 wounded. The incoming missile was not detected by the Patriot defenses, whose clock had drifted
The Ariane 5 satellite launcher malfunction was caused by a faulty software exception routine resulting from a bad 64-bit floating point to 16-bit integer conversion.
lots more here and here.
There is much cruelty in the universe, John.
Yeah, we seem to have the tour map.
Tracking #: 121144608
Title: Bush robot constantly makes grammatic mistakes and makes up words.
Problem Detail:
Corporate puppet robot model George W. Bush (serial #44625441) exhibits erratic grammatical behavior when deviating from scripted speeches. Often uses words like "subliminable", "methodological", "mispronunciated", "stregic", and "permanency" in place of their English equivalents. Platinum users (Haliburton, Exxon/Mobil, Chevron, Bechtel, Kenneth Lay) have noticed other erratic grammatical behavior, including such phrases as "is our children learning", "we need to make the pie higher", and "will the highways on the internet become more few". Strongly suspect some Jim Beam spilled into the model's grammar logic circuits during an all-night instructional binge session with Barbara and Jenna. Suggest immediate implementation of gaffe-filtering algorithm on all corporate media modules to limit the damage from this bug.
Problem Resolution:
Media filters in place as of 12 SEP 2001. Language errors are no longer being reported in the corporate media. Suggest further workaround of detaining at Guantanamo Bay register all non-corporate media modules that are incompatible with gaffe-filtering algorithm.
"You done taken a wrong turn."
-Bill McKinney, in Deliverance
Taking logs...
Not computers but my days in the Navy were all about logs. Every 15 minutes. We logged, calculated and signed everything, about 50 parameters on the nuclear reactor plant control panel alone. Funny thing was I got VERY sea sick. Not a problem on a submarine when it is submerged but pure hell for me on the surface as a submarine bobs like a cork and has no windows to see the horizon for my internal reference. I would utilize a garbage bag about every 15 minutes until we hit the end of the continental shelf and could submerge which took about about 6-8 hours on the east coast, and under 2 hours on the west coast to reach. To try to limit my sickness I took motion sickness pills (they were commonly refered to as the pink pussy pills). These worked a little but limited my straight on vision. Basically I could not see things things very well that I looked at directly but could see the peripheral things. Odd situation since I was the reactor operator. The puking, lack of 100% vision and the emotional issues of knowing your leaving the world for a few months made this an interesting experience.
Bad boys rape our young girls but Violet gives willingly.
Back in the day, we had bugs the size of little bitty moths -- not like these modern, new-fangled, gargantuan bugs, the size of whole windows.
...and we were grateful!
I have been developing code for 30 years now.
Early on in my career, in the era of large decks of punch cards, I dropped a deck of cards on the floor.
I picked them up and put them back in the right order (an ugly job).
When the job was submitted and the print-out eventually returned (1 day turnaround), the compile failed. I was surprised as the deck was basically unchanged from a previous run.
I checked the output and discovered a syntax error. I then checked the card deck and discovered an insect that had gotten squished into a hole punched in a card, which changed the resultant character and caused the syntax error.
Nowadays, my bugs are all my very own.
Back to unit testing ...
Upon opening the box, we found a mouse had been living in the box, died in the box, mumified in the box, and finally his old nest caught fire (well, maybe not on fire, but blacked it). We're not sure how long the mouse had been in there, but it was long enough to gently bake him to perfect mumification. The theory was that with all the moving going on, the mouse had gotten in through a propped open door, through an open accessory panel in the back, and made a nice nest in the warm computer. How he actually died, we're not sure. Maybe he killed himself from the misery of NT 3.5.1 because *I* sure entertained the idea.
Then there was the time we found out that the entire DNS for our networks in France was on an LCD 486 laptop, originally used to test the DNS setup, but then it never got updated at production, and had been running for about 2 years before it failed (we found it sitting on a desk in an abandoned office, the original employee long since moved on).
Case #3: Rats nests inside the computers chewing on cables etc. Big problem at one Texas co-lo. Had to replace all the ethernet cabling.
Willow Run Labs of the University of Michigan (of BOMARC / Sidewinder fame) built their DIANA analog computer (those were the days) in an old bomber-plant hanger. Room with raised floor in giant wooden building built on a slab, in a rural area.
So of course some rats got into the area under the raised floor and started chewing up the cables.
So they got a cat. And they took out a square of raised floor. Cat would go out thorugh the guard station to do his business, then come back in and dive under the floor to do his work.
This being a classified site, there was a 24-hr guard. Everybody had their badge, which was left at the guard station when out, pinned on shirt when inside.
In good military tradition (for instance ship's cats and other working or mascot animals are on the personnel roster and recieve commendations and court-martials for exceptionally good or bad behavior), the cat was taken to the security office, photographed, assigned a number, and had a badge made.
And from then on, when the cat came in he'd stop at the guard station while the guard clipped his badge on his collar before he dived under the floor, and again on the way out for the badge to be removed.
The cat seemed to have no trouble with this procedure. (No doubt because he saw that everybody else had to go through the same thing - except for doing their own badge pinning.)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
You see, the opti 895 was a chipset for a i486 processor based motherboard. The 486 processor's ZIF socket (The mdern kind with a lever, before that you had to press the procesor into a socket and hope that you aren't breaking the traces on the motherboard) had an extra row of pins to accomidate the Pentium OverDrive Processor. This processor actually put a P5 core in a motherboard designd for a i486 processor. The nifty thing was ha it worked at all.
Getting to the bug: The outer row of pins on the socket for the 486 were only power and ground for the extra power consumption for the PODP. The specs were clear which ones were Vcc and which were Vss. Well, the opti 895 had 2 of the pins backwards. This was never found in testing. Many many boards were sold from various Tiwanese manufacturers. The boards ran fin until you purchased and installed a PODP into yhe board and powered up. The chipset would short, get HOTHOTHOT, start glowing, and burst into flame within minutes.
This was bought to out (I was working for Intel as OverDrive Processor support at the time) about a week after product launch. Can you imagine how that call went?
Caller: Uhhh... I installed tha part into my computer and it burst into flames...
Tech: Yes, the speed improvement is quite impressive.
C: No, you dont understand. My computer actually caught on fire.
T: (silence)
C: Hello?
T: Am I to understand that you have a fire in your computer?
C: Yeah, the smoke is getting pretty bad.
T: You mean to tell me that it is STILL ON FIRE?
C: Well yeah, the manual says to call you with the system in the current condition.
The motherboard was sent in (we replaced the system with a new name brand machine) and the chip was redisned so that one of the pins was removed. (Pin A4, IIRC)
I have NO idea how many motherboards we ended up replacing , but I know it was a bunch, even though it wasn't Intel's fault that opti couldn't read a pinout diagram.
Problem description: At approximately the same time in the morning, on average about once a week, a job (different job each time) would fail with an I/O error on a specific 7-track tape drive.
It took over a year to track down the cause of this problem, which was very costly: the jobs were often time critical and mainframe computer time was costly anyway. We had top hardware CEs and systems programmers looking at this from every conceivable angle. Just about every component in the tape drive was changed.
The mystery was eventually solved by an observant computer operator. The tape drives were on the second floor of a building with a road passing just outside. At that hour in the morning, if the sun was shining, it was possible for the sun to reflect off the windscreen off passing cars and flash briefly on the read head of the tape drive. The tape drive interpreted this as invalid data.
During the industrial year of my degree (mumble) years ago, my first task was to modify a Fortran 77 engineering program which calculated intersection points between two pipes, so the correct cuts could be made and the pipes joined. We're talking big pipes here - the company built the Syney Harbour Bridge and the Hong Kong Shanghai Bank building in Hong Kong.
Lacking the modern tools we all love, debugging tended to be done by printing values at pertinent points. When the code was correct, I removed them all... and it broke; I put them back in and it worked; commented them out and it still worked; deleted the comments... and it broke again! These were basic, fundamental print statements, no fancy function calls with side effects. I eventually ended up with two 100+ page listings of the object code (working and non-working) side by side on the floor and had to compare the lot by hand until I found the difference, near the bottom of course!
It turned out to be a bug in the PDP Fortran compiler. It was incorrectly generating two identical labels in the same code block, but for whatever reason they were together in the working version and had a register being reset to zero between them in the broken version - the JMP was going to the second and therefore not resetting the register.
As an undergraduate at the time, I was in despair... my first 'real' job and I couldn't fix a simple program - little did I know what the final cause would be - nearly put me off software development for life! Bloody DEC and their shonky compiler, they didn't even give the company a free upgrade when the fix came out!
Go permanent? In your dreams and my worst nightmares.