Slashdot Mirror
What Bizarre IT Setups Have You Seen?
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
How about this one?
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
Why put that on there?
Just to clarify- the U-Bend is what prevents bathrooms and drains from smelling horrible. Inside the drain, shower water, sink water and toilet waste all mix together. As you can imagine this smells horrible. So, where every toilet, sink, shower, etc connects to the drain system there is a 'u-bend'- a downward dip in the pipe which stays full of water. This prevents air from flowing out of the empty drain.. png
Most sinks have their u-bend visible under the sink and look like this:
http://twenteenthcentury.com/uologos/ubend_shaded
Water flows in the top, and out the back. Because the back is higher than the bottom of the bend, the bottom stays full of water at all times, preventing air from passing.
Problem is, if you leave a drain long enough without water passing through it, the water in the u-bend can evaporate, leaving an empty pipe and allowign the nasty sewer smell to escape. Thus, leave a faucet dripping to keep the U-Bend full!
--IronHelix
Company moved into a new, larger building. The server room had a heating vent leading into it, and no A/C. They solved it by clogging the vent with a bag full of shredded paper and cutting a hole in the wall to install a small consumer single-room air conditioner.
Rur datacenter has plenty of servers that are not on rails, which is bad enough in itself but what amazed me is seeing 4 or 5 2U servers resting upon the following.... 2 old SCSI drives, a Solaris User Guide book and some other manuals that were laying around. No one knows how this setup came about since it was before all our time at the company.
I've managed to use some industrial strength velcro as an additional set of hands.
Two worst I've seen: 1) While I've chopped patch patch cables in half and turned 'em into crossovers, this one place I toured got a good deal on pre-made crossovers and chopped & spliced them into patch cables for over 50 PCs; 2) Where I work now, a former employee jacked a cable modem straight into a Win9x peer-to-peer network, despite my protests (scary part of that was that he said, and I quote, "Oh, I do this all the time and it's never been a problem before." I spent the next week reinstalling Win98 and software...)
"You know you're narcissistic when you quote yourself in your sigs." -- PRoPAiN!
I've generally known that contraption as the 'trap' -- although I recognized the descriptive name 'u-bend',
I'm just wondering if this wasn't the same drain that they were using to run their ethernet connection, That would explain why that sink was 'never' used.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
The fire marshall came in and said "you can't have those low-voltage wires run through that conduit, that conduit is designed for high voltage wiring." So the electricians came in and sawed off their beautiful conduits, leaving the wires draped between the four-foot-spaced supports. They tie-wrapped the bundles every foot or two, but it still looked like a dead python hanging between branches.
To this day I still can't fathom what the hell that inspector was thinking.
John
I worked at one place where our room was a couple of floors underground (very depressing place) and we wanted to listen to the cricket on the radio (pre internet days). Armed with a crappy radio we found we could get perfect reception by connecting to the air conditioning vents with a set of crocodile clips purchased from Tandy's.
Another one I remember is something low-tec invented by some admin staff, we had a policy set in place that locked workstations after 5 minutes of activity, the PC's were severely locked down so you couldn't change this. Turned out the admin section of the company despised this as they would do something on their accounts package, talk to someone on the phone and by the time the phone call had ended the PC had locked itself requiring their password to unlock it. One lady actually took a small clock, took the plastic front off and attached a piece of paper to the second hand, when she wasn't doing anything, she placed the mouse in front of the clock so that when the second hand went past, it moved the mouse slightly stopping it from locking. When the guys in tech support found it, she was visited by practically every IT person just to see it in action.
Task Mangler
I've seen untwisted coat hangers covered in electrical tape and twisted together used to supply AC between two buildings in tropical weather in PNG. The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless. I should also point out that they did not have power outlets for the computers either. They just cut the plugs off, stripped the wires, twisted them together and covered it in electrical tape.
After begging facilities since the previous year to upgrade the AC (and having one last big machine installed), we 'solved' the problem by buying a small, window-type AC, and poking it out the door. With this setup, we could generally get the room to stablize at around 30C (about 86F).
This worked until facilities showed up and complained that we needed to go through them to get any sort of AC installed, and demanding that we stop using the offending unit. (but required us to continue with the un-responsive process of getting the room AC upgraded).
Peter resolved the impass by calling the health and safety group, and keeping the door closed until they arrived the next morning to inspect a worksite with a temperature of over 100F.
The AC was upgraded in well under a week.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
My previous job was in Network Operations at a University. Our Marine Science department had a large grant-funded sensor network running in a river somewhere in South Jersey that needed to talk to their machines on campus. They did this by getting a 56k leased line dropped out to the end of a long pier, to which they connected a cisco 2500 series router (state of the art at the time). It was housed in a box with just enough ventilation to keep it soaked in condensation, but not enough to allow for adequate cooling. Because of the heat it was on a permanent shutdown/reboot loop for most of spring, summer and early fall. They were lucky if they got more than a few hours of readings per day.
Probably can't guess why this is anon...
... a Java system. Last time I checked, Java was supposedly this network programming language and has these really poor supporting libraries for distributed computing....
Worst running system: Exchange 4, 5, and 5.5 running in a single Exchange site for a former company. Needless to say, there were some issues with their email.
Most bizarre development: insistance that a newly designed and developed java system use CORBA to connect to
Most? bizarre configurations:
Use of a second CORBA stack in an appserver that comes with one.
Using Access for a multi-user critical application
Using Excel as the budgeting and financial tracking tool for a multi-billion corportation
Using a third party POS layer built on a RDBMS as a RDBMS.
On an essentially synchronous SOA implementation, the request/response pattern included the request within the response. One of many inefficiencies in this design.
About ten years ago, I was working for what was then a small, startup ISP doing tech support. For about the first two years I was there, we often had to talk new customers through locking down their modems to 2400 baud in the registration/installation program, because that server often worked best at low speeds. (We also showed them how to reset it to the proper speed afterwards because our POPs were just fine.) I later found out that this was because whoever set up our one and only (at that time) registration server had multiplexed 42 modems through one COM port.
Good, inexpensive web hosting
I had an instructor who used to work in industry. He'd told me about a company he was consulting for. They had a Novell box that they administered remotely. During some remodeling, the small closet/room it was in was sealed with drywall. It was 4 years before the box required maintenance and someone went about trying to find it and realized what had happened.
There was the time back in the late 1980s when the multiplexed underground cable that my college was using to provide terminal connectivity for their new-fangled computerized class registration stopped working, and they ran dozens of hastily-created RS-232 cables from the data center to the hall where the action was, half a block away... secured to the sidewalk by duct tape, of course. Which at least didn't remain in place as long as the 10base2 cable that connected two dorms, strung between their 2nd floors (until it was taken out by a lightning strike).
More recent ugly hacks that I can claim personal credit/blame for are mostly of the sort that involve pulling a rabbit out of my ass because a solution needs to be found By Tomorrow Morning... like for deploying 200 installations of Windows 95 in a week (in the days before Ghost, or even backup software that preserved Long FileNames) using DOS boot diskettes, Netware, a utility called lfnbk, and ncopy... or building an e-mail server out of RedHat Linux 6 and spare parts (no, I didn't even have a complete working computer at my disposal) when the company's glorified BBS mail software found itself unable to exchange mail with the standards-compliant system used by a major new business partner.
http://alternatives.rzero.com/
When I got hired as an Information Specialist for one of the government sponsored agencies in Hellinois, the people there would write their e-mails on a piece of paper and give those to their previous IT guy. He would then type them up and send them out via a yahoo e-mail. No kidding.
At one point, they had changed their routing so that they were using the new link but we hadn't, so we decided to see how a ping went.
A packet between the two machines would go through our router, over the ethernet that the two companies shared, out the (old) external router, and down the coast through Seattle, to California, then back up the coast to Vancouver, and then finally over the same shared ethernet cable that the packet had originally gone out before finally connecting to their router.
A cross-border round trip of a few thousand miles for a net distance of about 60 feet.
Oh, and did I mention that our server room was a converted bank vault?
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I helped design a setup that has an insane security system that's unheard of. It's trap based, as it everything I do. For instance, there's a keyboard sitting on top the desk and a keyboard and if any key is pressed on it, an alarm goes off. The real keyboard is on a pullout tray beneath that and all employees allowed to use the system are told that. If anyone sits down and doesn't know that, they don't get to use the computer. Simple yet effective. There's also some other absolutely 100% hacker proof crazy things in the setup that haven't been used anywhere else before because I invented them but I obviously can't reveal exactly how they work because that's part of why they work. But trust me, it's probably the most secure setup in the world and any hacker or person that broke in intending to do any sort of digital harm would end up confused and arrested no matter how skilled they were.
Google's Super Secret Search Algorithm: SELECT @search_results FROM internet WHERE @search_results = 'good'
I know the poster was looking for funny/interesting anecdotes directly from our community, but for those of you who haven't stumbled across The Daily WTF, hop on over to that site and make it a part of your daily reading. While the focus used to be mostly on programming, it's abstracted itself to the generic IT level in recent months, and you'll see all sorts of bizarre stories there.
:)
The Daily WTF is to IT workers what Jerry Springer is to everyone else. Just when you think you're having a bad day and your life is in the crapper, you can take a few minutes to soak in a situation where somebody else has it much, much worse...
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
I worked for an ISP that ran everything on one single SUN Solaris machine i kid you not! and it was so old when it rebooted it took over 30 mins to come back online thats if you where lucky sometimes it never came back. The poor thing just wanted to die and stay dead.. The funny thing is six years on and they still use the thing, but have over 20 servers now running linux.. not sure if thats saying something bad or good lol
I wanted to try out the option to have the server page me in case of problems. Only problem was that the only phone jack in the server room was on the other side of the room, and I didn't have a phone cable nearly that long. But I did have a box of old ISA modems and short phone cables. My intuition told me that the "Line In" ports were wired directly to the "Phone" ports and didn't require power or actual computers to drive them. So I daisy chained modem cards and short cables together across the ceiling, wedging the actual cards behind cable housing and drop ceiling tiles, until finally I got dialtone. My supervisor commended me for my creativity but made me take it down, since the policy was that the modems were not to be connected to phone lines for fear of people being able to dial in to them or something. Never mind the dedicated internet connection.
Back in the day of Atari ST computers, a friend of mine purchased several boxes of 5, 10, 20, and 40 meg hard drives from a computer store bankrupcy auction. He then build a wood shelving unit that mounted to the wall over his computer desk, and set all the drives on the shelves, using cereal boxes as separators. He then wired them up to scsi controllers, 6 drives per. On one end, he had a set of lighted low voltage toggle switches attached to low and high voltage regulators. It looked like an old computer from the 50's. Each drive had a label, and if he wanted something off a particular drive, he'd flip several toggle switches, throw the main power switch, and boot up. The entire wall would vibrate with the drives spinning. It was great to watch. A couple of drives would not spin up sometimes, so he'd pick them up and shake them until they started spinning.
Another one of my favorites was this database system running in the catalog sales office of an art gallery. I had taken them on as clients, fixing bugs in this database, and working on a migration path to Foxpro. One day I got a call that they had garbage data for the two years prior, when they went to review a customer's history. Turned out the hard drive was full, but the database system was happily writing new data over old files. Fortunately, I had all the data on tape as part of my development efforts.
Try this:
I was the "computer guy" at a fabric processor in a town in Eastern PA that Shall Remain Nameless. Being "The computer guy" meant that they blamed me for the outages, but of course gave me no spending authority to do anything to fix the problems...
About 1 month into the gig, I was in the front office which was connected to the computer room by fiber optic cable (probably the smartest thing this company did.) However, once the fiber terminated at the switch in the office, the horizontal wiring to the workstations was, God help me, silver satin cable. Telephone wire. The shit was everywhere. There were about 100 workstations salted through the plant (which ran high voltage AC and heaters and whatnot) and everyone complained about the server performance. I wasn't even allowed (!) to put a network analyzer on the wire and was too naive/stupid at the time to realize what the problem was. The guy who had the spend authority, the "chief engineer," told me the problem was lack of RAM in the server and was always harping on me to upgrade the memory.
Another time I opened a closet to find a splice of this satin cable (they must have bought it surplus, they had hundreds of reels of the stuff) and the splice was made with, I kid you not, wire nuts.
I lasted 18 months there. I heard they brought an ex-Accenture conslutant in soon after to fix the "computer problems" and she ran the company into the ground.
Mit der Dummheit kämpfen Götter selbst vergebens.
Many computer rooms have packaged units which both heat and cool, and some also both humidify and dehumidify. That's fine if you only have one. If you have more than one, they need to be interlocked so you don't get one cooling while another is heating, or one humidifying while another is dehumidifying. If you get into that situation, everything will seem to be just fine, but your energy bills will be maybe 5x what they should be.
Saw that situation in a server room at Stanford a few years ago.
I have a bunch of stupid cobbled together setups to talk about. It all comes from a combination of poor IT staff at university wages, infintessimal budgets and the overbearing institutional and faculty pressures.
i braries/sl8500/ (its $200,000). Luckily this one didn't actually come to pass.
1. A "server room" that was essentially the most worthless room in the entire building, a long skinny room with four windows (perfect for keeping an uneven temperature!). Rather than buy 19" racks or even wire racks, they found a bunch of tables and put one server on each all the way around the edge of the room.
1.a. All of the servers were in fact desktop systems; an Ultra 1 was the mail server, a SPARCstation 5 the print server, a Gateway Pentium Pro 200 desktop the web server, etc.
2. A lab had to be moved one room over, because its current location was deemed too valuable. The original room was designed for a lab, it had 20+ fiber optic networking ports, twist-lock power connections in the ceiling, that sort of thing. The new room had two electrical outlets, no dropped ceiling, and one fiber optic networking port. It had previously been used as a copy room/storage closet. The cost to move the fiber optic wiring (just one room over mind you!) was over $25,000.
So instead, I had the great idea to cut a hole in the common wall (above the drop ceiling line), purchase additional ceiling tiles and cut up 2x4's into wooden supports. The original ceiling boxes containing the networking were put on top of the blocks above the new tiles, and extension cables run through the wall into the new room. In the original room, which was turned into a lounge, you couldn't tell that there was anything funny going on.
The best part is that the lab manager, who insisted they needed every single network port, never used a single one of them in the new room. All of those cables now reside in a box marked "Giant waste of money".
3. The main Windows file server was purchased in 2002 and has an internal RAID (bad idea in my opinion). What was huge then is worthless now; 5 disks that total 135GB. To get more space, the administration begged for a single external 250GB USB drive to host all user data. Nevermind that there is no redundancy, that an external drive is more suspectible to theft or failure, and that USB is unnecessarily slowing things down.
4. A system administrator got it into his head that rackmounting was the way to go (I agree). So he begged for a 19" rack to be ordered, and placed all of his servers into it. Except he doesn't have a single rack mountable server, and he didn't get the rails for any of the cases either. So now he has one $500 rack, and 8 $100 shelves to go in it. Same guy also switched the KVM monitor to a 15" LCD that doesn't support the resolutions of 9 out of 10 systems connected to it.
5. A consultant was brought in to tell us what needed to be done with the computing infrastructure (what DOESN'T need to be done is more the question). His main suggestion was to set up a central backup service just for this college, so as to avoid paying the central university IT group fees to use their central service. OK, thats an idea I guess... except that he wanted us to buy this: http://www.sun.com/storagetek/tape_storage/tape_l
Basically every day is a new adventure in ridiculous IT methodology.
Heck, feel glad that the "U Bend" (a.k.a. trap) had a water faucet. when maintenance boxed in our sink they took off the spigots off the mop sink, but left the drain functional. Then they boxed it in plywood. After the A/C was installed (and had dehumidified the room) - and the building's humidified air was shut off to the room .. no more humidity for us, and the drain dried out.
.. ugly hacks?
... that someone stuck a 10base-T hub on at 200 feet (give or take) because the hub actually strengthened the signal? Then they filled the hub with 8 10baseT computers..
.. they put in BNC .. then ran ARCNet over it (averaged about 300kbit/sec). Then complained it was too slow (well no kidding!!)
... after they'd sliced through it moving office furniture ... (yeah, it kinda worked)..
Hmmm
How about a Netware 3.x server stuck in a closet between two 10base2 (coax) runs, connecting one segment with another (glorified IPX router).
How about a 395 foot run of BNC
How about a 450 foot run of BNC with old, frayed screw on ends. That ohmed out at about 76 ohms. And they wondered why the network was slow.. (I re-crimped all the ends and cut the wire approximately in half, and used a second NIC in the seerver). In that same place, one of the workers figured out that if they took one of the BNC connectors off the T in the back of the PC, the network would go down and they could just sit there and do nothing...
The company that had a 1000 foot run, so instead of buying ARCNet wire
The BNC wire I saw that someone had repaired with a paper clip and electrical tape
And we won't even talk about how many networks I ran into that looked like
Server ----- hub ----- hub ----- hub ----- hub
and they wondered why the people on the 4th hub would lose server connections randomly..
= Grow a brain...
Once the graphics cards were removed, the machines defaulted to booting with Serial consoles. This meant that if anything went seriously wrong, just about anything other than hardware maintenance could be done by SSHing to machine X and using a terminal program to connect to the console port of machine Y (or vice versa).
This included the ability to do a complete wipe and install, needing only to instruct the CoLo staff to insert the install CD (which were left on top of the machines) into the appropriate box.
One of the monitors ended up on my desk. I can't remember who got the other one.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I don't think I've ever hit a limit on single SMTP attachment size unless sending to freebie providers (Hotmail, Yahoo, etc). Although I'll admit that if I want to send a large file to more than one person, I'll usually put it up on a server somewhere and send a link instead of forcing the file upon the recipients; and most of my dealings with emailing insanely large files (sometimes into the gigabyte range) have been in enterprise environments run by competent admins.
I do run into "quota exceeded" from time to time. This happens less and less frequently, though, particularly with even the free services constantly trying to one-up each other. My Gmail account says I've got 2801 megs of storage, which is far more than I'll ever use, but I imagine they'd give it to me if someone sent me that much.
You've got me trying to tease a long-dead part of my brain that wants to remember what the upper boundaries were on AOL back in the day, 1994 or 1995. They implemented quotas in a unique manner. I want to say that your mailbox could hold a maximum of 550 messages, 1100 if you worked for them, regardless of the message size. Their attachment limit was 16 megs, I think; this would have been nearly unheard of at the time with the exception of all the warez traffic.
Anyone out there remember the details more clearly than I do? We're talking 12 years ago, and sometimes I'm fortunate to remember 12 days ago...
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
I and a few guys were doing customer phone support in a remote building (ten years ago or such some). Soccer euro cup was up, and a collegue was desperate to find a way to watch the games, as the company (ISP) has just started operation, and callers were few and knowledable (so it was actually fun). Opening the cable funnel, he saw a TV cable. He spliced it up and connected it to a RJ45 jack. He then installed a TV tuner card into his PC, build a network cable look-alike to connect the TV card to the fake network jack, and voila - you could not see he was tapping the TV signal (the cable funnel was very visible, the computer was under the desk).
:-)
As we left the building about a year later, the fake jack was left there. I wonder what kind of head scratching this caused for the future tenants
The AC guys were fixing the unit on top of the building. They ran a garden hose through the overhead space above the IT department to pump green coolant fluid from one unit to another. The IT director heard a loud rumbling above his work area and evacuated from the area. The garden hose exploded. Coolant fluid broke through the ceiling panel to soak the 19-inch CRT monitor. That was a mess.
When I worked for one ISP, the main servers were connected to the internet via a stack of OC-3 links, but the building where all of the admins, support people, etc worked (3 floors of a decent-sized office building) was served by a single T-1 line. The ADSL link that one of our technical support people had on a test bench (it was for testing, so we couldn't route general traffic over it) turned out to be much faster than the link that the rest of the building shared.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
The PR firm I worked for had a 66mhz 486 box with 24mb of RAM... this computer was running NAT, mail server and web server on Linux. It ran adequately, 24x7 for probably four years until a hardware failure. Suddenly there was no network, mail or web server.
There was an identical PC chassis collecting dust... I took that, installed the impressive quantity of 72 mb of RAM in it (maxed!), a new drive; two identical Ethernet cards; installed NetMax "firewall in a box" on it; transferred e-mail and web service to a vendor we were working with (the other unsung heroes of the story) and in about 72 hours after propagation, all was well. Total expenditure on this "rescue" was about $250.
This day-and-night effort to put the network back on its feet earned me the undying skepticism of the CEO, who postulated that I had somehow caused the old setup to fail in order to justify new purchases. So it sometimes goes. YMMV...
You do know that that grommet is to keep the sharp edges of the case from cutting into the cable?
In the early 80's, I was working for a company that did lots of its own kernel hacking on UNIX and VMS systems. They had a habit of implementing lots of their own software systems, rather than using standard ones. Some were not very clever. For instance, they had a communication "protocol" that ran over ethernet cable, but it didn't handle collisions. Yes, we had thick ethernet running to every office, and when anyone wanted to use it, they'd run out in the hall and yell to make sure it wasn't in use. If there was contention, data would be corrupted. Eventually, we punted on this stupidity and used TCP/IP.
A small business asked me to help them with their reliability problems. The company had been there for 20 years and the cabling for about 6 workstations and 4 or 5 big production printers consisted of premade cables tacked on the walls and ceiling, and one or more ancient efforts by electricians. There were serveral 10/100 hubs to extend the distance or 'multiplex' a single cable back to the internet connection. I went in several times to plug power into hubs, or pull cable out of a doorframe where it was being pinched. It was apparent that the main issue was this messy, poorly designed cable infrastructure.
So I got a data cable guy in to quote a clean installation-- home runs back to a single switch, cable in conduit, all clean and reliable. It was a few thousand dollars.
The owner decided it was too much $$, so he got an electrician to do it and now he has 3 more switches and even fewer home runs than he had before. But it's now Cat6, so he somehow thinks it's better.
I've already been back to troubleshoot... bad cabling.
A Very Large Telecom Corp(TM) had let a contract for a hardware subsystem that was to be connected to their very expensive network monitoring system (probably HP Openview). Anyway, the vendor couldn't quit make this work. So, to satisfy the contract, they had a tape monkey with a laptop in the NOC. Whenever an event happened on the subsystem, he'd manually copy the message into a dialog box on the master monitoring system, at which point it'd pop up on the regular NOC alarm system...
"Not an actor, but he plays one on TV."
This happened just this past year.
We had moved into larger building with a server room in the basemnent (cue ominous music).
We rapidly began to run out of space so decided to place the chief sysape in the basement near the servers, which made sense. We cleaned up some items in the basement, moved them into storage, carpeted, dry walled etc. Since it was in the basement it needed an egress window with a steel casing and ladder. This actually turned the office into a nice garden level. You could look out the window and watch the sprinklers, see trees and grass etc.
On day, the chief sysape comes in and notices water on the floor. He looks over at the egress window and there is about 2 feet of water collected in the base of the exit well.
Well, they shut down the water to the entire building. Luckily the server room actually had about an 18 inch raised floor, so no damage.
To make a long story short, upon investigation it turned out that when the sprinkler system was installed, instead of capping off the ends of the plastic piping, they folded it over and crimped it. They relied on the mass of the dirt to keep the ends crimped, and for years it worked. Until the egress well was installed and the dirt was disturbed. Once it was disturbed, the crimps began to fail under water pressure. Leading to a near IT disaster.
putting the 'B' in LGBTQ+
I once worked on a project where we were developing a new telecom product. Our development servers and source code repository were in a lab with no uninterruptable power supplies (UPS's). We kept having our servers rebooting at odd hours. Then we found out that the cleaning crew was using portable "backpack" vacuum cleaners. They would plug into a wall outlet that was on the same breaker as our lab. When the breaker popped, they would reset it, and continue cleaning. There was a tape backup to preserve our source code, but unfortunately the backup was bigger than our tape, so the backups kept failing silently. We requisitioned some UPS's, and they kept failing. We found out later that the IT guys had inventoried some bad UPS's and those were the ones we got. This was the same project where I had to go to the PC graveyard and scavenge parts to build our production servers. Serious hackage, but a pretty fun project to work on...
The machine with the user accounts on it had a few more hot, high-speed disks than the case was really designed to keep cool. It got hot and beeped. My boss wouldn't consider replacing it or even getting a new case. So I was forced to improvise: I cut a hole in the front panel and fitted a spare case fan into it. Then I realized that the motherboard didn't have another power connector for the case fan ... but I had a spare 5V wall-wart. A little wire-cutting and electrical-taping later, I had an externally cooled disk bay.
That "machine room" sucked. It was in the corner of the basement of a college office building. In the winter, the (crappy, household-type) AC unit iced over and the servers overheated. One summer, the facilities staff decided to power-wash the wood siding of the building. High-pressure water ran up through the wall and rained down right onto the server shelf. The only thing that blew up was the fancy new monitor that had come with the expensive and utterly overpowered RS/6000 just purchased by the library.
A couple of years ago when I visited the campus, they were still using that wall-wart-powered fan to cool the disks ....
-b.
When I started working at the school where I still work we were in two separate office buildings separated by more than 100m. I eventually ran coax through the wall via a light fixture, along a fence for about 100 feet into a tree to the roof (where it was held down by a sack of river rocks attached to some plywood) over the roof down a rain gutter and under the door. The building landlord was actually ok with this setup. It was mostly hidden except for the hop from the building to the fence and the whole tree to roof span.
Better than what a certain electrician did in an office that was being renovated. The Ethernet wiring was already installed. He decided, for some reason known only to himself, to pull BX cable (the heavy, metal-armored electrical cable) through the same conduits and cable grommets as the Ethernet cabling. Sans lube. Needless to say that a lot of new Ethernet cable got pulled to replace the sliced up cables! (He also had actually removed some cable grommets where the cabling went through studs to make more room for the BX cable since he was too freakin' lazy to drill extra holes!)
-b.
OK. This was in the late nineties for a small computer hardware firm that had been in business since the '70s... but still, there was no excuse for this. It was a rambling wreck, a crazy collection of every ethernet standard implementation, and a few that were decidedly non-standard, just sort of tossed into place as time went on.
The backbone was a five port AUI concentrator... it was too primitive to be called a hub. (AUI was Sun's insane proprietary ethernet connector.) Hanging off that was a Sun server that was shiny and new when the Soviet Union was still in the news, which was the router to the DMZ, and a media adapter for thicknet. That thumb-thick yellow cable snaked over to the engineering cubes and hardware labs, with "vampire taps" hanging off it everywhere - vampire taps have a screw that drills into the cable, which is how you hook stuff up to thicknet. No lie. These were connected to 10Base-5 thin-net adapters, which hooked up to co-ax concentrators, which hooked up to AUI media adapters which hooked up to the various Sun workstations. I had never seen before, nor have I seen since, a BNC co-ax hub used just to hook up workstations in a star topology... for whatever reason, they decided that ring topology wasn't good enough to string five lightly used workstations together. I have no idea why any of this worked. It usually didn't, and needed various pieces of arcane equipment power-cycled and jiggled and cursed at to get any data to make it from the file servers to the workstations.
It gets worse. Another port on the AUI concentrator went to the Cabletron TPT-2 setup, which took care of accounting, sales, support and executive row. This was like 10Base-T ethernet, with a patch-panel that was wired to RJ-45 jacks in the offices and the cubes, except it was completely incompatible with 10Base-T equipment. Media adapters for all! And when one of the adapters goes down, the whole TPT-2 system locks up, a hundred or so systems. Let's play the hunt-the-locked-adapter! So much fun when the CIO is screaming at you.
I went on vacation, and the engineers were left to figure out how to bring the network back on when one of these adapters froze. You'd think they would unplug the patch cords one at a time in the computer room until the network came back up, but no. They just remembered that I told them power cycling an adapter would usually bring it back online. So they powered down the building. Serious. They needed to reboot the building... by this time all the critical systems were on UPS, so nothing was fuxxored, but still.
I eventually got the penny-pinchers in charge of the business to invest in nice 100B-T and 10B-T switches and AUI adapters and a few nice new Sun servers. Worked much better thereafter.
I was once working in a datacenter on a machine close to the company email servers. There's a message on the console for the email server saying that the email database was corrupted and that some utility had to be run to fix it. An IT guy walks up to the console and looks at the message. He then proceeds to reboot the server. After the reboot, the same message appears. Reboot again. Same message. Reboot again. Same message. The IT guy repeated this 10 or 15 times. Then he walked away perplexed at why the email server wasn't working.
-b.
College internet service that is 45 Mbit in to over 4000 students living on campus, over 8000 total, switched to 100 Mbit to go between academic buildings, 10 Mbit to each dorm, each dorm then has a 10 Mbit switch in it and each floor has a 10 Mbit HUB (some dorms got lucky and had theirs replaced with switches). That's right, 10 Mbit hubs! Oh, and this isn't a flashback story, this is how it is now. Plus side: They are upgrading to 90 Mbit so that students will stop complaining about the slow internet speeds. Larger plus side: That's still nothing, students will still be complaining, some of the students get 30 / 4 to themselves at home.
"Fill the trap with cooking oil - it will stop the smell and will not evaporate as quickly as water would."
:)
A good example of lateral thinking. Good thing they're cloning you.
Furthermore, a lot of the rural phone system was still very old-fashioned. A different uncle's phone # was "42" in a certain small town in the Beskidy Mountains. The exchange itself wasn't automatic - you'd pick up the phone and give the number you wanted to the operator, who'd connect you (or call you back after 5 min. if you wanted long distance).
-b.
If you've ever seen TV coverage of a Progress or Soyuz docking to the International Space Station, you've probably seen the ubiquitous black and white docking camera video with data overlayed on it as the vehicle approached the docking target.
1
Unfortunately, this television signal was only within the Russian Segment, and could only be downlinked through Russian communication assets over Russian ground sites. That limited the video to around 10 minutes each orbit, and required the docking to physically occur over Russia.
The US segment downlinks television via the Tracking and Data Relay Satellites (TDRS), which have more or less worldwide coverage. But the US segment and Russian Segment systems used incompatible video standards and weren't physically connected.
Yup, two video systems that cost tens of millions to develop, and they can't talk to each other. Classic "square peg, round hole" problem.
So we devised a setup where the crew ran a cable from the Russian Segment TV system into an IBM A31p laptop which converts the Russian SECAM signal to US NTSC video. The output from the laptop is connected to another cable strung down the stack into the US video system and downlinked via TDRS. Voila, greatly increased video coverage thanks to a lowly Thinkpad.
Details of this being tested can be found here: http://www.spaceref.com/news/viewsr.html?pid=1879
Worst...sig...ever!
Some of the bizarre setups I've witnessed on previous and current work locations:
* Intranet Application server also running a TV tuner for the workers to enjoy their favorite TV shows without a hassle.
* The server room is actually the living apartment of the boss. He has a cat. The cat is occasionally found napping on top of the servers, despite attempts to keep it outside that specific room inside the apartment.
* A guy running a bunch of servers decided that using electric socket splitters is too messy, so he instead cut a bunch of PC power cable and soldered them directly to the bare wires in the wall.
* Mission critical databases backed up daily to a collection of attached USB (mp3 player) flash sticks.
Your email address indicates that you are from Case Western. Might not want to let your boss see that...
I have a great idea for them to spend money on: Why not invest a ton of money into ATM networking? It's going to be the wave of the future!
Speaking of library computers, where I went to school (small school in SE PA) was using dumb terminals in the library connected to a VAX for their catalog system well into the 2000s - they still did as of my 5th reunion in May 06 actually! The wierd thing was that if you wanted to check e-mail and all of the public lab computers were taken, if you power cycled the terminal quickly, you'd get a command prompt. If you then typed C , you'd be able to telnet into the mail system. This worked fine until they made SSH mandatory in 2000 or so.
-b.
The server room was a fairly large closet with an a/c outlet and a combination of wire racks and IKEA shelves. Nothing too bad, there; it all worked and everything was strapped down in case of a quake. However, to get to the server room you had to go through the breakroom and pass by the kitchen. Which had maybe two outlets and a hardwired coffee maker. Which shared a breaker panel with the server room in the hallway behind both rooms. Can't tell you how many times the admin assistant killed the server room trying to shut off the coffee maker. On a Friday, at 5 p.m... Funny thing is, the Head Cheese only worried about his coffee, not the servers that housed our precious COBOL and account information and wouldn't authorize either a separate breaker panel or a good UPS. Then again, that same admin assistant had to print out the boss' email and Excel spreadsheets and then re-type in his modifications...
Obligatory bash.org quote.
<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
I ran a cat5 cable down the side of a 5 story building to a conference room, because we didn't own the floors in between. The PHBs needed a connection to demo some things. Later I learned that they sold the company, and then they downsized me because the other company already had an IT department, which ran on M$ Windoze. I was a happy Linux+Sendmail+Netscape admin. Great job.
"using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?)."
If you have a u-bend, you do need to keep water in it. Just wait until you see what happens when the u-bend dries out... it's a very smelly affair.
Oh, you're not stuck, you're just unable to let go of the onion rings.
A friend of mine has worked at Novell for quite a while. Quite some time ago (10 years, maybe a little more), he took a support call from a company complaining about how slow their network had become as they added more stations, and that it had eventually become completely, totally, unusable for any task.
It turned out that they had at least 1,000 stations (I want to say 3,000, but I'm not positive) - all using ethernet *hubs*, not switches.
Oh, you're not stuck, you're just unable to let go of the onion rings.
In the mid-90s I did tech support for a regional ISP with a pretty large presence in its home city. We ran our own POP there with several hundred lines. The lines were digital right up to the demarc, where they were split/converted/whatever out to several hundred copper pairs. On our side, the several hundred copper pairs were immediately converted *back* to digital and sent off to our USR Courier digital modem banks. The reason for this was that analog lines were tariffed cheaper than digital ones, so it was actually cheaper for us, even though it was more work and complexity both for them and for us.
I used to be a customer at a GlobalCrossing, which became Exodus facility. It was two stories high. However, the elevator was smaller than those in most office buildings, being that it *wasn't* a freight elevator. However, they did on the second story, have a rollup door installed in the wall.
We (being a service provider that provided storage to the other customers in the colo) had to spin up a EMC Symmetrix (these are 3x2 floor tiles in size) for a large customer. We then had to hire a forklift operator to lift the symm into the open rollup door on the 2nd floor.
In a different colo, we had a 1k sqft cage, and we had quite a few EMC symmetrixes (the 3x2 tile ones), when we moved in there were almost no other cages in this part of the building. When this symmetrix came off of lease, exodus had built up the other cages and there was no way for us to roll it down the hall without taking off the walls of every cage we passed. We then, instead of rolling out to the loading dock, we could only get it through the front lobby as that would require fewer cages to be taken apart. While the moving company rolled it out through the lobby they cracked half the tiles the symmetrix rolled over.
I watched a customer of mine that was running a pair of sun boxes (e450s) say "I don't want to manage my backup database server via telnet/ssh, I need to run X". He them promptly unplugged the keyboard/mouse from his primary database server and plug it into the backup. This *mysteriously* triggered his cellphone to ring, as his boss asked him why the website was down. For those that don't know, this would cause a solaris box to drop to bootprom.
My first job in an IT room was a short one. Was asked to come in over the weekend and aid the head IT hotshot. Guess he never checked his email cause he never showwed and left me w/ the owner of the company (Very mad owner). Owner walked me to the tank where the servers where kept and was telling me how i would be building new workstations. He then showed me the "bench" that i would be working from w/ 20+ copies of Windows and Tons of hardware laying about. I asked if there was a lock policy on un-used hardware and software and he proceeded to show me a 10x10 closet w/ a 3" solid metal door w/ passcode lock. It was epmty of course since the IT hotshot did not care. Spent couple hours putting things away before getting to work on my first system. All the cases i was to work on where being refitted w/ new gear CPU, Ram and Hrd Drv. Open 1st case and had 3 Hrd Drv Ducttapped together and placed inside the drive bay and then taped inside the bay. Grabbed a Digi cam that found in the lock room and took couple shots, then grabbed the owner and asked if this was the companies idea of IT work. Owner was shocked and was even more shocked when i opend several more cases to find more tape and random part shoved in cases. I didnt go back on Sunday. Called them on Monday and asked them to mail my check. There was no way i could clean up after a person like that or even let them call the shots and make poor choices like that.
At Denny's
A few years ago I used to get together with some mates every weekend for a little LAN party. The small apartment we met in grew too small for the number of participants, however fortunately one of the newcomers lived in the apartment below, so some CAT 5 was run out on to the balcony, and down to the next floor. Instant room for expansion on the LAN parties.
The cable remained there for many months - rain, hail and shine until this guy moved to a house directly across the street. There was talk of running the CAT 5 across the street alongside the power lines, however it never happened.
On the work front, once I visited a client complaining of slow network performance. Following the cables, I popped my head up into the false ceiling to find a roll of 100m of 10B2 cabling.
RedFive jedi_knight111@hotmail.com
My former dial-up ISP of many years ran the entire shop (authentication, proxy, web, email, billing, etc) on a single Sun SPARCstation 5. IIRC, their website bragged for many years that said machine featured a pair of 2 GB drives!
Baffling, but true.
I've seen people try to use MS Exchange in place of a mail server.
Hey, you did ask.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
Well now, remember I said it had no BIOS? What it *does* have is an octal debugger, similar to DEBUG in MS-DOS, called ODT. This is actually built into the microcode of the CPU; the CPU requires a console serial port to be present to even POST. If it's not there, a little LED lights on the edge of the CPU board and the machine will never come out of halt. So, at worst, all you need to do is hit <BREAK> type in the boot loader code on the terminal, and the machine will boot. Right?
Right. But that's a pain in the gluteus maximus, because it means typing in a load of stuff like and so on for a few dozen lines. There must be an easier way. What, like burn them into an EPROM? Well yes, but I don't have an EPROM burner. What I *do* have, though, is a VT-510 terminal, which allows you to program key sequences into the function keys. So, what I do now is power up the terminal and the PDP11, press HALT and then RESET on the front panel, hit a key sequence on the terminal, drop back into RUN once the disk seeks (controller is ready) and it's booted.
Yes, I'm buying an EPROM blower off eBay...
I recall installing some additional power outlets in one building in the tropics and worrying about violating code by doing it myself. The building was wired with 8-gauge multiple strand copper wire (for 110VAC 60Hz) with one hot, one neutral and (ostensibly) one ground. Once I got a look under the faceplate, however, I realized that code wouldn't be a problem.
The wiring method was bizarre: at each terminal (screw) the wire was stripped of insulation by tearing for about 2 inches (~5 cm), wrapped around the screw and then continued on its way to the next. It went like that for all the outlets we examined.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
I consulted for a smallish private girls high school since the time they barely had two networks of Mac SE's. One set of 4 for the faculty and 6 for the computer lab, which had a nice raised floor, and someone had an office in there that was huge.
I started deploying PCs and servers for the administrators. Eventually things grew large enough that we needed a central place for a server room, so we picked a store room in the basement. It was within 100m of all the telco closets in other buildings, so it seemed like a good place. We could hit the rest of the campus without having to buy routers, which were quite expensive in the early 90's. Switches were barely even talked about back then.
Only one problem though. The store room was adjacent to the boiler room. A lot of heating ducts ran above our server room, and would cause the computers to overheat. The only solution for a couple of years was to cut the heat to certain parts of the school. The girls froze their butts off (well, as much as one freezes in Southern California).
Eventually, the person hogging the office in the room with the raised floor left the school. I swooped and made it into our new "data center" (lol it had 2 racks...data center, but calling it that made it seem important and garnered a lot of support which was needed because a couple of other factions had plans for that space too). We pulled a bunch of cat 5e from the old server room to the new "data center" and installed a switch in the old room. That way we kept the hub count within spec.
cat
I once visited a company that used Microsoft Windows on every computer in the company!
I know it's hard to believe it but i saw it with my own eyes!
A little over a year ago I had the first ever chance to do something right with regard to IT infrastructure. Prior to that we were based in an office where our 'server room' was a closet. The room in which the closet was located had power issues, bad air conditioning, etc. and we'd regularly have issues with heat or power. When they looked at moving us I jumped at the chance. Got a 600 square foot room with fully independent 480V power service. That power service includes an APC Symmetra with a nominal 15 minute run time. That's backed up by a 125kW natural gas fired generator. We also extend a tendril out to the MDF in the building to power our ISP's routing gear there. The power system is regularly exercised and tested. Air conditioning is provided by two independent systems and the room is kept at 65F. There are forty servers in the room. One day we decided to see what would happen if there was air conditioning failure. In the space of a half hour the temp in the room went from 65F to 85F. So we know we got the cooling system specs right. Now I'll explain the flaw in our whole system. We depend on someone else for DNS resolution. When they go down, we go down. They're finally seeing the light and putting DNS on somewhat more robust boxes and power systems. Just goes to show, you can plan for everything but you'll never find it all.
Thats it, just qmail.
It doesn't get much more bizarre and hack-ish than that.
In the free world the media isn't government run; the government is media run.
Security by Insanity.
Proud owner of BOT2K3 [ bot2k3.net ]
The all-time greatest hack (if it's true...)
m agic
http://www.google.com/search?q=magic+switch+more+
No sig today...
I worked at an insurance rating company during the DOS/Windows 3.1 days. The in house language we used required that we compile with the in house compiler. This compiler required that we have 654K of the 640k base memory free. When a new guy went in and modified the compiler to work in 320k, the VP/Lead "Developer"/Original developer of the compiler, found out, he made the new guy put the code back how he found it. Ahh... Those three weeks that we only needed 320k of free base memory were bliss....
Working in a fairly large software company (the technology will probably give it away but I still dont plan to name names) our department had our own private kitchen and espresso machine (because the site canteen was heavily over priced)
We had an honour system for payment - an old desktop PC with a card reader. You swiped your ID badge through a card reader. All this did was extract the card ID string and send it through a shell script to a mysql database which then deducted from your balance the cost of a coffee - hand cash to the secretaries to top up the balance (I'll admit on average most people were in negative balance though every now and then the worst offenders had their balance details mailed to the whole department to shame them into paying up)
The actual purpose of the card reader PC? It was the DHCP server for the (still in use at the time - 2002) token ring network.
$_="Slashdotter";$syn="OTT";s;..;;;sub _{print shift||$_};s!ash!Perl !;s=$syn=ack=i;tr+LLEd+BLAH+;_"Just Another ";_
I worked as a contractor at a major manufacturer of snack foods in Plano, where all the permies were semi-retarded and used new PCs running Windows 3.1! Nevertheless I was further shocked when during the team-building exercise we draped each other with (thankfull unused) toilet paper. It was the second place I was fired from (the other was also near Dallas, the home of Buckwheat).
Since some people seem to be sharing horror stories, I'll share mine. Kind of tame...
This small company doesn't have an IT department, they have a person who finds time in her day to do IT -- maybe. I occasionally get to come in and try to clean up the mess.
This is a small business. By "Small", I mean "Roughly ten, and no more than twenty people in the company." By that, I mean that there cannot be more than about ten workstations, all running Windows 2000 or above. At least they aren't still on Win98 and Win95...
They have a server -- a fileserver, on which all their data is stored, which also runs the FileMaker server, and is (theoretically) backed up nightly. So far, so good, even if it is done with Microsoft Backup.
So, not good, but not horrible.
Here's the problem: They hire some local IT company, at a rate of some $75/hour/tech for two techs, to handle just about anything that goes wrong, except on rare occasions where I convince them I can do it quicker and cheaper. These guys have set them up with a fucking NT domain!
Would someone please explain to me why the fuck a company that small needs an NT domain?
I mean, they aren't swapping computers -- EVER. Theoretically, everything's backed up, but they never test that -- everyone has their POP3 email downloaded locally to Outlook, which is going to bite them in the ass someday. Everyone has access to everyone else's files on the server, to make sharing stuff easier. The office is small enough, and has no wireless to speak of, so it's not as if anyone's going to be snooping around their network.
I just don't see why their needs couldn't be met with simple Windows filesharing, or a Samba server.
Now, why not have an NT domain, especially if they've already paid for the "Server Edition" or whatever? Well, it costs somewhere between $200 and $1000 of tech time to set up a new computer to operate with "the network" -- which basically means, install printer drivers, get it onto the domain, map a couple of drives, and move "My Documents" to the fileserver. I do not know how to admin an NT domain, so I cannot do this myself.
This also means, no reinstalls, really, because a reinstall ends up costing almost as much as a new computer.
And, of course, the techs refuse to teach any of us how to admin our own fucking network, because if they did that, they'd be out of a job. That is why it's a problem. My 15-year-old brother can work dirt-cheap and setup Windows filesharing, and even Samba servers, all day long, but neither of us knows what to do with an NT domain -- but these techs have convinced them that it's somehow "more secure".
That's the one thing that jumps out of me. I could make a laundry list of other offenders -- just about every machine there is dog-slow (likely spyware), almost everytime I sit down at someone's computer there's some 30 critical updates waiting, they've been known to do things like wipe XP Pro off a laptop and put 2K Pro on in order to have something "more familiar" (they've since learned from that mistake)...
I remember offering to setup a VPN for them, a simple affair with a Linux box and OpenVPN, figuring I'd just route things like Samba and FileMaker connections. And it worked, flawlessly, except I couldn't get on "the network", because I couldn't make the NT domain go across the VPN. And so, within a day or so of my attepmt, they called in The Techs to set them up a VPN (since they'd heard from me that VPNs are a good thing), and now they use Remote Desktop and call it a VPN. True, they can do what I was promising -- they can access their email, files, and FileMaker database remotely -- but it's also damned inefficient, and it runs over a VPN because RDP doesn't do crypto.
Anyway, bit of a rant, but if there's a moral, it's this: If you can't afford a full IT person, you certainly can't afford freelance MSCE dicks.
Don't thank God, thank a doctor!
Some years ago ( 2000-ish), I worked in the R & D department of a French software company. We were with about 20 engineers, and the company was investing its very last cash in development of the network management tool we had been working on for already 2 years. ( BTW: It was a major success: just in time, we were acquired by a major US software editor, no, thank Gawd not Microsoft ). Every evening, there would be a build of that day's version of the tool. It always involved the guy responsible for the build going downstairs to the server room; he would come up after a few minutes and run the build. After a few weeks, I grew so curious that I peeked into the server room while he was doing his trick: he was booting an old PC, built from bits of other discarded PC's, from a diskette. The diskette was in a diskette drive that dangled out of the PC, held back by the yellow-and-red electric wires. When I asked the guy about the "why" of this setup, he explained that our salaries were eating up the last of R & D funding, so he couldnot ask for a decent build server. Incredibly, the bits-and-pieces-PC held out until the acquisition by the Yanks. The day they arrived, however, we had to hide our "build server". They never knew...
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
Here in Trondheim, Norway one of the weirdest IT-setups I've heard of comes from one of the largest high schools (let's keep them anonymous) in Norway.
Four years ago (2003) The IT-department were to re-wire one of the older buildings (that had been refurbished in early 2000) and to get the job done they had to remove old cables from the building to give space to new ones. In the process of this they decided to make sure there was no devices still "live" on the LAN in the building. After shutting down all networked computers, printers, etc known to the techies they were a bit puzzled to find one switchport still alive, besides the uplink. To resolve the matter they followed the cable from the switch, down a hallway and.. straight into the far end brick wall.
There was no doors in the wall, and when they went around to look for the cable exiting on the other side they found no trace of it. After a little investigation back at their office they found that the wall had been put up during the refurbishment more than 2 years earlier. They decided just to unplug the cable and proceed with their network re-wireing.
Two weeks later the students and the employees returned to their building for a new term. They had only been there for a few hours when IT Support started getting calls about printing not working. A techie was dispatched to the building but could find no printer errors. The techie asked one of the employees to show him what they were trying to do when the error occured. The employee quickly pulled up a file and pushed print. No response.
One week later and still no printing in the building the team that had been doing the network re-wireing was asked to resolve the matter. They unplugged all ports in the local switch and tried to put them back in one by one printing one page each time. Still no pages showed up at the printers. Then one of the techies decided to put the last cable in, it was left over from before the re-wireing. It was the cable that went into the wall. Right then, after putting the cable back in the printers all started spewing out documents queued up for printing. Confused by this the techie went back to the office and asked his boss to ask the janitor about the cable in the wall. The school janitor told them that he had no idea, so they had to go pull the blueprints of the building up for reviewing. After looking at the blueprints it turned out that the wall had been put up to mask a door into a part of the building that had been removed. On the side of that door there was another door.
Armed with a sledge hammer the janitor and two techies returned to the building and smashed a hole in the wall at the point the prints said there was a door on the other side. Behind the wall they found a door, and inside was a little room with an old networked computer on a small table by a wall labeled "printserver", happily humming along. The room was full of dust and very hot. Within a few days the edges of the banged up wall was fixed, the dust removed and the door replaced with a new one. The printserver inside were left to do it's task.
In the end the morale of this story must be that when refurbishing old buildings one should always consult IT about the network before -- not after the job is done. But we all know that, ofcourse.
"-Who said sit down?!"
-- S. Ballmer @ MSDC 2003.
Not quite as crazy or funny as some of these but in my younger and not so professional days me and my faithful right hand man did the following (the we can remember) :
:-)
1. We had Dell rack mountable servers but a server cabinet without the depth to properly rack them, we fitted them anyway but the rack (and of course the server) was overhanging out of the back of the cabinet by 4 inches or so. The weight meant this was likely to collapse at any time, our genius idea was to use cable ties (daisy chained together to give us the required length) to tie the over hanging rack to the holes in the cabinet. 3 years laters there still holding strong
2. Our old office "server room" was just a cupboard with computers crammed in it, you could access the front of the boxes but nothing else. We didn't have aircon so the boxes kept crashing, the solution was to remove ceiling tiles and allow the air grate to the outside to cool the cupboard, unsurprisngly it didn't work.
3. We lost our keys to the cupboard when we moving office so we used a screwdriver to break the door open
4. We were working a weekend (software rollout) and decided to play football with an old telephone handset as well as playing frisbee with the software install CD's
5. On the topic of the CD frisbee's, we had a second floor storeroom at our old offices that we used to play frisbee in for hours on end whilst claiming to be working.
I think thats all for now.
Running a system to monitor weather & pollution stations with commodore 64's hooked to radios. It worked for ten years.
Recently I did a short stint for a household name UK company in their reporting department. They needed to report accurately on the performance of their call centres which were run on a number of Aspect ACDs, the call data from these ACDs were dumped into some kind of data warehouse.
From their the department needed to create a reliable and accurate reporting infrastructure to deliver key business data throughout the company.
What they had was a "primary layer" using Crystal Enterprise to create a number of huge spreadsheets containing the days data in large Excel files. The "secondary layer" were a Excel spreadsheets filled with VBA code to read the files created by Crystal process them a bit and output them as new files elsewhere. The "tertiary layer" was another lot of Excel spreadsheets filled with more VBA code to read the secondary layer and add drop down boxes and graphs etc to format the data. In some cases there was a "quatenary layer" which read the tertiary layer and did some more processing.
Needless to say all the VBA in the Excel files was mainly recorded macros with no attempt to check whether any of the various files it relied on were present or showing the correct data. All the files were scattered willy nilly around the network and all of them had to be ran manually in the correct order every day.
Frighteningly they had a huge backlog of further Excel files to write to do yet more processing and they were rebooting Crystal on a daily basis. I left after a couple of weeks because I didn't want to be around when the whole thing collapsed in a giant mess around their ears.
This reportedly happened at our corporate datacenter.
Our then-CEO liked to appear on-stage and in the company's promo videos (he was quite good at that, mind you). One day, the film-crew was filming him in the "Class A datacenter" (it reportedly had these piezo-electric windows that you could darken with a remote-control).
Anyway, the heat of the lamps somehow tripped the fire-alarms and the sprinklers went off.
The water caused the fuses to trip, which in turn reportedly put the UPSs into force (causing more short-circuits, because of the water).
I don't know how that worked.
After the dust had settled, they needed several days of non-stop work to get things into shape again.
Windows 2000 - from the guys who brought us edlin
I believe you are referring to "Bedlam DL3".
Until quite recently I worked at an Aussie telco, not one of the massive ones but in the top 5. I noticed the guy at the desk next to me had a mobile phone plugged into a charger, stashed on the floor under his desk, 24/7. I asked him about it once, and he darkly warned me never to unplug it, answer it if it rang, or otherwise touch it.
I later discovered its purpose; basically it all came to do with "legacy" MMS messages, which are MMS to phones which don't have MMS capabilities (SMS only). The network instead sent the message to the customer as an email (SMTP, which is what MMS uses anyway), with the sender address of a certain "special" service. Apparently, due to the way the providers' network had been designed, if the GSM service it believed these messages were "from" was not "alive" on the HLR, it would refuse to send these thousands of legacy messages to customers.
I always wondered what would happen if one of the cleaners came through afterhours and decided to help himself to a free phone...
Many years ago I was called on a Friday lunchtime by a very scared IT manager.
The computer that was running their organisation had been struck by lightning.
They offered me a lot of money to get the system up and running by monday 09.00.
Naturally I accepted the challenge and made my way to the site, it was unbelievable.
The lightning hadn't actual struck the computer, it had hit a power line 100 yards away, but the damage was incredible.
I thought he was joking when he said it had melted.
Well over the course of the weekend I managed to either build, repair, or source replacements for everything except the memory. It was now about 03.00 monday and I hadn't slept, eaten, or had a beer since about 12.00 Friday.
Fed up by now I hacked up a little serial interface and connected it to one of the tape drives that had survived the blast, man those things were tough, then created a little board on vero that took a memory address and serialised it. This was hacked into the system and I punched in the boot code.
10 minutes later, a monitor sprang to life.
Well when I say sprang, it more like oozed to life.
One character per second, you had to hold a key down until it appeared on screen then release it, but it was working.
Amazingly all their records had survived as well! ( found out later that all the tape drives were run off a seperate mains feed with a surge supressor, at least one engineer must have known what he was doing.)
They paid me off and ran like that for two days before they could get a replacement ram board.
They also figured that server backups were probably a good idea, since they routinely handled millions of pounds of transactions per day in that one office alone.
And since they were accountants, they naturally picked the cheapest backup solution they could dig up, which was a 40-dollar backup box that used VHS video cassettes, underneath a beancounter's desk, right by his foot. I shit you not: every few weeks, it would occur to him that a backup hadn't been done in a while, so he'd shove the VHS cassette into the backup box with his foot, then nudge the start button with his foot, and return to counting beans. The cassette would pop out when it was finished, and that was proof positive for them of a job done properly. They never even bought a second VHS cassette. Amazingly, the thing never stretched to snapping point, but it was undoubtedly unusable for restores (it never occurred to them to do test restores), making it genuinely much, much worse than useless.
At the office on the other side of town was the accounts department for another division. They also used VHS backups, but felt that doing backups was a bit beneath them really, so instead they had the office cleaner shove the VHS cassette into the 40-dollar backup box next to the office door every night on her way out. One night she was home with the flu, and hadn't left instructions for her replacement to do the "backup". Sure enough, the server crashed that night, and the stale backup wouldn't restore. The poor cleaner was immediately fired, but not the asshats who delegated mission-critical IT chores to a cleaner, on dimestore reject equipment.
I felt duty-bound to tell these fucking morons that they were really making a false savings on backup equipment, and needed to buy real backup gear, with someone trained to monitor the state of the scheduled nightly backups and do scheduled test-restores. This company was pulling in 13 billion US dollars in revenue a year, so 1500 dollars for an internal tape drive and a copy of Cheyenne to protect hundreds of millions of dollars worth of data sounded like a pretty unbeatable deal to me.
Not to them though. "You IT people", quoth a senior beancounter, shaking his head, when I took the purchase requisition to his desk for signature. "It's always more money for the latest damn thing, isn't it."
Cheapest of all would have been for them to simply use the central Unix servers, which were run properly with tested and reliable disaster recovery by experienced sysadmins. I tried explaining that there'd be no change to their DOS PCs, and they'd still have the same F: G: and H: drives, with no visible change to their working environment. I even offered to pay for the new client software. They'd save money, and get vastly better care of precious data.
The reply: "Heh heh heh! And then next year there'll be some reason why we all have to get rid of 1-2-3. And after that there'll be some reason why we have to get rid of DOS. No thanks! Heh heh heh! You guys never quit, do you!"
Hmmmm.
Seen the whole machines-in-closets. A couple BBSes (QuartzBBS at Rutgers and later ISCABBS at UIowa) ran out of closets. I worked for a company whose mailserver was a 386 running BSD, in the coat closet.
Also worked with a company that was running Advanced PICK (bizarre) atop SCO OpenSewer (bizarre) on an old beige Dell PowerEdge... sitting on the counter next to the sink in the men's room, because they had nowhere else to put it.
Village idiot in some extremely smart villages.
http://www.42inc.com/~estephen/humor/support.txt
It's really old, but it makes me smile time and time again.
WYSIWIG, but what you see might not be what you need
Shrug and reboot....
A house divided against itself cannot stand.
Stick it to the man. Reminds me of the time I was at UBC working in the "server room" and we had water running out of the ceiling onto the server rack. Mark went and got some plastic shield and thus a splash guard deflector was instantly fabricated. The Netware servers were saved. Turns out the building's third floor had a lab where they left a drip running but the bottle slipped and plugged the drain overnight. I think a splash guard is a good idea now for any mission critical rack. It is cheap as chips and simple. -- Ciao, Dave
There was that elderlish guy at a large german university where I used to work whose network was no longer working. We did the standard procedure (with 3000 machines spread over a city the size of Munich, you don't jump to the desaster area right away): Check if routers worked, pinged some machines in the building he was in, looked at the logs... to no avail: I had to go over to him (3km, in deepest winter, at -15C, with rush hour traffic jams that rendered cars basically unusable)... ... When I arrived, I ran some diagnostics on the machine, but it seemed there really was no network connected. I checked the cable box in the wall and the one on his machine. We used to deploy 10m-cables, because some genius bought them in bulk, and the PHB figured the were "cheaper" because of this bulk-buying - even if the distance between wall and machine was less than a meter.
Well, the cable was connected both to the wall, and to the computer. Unfortunately, it was clearly CUT right in the middle. When I questioned that elder superhero, he stated that he found out years ago that you could use a copper cable as an TV antenna, and he received a memo the day before that WiFi was now available in all university buildings - so he decided to cut the Cat5 to serve as a WiFi antenna....
Screw the FSM - Real geeks believe in the Invisible Pink Unicorn
A while ago I ran an Amiga software development company. Our designer (Mark) had an Amiga 4000 with various external SCSI devices running off a notoriously unreliable Commodore A4091 SCSI card.
I went to his desk once trying to access a file on his external drives, and I kept getting disk errors. I called him over, and he said "Oh! That disk won't work unless you open up the system clock and resize it to this kind of size, and put it on the screen here". He opened the old analogue-face clock program that came with the amiga, resized it to about 200 pixels square, and stuck it in the top right of his screen.
I stood there smiling. He was, after all, a designer.
The file opened fine though after he did that.
I did some messing around on his machine afterwards. I was convinced there was some kind of obscure problem that we were missing - incorrect termination or bad cables maybe. I put the clock incident down to coincidence.
I could find nothing else wrong - but I still couldn't access the disk. So, I opened the clock application. I tried it on one side of the screen. File would not open. Moved it to the top right corner. The file opened. I did this about ten times as I couldn't believe the results myself. Every time I had the clock in the top right corner, the external SCSI disk behaved itself. I tried different applications, none of them worked in the same way - it had to be the clock.
I was completely spooked by the whole thing, and decided this was something sent by the Gods of SCSI to taunt me. The logical side of my mind believes that it is probably some obscure DMA issue, the rest of my mind believes the machine was possessed.
The thing I was never able to figure out was how Mark discovered the SCSI-healing properties of the Magic Amiga Clock and why he felt it was perfectly normal behaviour for his machine!
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
1) The computer room floor built with a 4 foot void rather than 4 inches because the builder read the plans wrong. Mid you, there was room for a lot of kit in this 'split level' computer room.
2) The Netware 3.x file server which was a Toshiba T3200 plasma screen laptop locked inside a filing cabinet (a very secure solution on a military base). While I was working on it, a telephone began to ring in the next drawer up. I mentioned this to someone as nobody seemed to have heard it and the reply was "Oh, we don't answer that one"
3) The Olivetti M24 (AT&T 6300) that lived in a milking shed in the middle of a dusty field that eventually died and had to have a 2-3 inch layer of 'field' vacuumed out.
4) The computer room built with the existing radiators walled in but not turned off - took ages for the aircon guys to figure out why the room never cooled to the calculated temp.
5) The installation test of a new halon system (with a cylinder of CO2) where the engineers had not properly screwed the nozzle onto the 'j' pipe in the centre of the room. When the system was fired, the nozzle shot through the false ceiling, the gas followed it and the pressure blew down all the ceiling ties - the computer room looked like a scene from Die Hard.
6) The school network that comprised 5+ 'backbones' of 10Base2, each with around 20-30 D-Link *hubs* wired directly to cat5 outlets. Netware servers strategically placed round the building acted as repeaters with 2-3 NICs in each. We also found some Cat4 cable buried directly into the walls (no trunking).
7) 140m of Ethernet coax buried below a school field to link two buildings.
8) The over-length Token Ring network that included specially designed and developed repeaters that had to be 'tuned' using a screwdriver to adjust variable resistors to get the timing 'just right' so that the whole thing worked.
I have to add that I was *always* the support person brought in to sort things out - not the one creating the mess.
AT&ROFLMAO
I was working for a small newspaper as a Photoshop monkey. Being semi-competent in Linux, I was pleased when the company switched to OSX from OS9 since I would be able to play with something resembling a modern operating system. The IT staff was still stuck in the OS9 mentality, so I was the one that fixed computers when they broke.
.filename filename, something I could have done in seconds if I'd had access to the company network but which took thirty minutes over the phone with a panicked, menopausal woman on a deadline.
Keep in mind that the photo department doesn't generally employ technically proficient people, they're usually of a more artsy background. My former employer, surprisingly, had promoted two technically proficient photographers to photo editor positions, but the third, incompetent one, was quite a problem. She called me one night in a panic to solve a problem we commonly had with photos that came down from the AP wire. She was the only one on duty at the time
For our paper, the AP photos came in via satellite and were temporarily stored on one of two aging OS9 G4s. The database software, which was used for layout and image storage, would grab the images, extract caption data, and store them on what I believe was a Sun server somewhere. After the designers placed the image on a page, it would go to the engravers, which would Photoshop the image so that it appeared correctly when printed. However, the machines the engravers worked on were OSX. Sometimes, for no good reason, a photo that we had to have came down the wire with a filename that had a leading period. OS9 machines, the ones that caught the images coming from the wire, had absolutely no problem with this. However, whenever the engraver saved the file out to his OSX machine, nothing would appear in the target directory since a leading period in OSX and other Unices hides the file.
So, I'd just gotten off the bus after a long night at work when the editor calls and wants me to come back in to fix this trivial problem. It's 11pm and I don't want to ride the bus all the way back across town without being paid for it, so over the phone I coach a woman who has never heard the word terminal used in this context to "Click on the hard drive icon, the one at the top right of your screen. No, you have to minimize such and such first, you minimize by clicking on this, it's located here... Are you seeing the icons on your desktop? No? What do you see? Hit Apple+M. It is the button with the weird symbol between CTRL and ALT. OK, now go to Applications. Click on it. Go to Utilities. Click on it. Go to Terminal. Click on it."
After that I just gave up on instructing her on what to do and moved down a layer of abstraction to tell her exactly what keys to press. The fix was trivial, you had to go to the terminal and do a mv
Olden days, I was doing microprocessor coding for robotics. We used an assembler/linker package running on a couple of 16-bit minicomputers. One machine had enough RAM to do the assembly, the other had a hard disk that let us do the link of the object code to produce an image we could blow into an EPROM. Neither machine could do both parts.
Problem was the two machines were in adjacent rooms and networking was nonexistent. We'd run the assembly process, produce a paper tape of the object code, carry it to the other machine and then feed it into its paper-tape reader to do the link. This took a lot of time which delayed development.
I noticed the assembly process ran quite quickly but the linker machine was noticeably slower, and the machines weren't actually that far apart in reality. I got a couple of empty paper-tape spools and clamped them to the doorframes to act as guides, then I took an assembly tape being punched by the first machine out its room, around the doorframe, into the next room and fed it into the other machine's reader. A garbage bin acted as buffer storage for overflow tape, and it worked. I had to baby-sit it in case of jams or tangles but it cut the total assembly/link processing time by at least an hour.
Once upon a time (+-1989), we had a set of some 50 Apollo workstations linked up via a Token Ring network. Not only did that ring have a habit of being physically broken every so often, the worst part was that there was no file server. Everybody stored his own files on his own machine. Project accounts were housed on the machine of the project owner. Nice and orderly, huh?
Well... except for the fact that there were people who didn't have a personal machine. Their data was initially housed on the machine of someone they cooperated most with. When disks filled up, new people without a machine would end up on whatever disk happened to have spare capacity. Then we (or rather "they", as I was there but not part of the IT gang) found that the amount of data people store over time outgrows the size of their disks, especially if you have shared project accounts. So ever so often, accounts had to be moved around. And sure enough, before you knew it the owner of the machine to which some high profile project had just been moved would complain that his box was overloaded doing other people's I/O. And just when that had been sorted out, there typically would be a reorganisation involving people switching offices or desks. Sometimes the machine followed its owner (not all were equally fast and some of them had black&white displays that nobody wanted to inherit), but most often IT would object to moving the boxes. By now the physical link between the data and its owner is totally gone. In the end, most people didn't have a clue what machines their files were stored on.
And now the fun really starts. We relied a lot on students and interns. In those days, if a student had seen a computer before, either it was a Commodore 68 or an early standalone PC. They didn't have a clue what the network was used for, so whenever they were done and went home, they'd physically switch off the machine they had been using. To make matters worse, even just keeping employee data storage away from the student machines was not an option, because there were not nearly enough student machines around. Typically, students would use the machine of an employee who happened to be out of the office on that day or during that night. Oh, what sweet memories... Not!
Not to mention the backup problems... Nor the fact that we also had a parallel experimental ethernet network with non-Apollo machines, of couse also without a proper file server. After a while, some data was being stored there instead. Now where the ... did I save that f... file last week???
Linux user since early January 1992.
Can we please put the 'itsatrap' tag to good use folks?
v4sw6PU$hw6ln6pr4F$ck 4/6$ma3+6u7LNS$w2m4l7U$i2e4+7en6a2X h
Central air conditioning has an engineering limitation: at extremely low outside temperatures, it fails. Most people are not affected by this...
I worked at a company with a windowless server room buried in the middle of the IT department. On the coldest day of the year, the AC quit. We had to open all the doors, and use fans to circulate air out of the server room. Needless to say, the staff -- especially those nearest the doors -- complained about this endlessly.
I told the CIO, an old friend of mine, that we should run a single 12" duct through the drop ceiling from an outside wall to the server room, and put a fan in the room to pull in the subfreezing outdoor air next time this happened. Of course, when the AC came back on, he forgot all about this... until the next year, when the temperature plunged and the AC failed again. (He gave me a look that said, "Don't say it... I know you Told Me So...).
Within two days of that second failure, workers were running a flexible duct through the ceiling.
I work for an outsourcing support company. The ISP/Cableco who was our client shall remain nameless, but here's what happened: They transitioned to a completely new billing system and whoever sold it to them had assured them the record would transition seamlessly from the old system, but they didn't. A percentage (forgot how many) of customer records had been goofed up in the transition. To fix this the home office needed copies of certain screens (about five different ones for each account) from the old billing system, so they knew what values to type in by hand on the new system.
They didn't have the time at the office to look up all these records themselves. Their solution was for us, the company providing customer support, to (I'm not making this up) take screen captures of the needed screens on the old billing system, print them out, and fax them to the home office halfway across the country. This faxing had to be done after normal business hours as our machine had to be free for other use during the day. Anyway, to give you an idea how much paper this was wasting, we were measuring usage by the ream, not the sheet. All of this paper was being fed through an autofeed on the fax/copier that would occasionally jam up, too. I volunteered for this assignment for the overtime.
Anyway, things were moving slowly and bugging from the fax/copier hiccups (this was a 30,000 page/month duty cycle copier that was used at closer to 50,000 pages according to HR) at about 2:00am I came up with a better idea, why are we printing these out and wasting all this paper when we can send them the screen shots themselves? I was noticing the readability was quite poor from some samples we sent back and forth with the other side. So we pasted the screen shots into Word documents, and then we tried to send the enormous Word documents over email, but the company's email server was complaining about the attachment size.
Well a co-worker had a GMail account he had been playing around with (this is when GMail was brand new, so he had gotten an invite for it) so we decided to try it because of the 10MB attachment limit. We could just upload the files and give the other side the log-in info and they could download the attachments at their leisure. Well, that didn't work out either, I think some of the files may have been too big or there was an issue getting the files uploaded, I don't remember anymore. But it didn't work out.
My next idea was to go back to faxing, but paperless. There was one workstation that had a dial-up modem instead of a NIC card, it was normally used to test access numbers for dial-up ISPs clients we had, but we hadn't used it in so long we didn't remember the login name to get into it, the password we were pretty sure of. After trying to guess it for a couple minutes I got the idea of booting the machine using a Knoppix CD and looking in C:\Documents and Settings\ to see what user folders were there (as I'd spot the correct login name amongst them). After we got logged into the machine we used a flash drive to transfer the files from the other machines we had been using to compile the Word screen shot documents. Then we'd open a document and fax it using Windows XP's built-in fax capabilities to the fax machine at the home office. So soon we had an over 50 page Word file printing over fax to the east coast with several more just like it queued up. It was moving slow but seemed to be working. The idea was we could now leave and let the machine fax the rest of the night.
I was excused to go home about 4am. I came in the next day around 1:30pm and found the fax calls had been interrupted around nine that morning. Apparently the home office had called because the same page was repeating over and over again on their side (which they naturally claimed must be an issue from our end). I didn't hear how they finished getting the records transmitted, but I think they went back to paper faxing again.
Now that I think about this, It would have probably been a better idea (if we'd had more than one day to do this) to just to take the huge Word documents and burn them to a CD and then Overnight the CD to the home office.
The best I've seen is in a place I can't talk about very much, however...
The room is a brand-new medium sized conference room with full global video-teleconference capability. The table seats about 20-30 people, and each seat has either one or two computers feeding a single LCD/kb/mouse per seat via an A/B switch. The funny part is that the requirement of how many seats would have 0, 1, or 2 computers grew between concept development and implementation, but they never sourced any kind of rackmount setup or even any place to put that many computers. So in the closet are dozens of PCs (mostly Dells) lying on their sides stacked from the floor nearly to the ceiling, hooked into 2 separate networks that must never be connected together.
I heard a good story about what happened when a computer near the bottom of the stack failed... It took half a day to unwire, unstack, fix, re-stack, re-wire, and then re-wire again because they'd hooked up the wrong computers to some of the conference table seats. I'm sure we "saved" a bunch of money going with all those Dells instead of a rack of blades or whatever, and the computers by themselves fit the requirements, but maintaining them is quite a chore.
Pour some olive oil in the U-Bend, it takes ages to evaporate, stops the smell and doesn't clug the pipe should it be needed.
Hmm. Sounds like our server room. In the summer we throw the windows open and have one fan blowing air in through one window, and another blowing air out through another. There's a thermometer attached to the inside of the window in the door so that anybody walking past can see the temperature and take action if it gets too hot. It's a terrible bodge, but the old building can't cope with more air-con than it already has, apparently.
Our "server room" is absolutely tiny. We have two full 42U racks in there but to get new stuff in them you have to take them to bits first, they do not fit through the door. The only access to them is from the front, you cannot get around the sides or back. Getting servers in is possible only because I have long arms. Getting them out requires twisting the rack rails until the clips pop and then putting your arm under the rack enclosure (nice and safe) to get them. We have UPS for everything but its never been setup and someone chucked out all the software and documentation.
Most of one of the two racks is full of desktop machines stacked on top of each other using 15U of space for a job which could be easily done by one new 1u box.
Normally the rails need to be bent back into shape with a hammer and pliers, sometimes we just have to throw them away. Heaven forbid you have to swap a cable from the back of a rack as you cant unplug them before pulling the chassis forward. All the cables are left unscrewed because of this, except for network cables which have to be ripped out. This often damages the ports beyond repair after a while. We do not use patch panels.
All of our core operations servers are out of warranty. All of the RAID arrays are degraded and cannot be rebuilt from the OS. None of the management tools work.
It was over a year after I arrived that I noticed our Exchange server (100+ mailboxes) & Windows 2003 was running on a system with 256MB RAM. Regularly swapping out 4GB on to the RAID5 array.
90% of our servers are not in the server room. Mostly they are piled into corners in little hills starting with the biggest and the bottom. This is particularly dangerous as our Building leaks and it isnt unusual to see water streaming down the walls after heavy rain. Often parts of the roof collapse because they are so water logged, often without warning. It has been known for me to be racing to unplug a machine and move it underneath some roof which is about to fall in.
Our network core has a variety of colour schemes in use, all by different people, all different schemes. The result is pretty but useless in any practical way. Many of the cables are no longer used but knotted in with our cable so tightly the only way to get them out is use a knife.
When I arrived we had a firewall in place but it was not configured and none of our public machines were covered. You could browse the power management interface and reset all of the machines.
Nobody knows how anything works except for me, and even then I often forget things. I have no idea how the telephone system works so most people have someone elses name as their caller Id.
I suggested remote backup but it was veto-ed as being too expensive so I just do it to my home for now. My ISP doesn't seem to mind the usage which is handy.
An ancient laptop we use for connecting to other organisations VPNs regularly gets infested with virii and has to be rebuilt. All of the vpn clients we use are incompatible with each other.
I opened the Help Desk at 7 AM every morning for a company with sites in several states. Previously, if a server or a link was down, it would often go undiagnosed until 8:30 or 9 AM because not enough people would call the Help Desk to complain until then.
I wrote a simple batch file that would ping every server on the WAN with two packets, dump the results to a text file, and open the file in front of me when I logged in. I would search the file for the word "timed" (Request timed out) and start troubleshooting and notifying by 7:05. The CIO, who came in at 7:15, loved knowing that his network was up (or not) as he walked into the building.
The only problem was, my old buddy the CIO was paying for an expensive network monitoring system, which I ignored completely because it was cumbersome, expensive, and I had never learned to use it. One morning, after I had scrambled resources to track down a server outage reported by my batch file, he insisted that I work with the network admin to track the problem through that system.
Half an hour later, I went to his office. "Dave, I found the details of the failure in the monitoring system: it couldn't ping the Exchange server in Michigan..." which, we both knew, was exactly what the batch file had told me earlier that day.
He never bothered me about that system again...
What are some you've seen?
In a high-security building, running 10 Mbps Ethernet over fiber for fear of radiating information. Another building had simply been built as a Faraday cage.
Awhile back we needed to run a length of telephone cable (probably around 100') in an emergency. Only problem is we didn't have near that length handy, but we had a bunch of shorties.
Damn! nothing to connect those together with.
Hey! A ton of machines in this building have internal modems that they aren't using... lets snag those. The line input and telephone input should work for us.
Voila. Put a plant in front of each modem, and you've got a temporary extension cord =-)
It's drinking the water!
It's going back for more! Ee-heee-heeeee!! This is the greatest invention. You're gonna make a million dollars!!
(Some asshole is even selling one on eBay "as seen on the Simpsons".
Do you or your partner snore? - Visit www.snoring.com.au
Story 1: My first network position was at a Credit Union in Toronto. ARCnet, LAN Manager, network boot disks, the whole frigging shebang, all installed by a consultant, now long gone. One day, the HR department had a major system failure - no one could log on. Hours of troubleshooting revealed that they were on a completely separate Novell network, with a hub, file server all hidden in a closet that no one, I mean no one (except the consultant) knew about. Secure? Yes! Story B: Our air con in the server room was terminal, when I'd walk in to operations early in the morning, there'd be the female computer operators sitting there in their underwear. Downside: wrinkles.
Twelve years ago, I worked in the I.T. department of a small private college in upstate New York (around 4000 students). Our main server room was fairly meticulous and well set-up (we had a perfectionist geek as the main sysadmin at the time). One summer, that building was scheduled for new electrical service to some new science labs in the upper floors of the building. A few hours after the electricians showed up, they came running to our offices and insisted we follow them down to the basement. There, they showed us the wiring to our server room, installed just a couple of years earlier: it was not actually physically connected. The wires had a small gap, and the electricity was simply arcing over. One serious bump of the box would probably move them enough to cut our power.
So their first task was to fix this. They would turn off the power for 30 minutes while we ran all the servers on UPSs, then temporarily reconnect power for awhile to recharge the UPSs, then turn off the power again and work... took all day at this rate.
$nice = $webHosting + $domainNames + $sslCerts
Not so much a setup but a work-around gone wrong.
I worked (briefly) on the computer system for the Iirsh Police Force - 'An Garda Siochana' http://www.garda.ie/ containing all of the 'Bad Guy' data. It was astonishingly slow due to incredibally poor coding and a crap network.
One day a few moitors were returned from a divisional station to the hardware section in HQ with dots of TippEx http://en.wikipedia.org/wiki/Tipp-Ex covering the screen.
Bewildered, the monitors were replaced, but curiosity got the better of one of the hardware guys who decided to go on site a see for himself the Guards in action.
Since the system was so slow, the Sergeant working on Warrants, who would have to jump between the Incident screen - to the Charge screen to the Warrant screen, would place TippEx dots on the screen to indicate where he had to click. The Sergeant would click the dots in a particular order and go for a cup of tea or a cigarette. On his return he would find the cached clicks would have brought him to the correct screen.
The system worked (fine considering the system) until another Sergeant with a completely different responsibility came along, copied his idea, and put his own set of dots.
Soon after, the monitors were filled with dots, and nobody knew which was which.
I worked in a large Data Centre in the mid 80's, with long wide ramp to bring up pallets of paper for the printers. One quiet weekend we drove a car into the room and popped a row of floor tiles out to create an instant inspection pit.
I know some of my former colleagues read /., so maybe they'll spot this (hi guys!).
:)
When I was sysadmin for a certain Boston-based startup, we moved into new digs that didn't include a proper server room. What I wound up with was a tiny office with enough room for a couple of 42U racks. This being our build farm, you might imagine that it got pretty hot in there. It did, and leaving the door open didn't help. Nor did setting up a couple of fans.
I knew I wasn't going to get a real server room, and we weren't going to get a separate A/C system for just this tiny little room. So, I decided to sell my boss on a portable A/C unit. Not one of the window units; one of the freestanding units with a flexible exhaust duct. I didn't want to snake the duct out the door (it was about 1.5' in diameter), and the room wasn't near an outside wall (not that I would have been able to cut through the wall anyhow). So I bought an adapter for the end of the duct that allowed it to replace a ceiling tile. Problem solved? Nope.
The A/C was exhausting nicely into the space above the ceiling tiles, but for some reason it was just not working very efficiently. I climbed up on a chair and looked around, and realized that the walls extended above the tiles all the way up to the true ceiling. Crap. So, I decided to make some after-hours "modifications" involving a drywall saw. Presto, some stealth vents into the surrounding areas. Problem solved? Almost.
The thing about A/C is that it dries the air as it cools it. In the winter, you don't notice too much condensation. In the summer, that damned tank fills up every few hours. And the unit shuts down before the tank overflows, which is good, but it means that the server room has a tendency to get really hot before someone gets in to lug the stupid tank all the way to the other side of the offices to the sink.
So, with an online order for a submersible pump (like for fountains), a couple of float switches, some pvc tubing, and a visit to the home goods store on the first floor, I've got a solution to the condensation problem: A humidifier on the outside of the server room! Yes, I know, I'm a genius. Email me for details about where to send the MacArthur grant.
It actually worked pretty well. Of course, a real server room with a real HVAC system designed by an actual HVAC engineer would have been best. But sometimes you just don't get what you want. In a later job, I did actually get to build out my own server room. Proper A/C was at the top of my list.
A host is a host from coast to coast...
Unless it's down, or slow, or fails to POST!
Anyone who has been in New York City's Pennsylvania Station (Manhattan's train terminal for Amtrak, Long Island Railroad, and New Jersey Transit) has seen the television screens with the arrival and departure times. These look like fairly basic systems (simple white text on a black background). I commute into the city, so I'm there every day. A while back I was waiting for my train when the departure screens suddenly switched to the Windows blue screen of death (complete with error codes, etc.). It was like that the next day as well. I really wished there was a ctl-alt-delete button beneath the departure board.....
$nice = $webHosting + $domainNames + $sslCerts
See iterate(v).
I ran a small network whose file server was a Linux box named "host". The person who set it up had never set up a server before, and all the instructions used host (in italics) to describe how to install it. So he named the machine "host". The problem was that so many configurations had "host" that I never had the time to go around and change everything.
I've found that my posts don't format quite right w/o a sig.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
A fun story was posted here a year or so ago, http://www.techweb.com/wire/story/TWB20010409S0012 , about a server that had been running for awhile without anyone knowing where it was at. When they tried to track it down, they finally discovered it had been entombed withinin a wall of a building.
I work for the Department of Redundancy Department.
10 years ago, the sysadmin of our department left the company (and the country), and somebody else
became responsible for the backups and other IT tasks. After some time, one of the server disks
got broken. This disk contained several years of work, so retrieving the data was very important.
It didn't took long to learn that all the backups were made on a cleaning tape. Restoring from old
tapes was not possible because nobody knew how the previous sysadmin made the backups, and he
was not reachable anymore. Lucky for us, the company of the server was still able to get all data
from the broken disk.
One company I worked for redesigned their data center. The guy in charge wanted to make it a "show place" so he had a glass wall installed at one end. Since the ops consoles were at the other end of the room, it was decreed that nothing over 3 feet high could be placed between the glass on the ops area so as to preserve the view. That blew half the room. He then had all the racks on the other side set up at a 30 degree angle to the floor tiles so visitors could see down the rows. Great, until anyone needed to get at wiring under the raised floor. He later complained when we had to remove the glass doors from the from the front of several cabinets to keep servers from overheating. Of course, this was partially due to the inability to place vent tiles were they needed to be. But it looked good...
I used to have friends who lived two houses away from each other. They wanted to split the cost of just-introduced cable Internet and also play network games together, so they got a 500-foot ethernet cable and laid it just under the sod, connecting their two homes while going around the house in between. They called their creation, aptly enough, "network neighbourhood".
He who lights his taper at mine, receives light without darkening me.
My friend worked for one design company as a SysAdmin. They had all their servers in small closet about a size 1m per 1m. There were some 6 servers and all networking gear. On shelves, one atop another. And absolutely no air conditioning. So we did very simple thing - we drilled some holes at the top of the door (hot air goes up), and two square holes in the bottom. We installed two little fans (those you can very often find in toilets). That did the trick. And looked quite funny, when all guys saw SysAdmin drilling and destroying the door.
I don't know if this contraption still works, he changed a job some time ago...
"an experienced, industrious, ambitious, and often, quite often, picturesque liar" - Mark Twain
A friend of mine who lived about 150 metres (~500 feet) down the road and I linked our computers together for doom deathmatches with varnished copper run through the storm drains.
We got hold of some old solenoids from old traffic light controllers. Using 3 solenoids and lying down on a skateboard and pulling myself through the underground drain pipes we ran a TX,RX and GND wire all the way and were easily able to connect together at 115200 baud. Actually, we used to be able to get 1152900 at night, but during the day we had more luck at half that speed. Never got it working with a laptop at one end, but desktop to desktop it lasted about 4 months before some bad weather put the cable in the drain out of use.
Oh yeah someone said something about power over ethernet? I know someone who was trying to run 12 volts through ethernet for a wireless access point but thelow voltage meant the cable loss was too high. He ended up putting a 50mA fuse at one end and ran 240Vac through the line with the switch mode plugpack at the remote end (it accepted 110 - 240V AC) and worked fine. I believe he had an input voltage of 235Volts and out the other end was getting about 225Volts and it was almost an entire 300 yard box of outdoor shielded cat 5e.
In the early 90's I worked for a software company that developed process control software for "sheet" (steel, paper, etc) manufacturers. On one of my install trips I was taken to the computer room, which was in a room at the top of one of the manufacturing buildings. Shortly after I arrived, there was a lot bang, followed by a fairly large tremor, I looked at my "contact" and he said "That was just a jumbo being dropped onto the cutter spindle, nothing to worry about". A "jumbo" is a large roll of paper, several tons. Half joking I asked if he had many disk failures, he said they didn't, probably because they used VAX's, built like tanks, and ran about as slow. ;)
That was also my first exposure to 10Base5 and "vampire" taps, all I have to say is thank you for XBaseT.
Guys, Granted this is not IT but it's funny nonetheless. In 1997, I lived in St. Petersburg Russia. Great experience. I rented an apartment from a Russian lady with a wart named Svetlana for $110.00/mo. One day, when I was going into the bathroom, I noticed a fine spray of hot water coming from the wall. Inspecting the area, I saw two verticle pipes going up to the apartment above me. They of course had a branch for my apartment. Upon closer inspection, I saw that the hot water pipe had a huge tumor of rust under 50 layers of paint. Besides the general wetness, I saw that the rust had so degraded the pipe that it was allowing a fine mist of hot water to escape. When my Russian friends (17-year-old girls from the neighborhood) came over, I pointed this out to them and they all, in unison said: "Ah, dolzhen zvonit master" (You need to call the master). Seeing lots of rubles being lost in such an activity, I asked them how I would do that. One of them, Natasha, said she would make the call for me. The next day, I woke to a furious knocking on my door. I said "Kto tam?" (Who's there?) and "Mahster" came back. I opened the door to find a very short man, spitting image of Chico Marx, standing there. Behind him was Harpo, just as short, with blond hair. The man entered my apartment and asked me "Gde deerka" (Where's the hole?) I led him to the bathroom and he immediately saw the rust tumor, which he flaked off with his finger. Nodding, he vanished. About ten minutes later, I saw him back at my door, and he did not have a replacement pipe. Instead, he was dragging a long pair of hoses to an oxy-acetelyn torch. He dragged it in my apartment and, without even turning the water off or scraping away the 50 layers of paint, proceeded to weld over the tumor in place! After about 20 minutes, the tumor was replaced with a large keloid of brazing rod and my leak was fixed! Only in Russia!
I have been told that, back in the late 70s or early 80s, when a new courthouse/office building was built in a nearby county, someone got the idea to use the heat generated in the computer room to augment the building's heating system.
As I heard it, during the first winter, the gas company sent inspectors to check the pipelines, test the meters, etc., because they couldn't imagine that a building of that size could use so little gas in the wintertime.
An enterprising group in our IT department could not find an English version of the product manual for a piece of software they were using. So they ran portions of it through some web-based language translator and made their own English version. They did later acquire an English version from the vendor, but I haven't heard how the two compared in quality.
Seriously, Don't take anything I say seriously.
During my high school days I had to spend a week in a IT lab by a real company. I (and 5-6 of my other classmates) were assigned a room with a PMD-85. It was a 8-bit computer a bit similar to old Atari or Amiga. As the wiki page says, it often suffered from overheating. So the solution implemented by the company was: to rip 5x10cm hole in the ROM module cover and ram a hair dryer into the hole (set to "cold" of course).
My home network has a wireless access point - just a plain linksys, nothing fancy. A couple of months ago it stoped responding, but a few pokes with the multimeter determined the fault was in the power supply - confirmed by the distortion in the case near where the regulator had burnt out.
It needed twelve volts - at two amps. I have plenty of spare PSU bricks, but none rated for anywhere near that current. How was I supposed to power it? Computers have 12V internally. The router, a linux-running PC, sits next to the router...
I choped up an old molex-to-sata adaptor to get the female molex connector, spliced it to the APs power cable, poked the frankenstein-lead through an expansion slot in the router... and the AP has been running happily since.
I worked in an office on a Sperry Univac BC7 mini computer. It had a LED panel that displayed certain error messages. I learned how to send messages to the panel. They had air conditioning, but were too cheap to set it to a comfortable level. One day I had a brainstorm while sitting there sweating. I sent "overheat warning" to the panel. I pointed it out to the office manager. He immediately turned the air on.
I was the head net engineer on the island of st thomas (us virgin islands) during the late 90's early 2000. There was not a lot of technical people so I got to do almost everthing by myself and with little support from the states.
I built a couple of ISP's, Govt networks and tons of small business systems. Here are some favorite hacks/stories.
-wireless in a bucket in a cave. We needed to link two communication towers but lacked line of sight. The owner had a small piece of land on peterborg which did have line of sight but was nothing but rocks and jungle. During a test of (breezecom AP's) between towers I had put the AP's in a 5 gallon bucket with some holes in it. The permit for building and power was almost a year away, but damnit I needed that link. So I ended up wiring in a 6v golf cart battery to provide power to 2 AP's and a crossover cable. I found a little cave on the property and put it in there (two big ugly 24db antenna's peeking out)... So now every 3 days or so the internet would go down for half the island, the pager would go off, and I would drag a charged battery down to peterborg and replace the battery. This went on for almost a year. I cant tell you how glad I was to finally see the building get finished and *plug* in the bucket.
-The territorial court system had a *old* system of managing the court docket. A large and heavy Book was used on the first flood to record incoming violations and pending offenses (in pencil!). later in the day it would go to the second floor where they would write stuff regarding logs/cases/dispositions. then at the end of the day it would go to the 3rd floor where they would write down the actual court proceedings into the book....in the morning it would go down to the first floor where they would *photocopy* the book (old skool backups). and start all over again... years later EDS installed a very expensive case managment system that automated this stuff. But since there were few people that were computer literate it hardly got used except to *scan* in the book for online backups.
-UPS's... since they are heavy and expensive to ship to a small caribbean island. We would take dozens of marine batteries a couple of industrial inverters and some chargers and dedicate a small room to house it. Then wire the inverters into the premise wiring and viola. site wide UPS! This is for a ISP's datacenter. Thank goodness the EPA never came within a 1000 miles of my island.
-critters like warm EM equipment.. I had to rebuild a number of servers as either a lizard would crawl into the box and short out the MB with its body (frying the poor guy in the process)... OR the big one which is ants are attracted to the EM fields and would start to congregate around sources of it.. So you would see lots of power supplies with ant nests in it, funny smell burnt electronics and cooked ants.
--jboss
-
See for yourself. No story needed.
http://www.cardope.com/weird/DSC00006.jpg
Just imagine what that thing was hooked up to.
Remember the days of cash paying banners? They paid you for viewing banner ads on your computer. Trouble is you had to keep your mouse moving. We used to install an entire screen full of the things and hook the mouse up to an oscillating fan.
I worked for a company that ordered about 100 compaq desktops and corresponding touchscreen monitors.
The problem is, the bios wouldnt let them boot without a keyboard. (Tested after the purchase of course)
Solution, buy 100 keyboards, open them up, remove the board / all the keys, saw off the parts you dont want, plug the keyboard into the PS2 port so that it recognizes that a keyboad is present.
At a major airline company at a major east coast hub...
They have one extremely large FDDI ring to connect all of the network equipment. One of the runs exceeded the distance limits. A 24 port switch with 2 FDDI connectors was placed in the basement closet of a hotel that was in between the two points. It was used to break the maximum distance up and allowed us to complete the ring.
Now the problems. Every time something happened to our switch in that hotel basement, it tooks us anywhere from minutes to hours to get access to that closet to check that switch. It seemed every time we went to that hotel, there was different people working there, some would just let you in, others required something in writing, others wanted us to call the hotels headquarters and speak to an IT person, some would send us to the building engineering people, the key on the door changed, it was after hours and no one to ask except for the receptionist or the door man, power was removed to that room etc...
It sucked. I never understood how that airline had so much for redundancy but allowed all network attached equipment to be attached in one FDDI ring that was many miles long.
Don't confuse *baud* with *bps* (http://en.wikipedia.org/wiki/Baud). Your 2400bps modem actually ran on a 600baud, and your 9600bps modem ran on 2400baud.
And for my next trick I'll tell you why a byte is *not* always 8 bits.
In a former life, I used to work for a sweatshop that called itself a PR firm. In the mid-90s, I was an administrative assistant who was eager to get into the IT industry. The penny-pinching management saw this as a way to get rid of the consultant they used to manage their unix system that ran their contact management database. They called this system their "mainframe," with the utmost sincerity. When I finally got my opportunity to help out with the system administration, image my surprise that they were running 30 dumb text terminals off of a 486 50Mhz, with 50 Mb RAM. And they wondered why it was sluggish from time to time...
I don't know if this would qualify, but I've noticed it several times. How many people out there have become the on-call support for your parents/grandparents girlfriend's parents/grandparents etc... Computer problems? Not going to go into the chaos of the actual computer configurations spyware/crapware etc., but I'm talking about the general config of the hardware. It seems to almost be a given in these situations that the computers and peripherals will be stuffed into some piece of furniture where it is fundamentally impossible to get them. Cables will be coiled together in an attempt at neatness and organization. A baffling array of connectors and adapters will be employed, e.g usb to serial cable to serial, serial cable to USB to connect two USB devices... Also cables will be stretched to their theoretical maximum, so that peripherals and the case itself cannot be moved on centimeter in any direction without decoupling the entire environment and its baffling array of cables and connectors adapters, etc...
In the IT department one day our old linux mail server went down. been so long that this had happened that no one new where it was. they had to trace the wires through the floor. where they finally came to a wall, turned out it been running without a bug for so long it had been walled in when they did some remodelling and no one had noticed for a few years.
When I was in the USAF and working in Saudi Arabia in the 1980s, we used data and voice hookups provided by the local telco to connect 3 sites scattered out there in the Kingdom...
Trying to trace some problems with one of the voice circuits, in one wiring closet we found the voice circuit had 3 twisted wire splices in one 6 foot section of cable...
Later, as a contractor running the email system for a not very large, but well known US government agency...
A few months after the agency had moved a lot of people into the new Ronald Reagan office building in Washington, I noticed their GroupWise server didn't seem to be functioning.
I called the local GroupWise administrator and told her they needed to take a look at it.
She called me back an hour later to say they hadn't been able to locate the server and were still looking, but she's sure she had seen it recently...
Two days later they found it in a closet in the old building they had moved out of months before...Still running, but someone had finally shut down the network in the building...
Goofy, Geeky Gifts and More!
We had a very smart guy on our team who had little operational experience and a great deal of theoretical knowledge. He set up our suite of unix boxes to share home directories - not a bad idea, right? Except that this brainiac hard mounted the home directories for all users.
The first time that the box hosting the home directories went down for maintenance, no one could log into ANY of the unix boxes - each was waiting for the hard mounted NFS resource to become available.
We switched to soft mounts of home dirs, and life was much less painful for us. (Mail was hosted on a different platform, and few users wrote to their home dirs much so we didn't need to worry much about nfs lock problems.)
But Herr Heisenberg, how does the electron know when I'm looking?
I worked for a HUGE multinational ISP once. We had just gotten France hooked up and they had been running fairly well for about six months after two years of testing. About 100k customers used the service.
... a 386 LCD laptop. The machine had died because the logs had filled up the 1.2 gb hard drive. We couldn't believe it until someone rebooted the damn thing, and DNS came back up. We had been running production DNS on this thing for over 2 years.
One day, DNS went down. This had happened in the UK a lot, so we barked up the wrong tree for hours thinking it was a Keyring issue over the Transatlantic connection. Nope. Hours later, we found the DNS for France was on a different subnet. This led to discovering that their DNS service was on a set of IPs that pointed to one MAC. Finally, the people in charge of the data center said, "That's not our subnet. I don't know where you are getting DNS from.
We traced back and back through routers, entering territory that got scarier and scarier. It went to an older building that were were in the process of closing down and selling. It also had a data center, but that room had been dark for months, and DNS had been working up until now. Back and back we went.
Finally we found that the trace went through a disused subnet through a former office LAN in that building. This traced it back to an office, which traced it back to...
Turns out that when the French network architecture was being set up, they had to transfer DNS somewhere temporarily as part of a testbed, so some guy had an old laptop in his office he just hooked up. Then he was laid off before we went live. Nobody ever switched it back, and since the office space was being abandoned, no one every went into the office to turn anything off, figuring it was somebody else's problem.
A week later, French DNS was running on a production server.
I am impressed it lasted that long on such a platform.
We also used to run the flight schedules for Lufthansa. It was a Windows NT 3.5.1 system that was running on a 486, and was running some proprietary terminal service and scheduler. It crashed once every 31 days (there was some bug where it would crash after xxxx hours which was between 30-31 days). The only way to fix it was to hard reboot the box, and the directions were scary: "Go down to the older server room, and find an unlabeled shelf next to the first door near the panic switch. On the bottom of that shelf is a box which is behind a stack of old 10base hubs. Hold down the power button until the green light goes off. You may have to lie on the floor on your stomach to reach the button. Count to ten, power back on. Make sure the amber light labeled 'turbo' is lit on bootup. If not, repeat, but wait 60 seconds before powering back up."
I sure hope they got that fixed, it was last like that in 2000.
I still remember (very fondly) a Novell Netware 3.x (I think it was 3.14) server we ran up until the middle of last year. It ran several key database applications for our company (in Paradox, of all things) for about 8 years, and spent the last two years of its life storing archival data. It was running on a 486-66 in a dusty little small-tower case tucked away in the corner of our server room. We threw it together from spare/retired parts one weekend after we moved into our new HQ building and realized much to our dismay that some of our database apps wouldn't run on Windows NT. To the best of my knowledge- it was never shut down (it was on an even older APC UPS that went through at least a half-dozen batteries during that time) or had to be rebooted in the 9+ years it was in service. We finally retired it last year, and had a little ceremony as we listened to it grind to a halt.
:)
Even my Linux boxes haven't proved to be as bullet proof as this server was, although the come damn close.
Before we moved to our new HQ, we ran an enormously complex hodge-podge of Netware servers on a spider web of a 10BASE2 (thinnet) network. Coax cable still gives me nightmares. It seems so archaic now with GBit networking and rack/blade servers.
How about a full hospital using only WinPopup for email, and refusing to do any more than that?
The plumbing term is a `trap`
I'm sure plumbers would mind you calling it a U-bend as much as you(we) would mind plumbers calling a data port a phone plug or calling Internet Explorer 'the internet'.
To bring this together, my odd setup was for a trucking company. A full rack with UPS, 3 servers, PRI feed - all in the bathroom.
Yeah, try to find the least hostile place where everyone smoked and the bathroom was it.
if you steal from one source, that is plagiarism, if you steal from many, well, that's just research.
So what happens when the air condition does go down?
Do the network guys get notified in time to fix it, or do you just rely on the thermal breakers?
It sounds like you don't have much time to react.
Fast, Soon, Correct. Pick 2.
Our office is located in an old steel mill, which now houses multiple companies. The demarc is located in another building. To get the PRI from the demarc to our office, our copper does the following:
We later found a hanging punch block the ceiling of our office. This happened to be the connecting point that we could never find, so had an extra 200ft of copper running for now reason.
This may not sound two bad, but for my boss and I, who have never really done any type of wiring work, we think we did reasonable well. Especially since the wiring job for the building was a mix of wiring from the 50's and a local wiring company's 'on the cheap' job. It only took us about 3 days and we managed to knock out another company's T1 with our tone out tool.
I used to work for an arm of a large multinational corporation, where I was working on a large project worth £Millions+ . Two things from that:
1. We tried to get board approval for the installation of a Biztalk server to perform everything we needed for the project we had, and this was quoted at $50k. This was duly refused, so we then hired 2 consultants to bodge around with the existing "temporary fix" system that had been in place for 3 years and probably blew more than that on wages. It was a nightmare to test, as there were about 18 different stages to the data flow through the system on 3 different OS requiring user intervention at pretty much every stage.
2. For the same project, we needed a dedicated liink to another (much, much, much! larger) company. This was something like a 512kbps pipe that transferred the grand sum of 3MB per day. Anyhow, our lot (the firewall experts) had to reconfigure the company firewall to allow the connections (using some odd piece of encryption software - scp apparently to easy) to go between us and them. After four months of trying to configure the rather nice shiny red Check Point firewalls, costing about £16k (after I jokingly said it could be done with a £50 linux box), and nearing the go live date, the project leader asked if I could actually do it.
One old desktop PC, and 2 hours later, we had a working setup. That old desktop PC remained in the main firewall rack humming away for a good 4 months until (finally) someone who knew what they were doing stepped in and opened the one port for the one IP address needed to make it work.
I still reckon that box was the most secure perimeter firewall in the company (and not sure anyone outside of our group knew about it as they would have gone mad).
So friend of mine was working in company where their all mighty IT manager makes decissions by the name of the product. Like which sounds beter Win 95(or NT) or Workgroup Windows(3.11)?
Well the manager thought of course we should take this workgroup thing. So they end up degrading all they computers to much older version of windows.
Don't know what happend after that.
Oh and this was in year 99 or 98 if I remeber right...
I was told from someone I worked with, apparently it happened to him. some higher up called in saying some server they used was not working and wanted it fixed asap as it had valuable data and the likes (Probably the CEO porn collection but anyway). Ok fine, got the details and such, and went to check on it. first trying to find the passwords and all that fun stough for the box as he had never worked on the machine before. No one had a clue what machine he was talking about, or where it would be located. He ended up going server to server in the large server, reading the labels trying to figure out which one the higher up was talking about. He ended up finding the ip address and tracing it to the specific wiring off the switches, and proceeded to follow the cat5 cable. And if anyone has ever tried to follow a cat 5 line through a large server room when all the cables are bundled together, you know this is a run adventure. the cable finely went through a wall, not a big issue, must head to the next room over correct, they had cables doing that all the time. but nope figures,cable comes out no ware at all. He finely found someone who told him that they had done work in the walls in that area even walling up some old non used closets. . Doing some measurements he found the wall between he 2 rooms was rather wide, and would have had closets/storage there in the past. Sure enough after a few holes n the way he got to look in on not one but I believe 3 machines just sitting there working away. One the non working server. How they got walled in without anyone knowing, who knows, but hey decently a odd one.
Sig says: There is no such word as "alot," and if there is, there shouldn't be. It's "a lot." Two words, not one.
Then, of course, there is the alternate spelling, "allot". Very annonying.
On the other hand, there is a precedent in the English language, "tuppence".
www.wavefront-av.com
One of the first jobs I was a part of early on in my networking career involved 'updating' these lovely OS/2 machines to DOS/Windows 3.1.
My consulting company assigned me to a bank that had just purchased a competitor. The competitor's system was ALL OS/2. Man, that was the most beautiful, efficient system I had ever seen. The tellers, ATMs, office computers, servers - all OS/2. Our banks? DOS/Novell/Windows 3.1.
It was bad enough to arrive at a building full of distrustful (and rightfully so given the fact that many of their coworkers were fired, salaries slashed, and meanwhile the execs got cushy bonuses), openly hostile people, but you should have seen the look on their faces when these systems were converted. All booting occurred off of the network; well considering their network was a 4 Mbit Token Ring, it could take 10-15 MINUTES for the pointer in Win 3.1 to even show up!
Then there was the 'in your faceness' of the new corporate overload logo that appeared while things were loading. It was miserable for them and for me. That saddest thing was, I was an Amiga user who understood why multitasking was so important to many of these people. They didn't have the computer knowhow or terminology to explain to their bosses exactly why this new system sucked (other than the load times). Funny how time is always money to businesses, and yet they typically ignore the amount of time a user has to wait in frustration to get their job done!
It was an ugly job, and I quit shortly after starting that project - mostly out of disgust.
"...Well, there's egg and bacon; egg sausage and bacon; egg and spam; egg bacon and spam; egg bacon sausage and spam..."
Some time ago, local branch of HUGE telecommunications company was asked to connect two buildings. Distance: about 5 km (4 miles?). Really no-brainer kind-a-task.
The tehnicians decided to use aDSL (128/1024kb) and make a tunnel through public internet. Only three computers were on one side, using RDP application, (minimal graphical setings) so they figured, that was engouh. But wrong they were. On one side Siemens aDSL router, and on other side Cisco firewall and router simply did not like each other. The stuff worked terribly when only one computer was online, and tunel crashed when one needed to print or to use more computers (two for example).
The price was low, as requiered, but it did not work. Then they tried to use their routers (much more expensive) to creat encrypted tunnel. It failed again.
Then local radio amateurs jumped in. They used their own 802.11g links and using old, for-junk-ready-computers with monowall and linux configured encrypted point to point tunnel through their network, using svereal routers and nodes. They shaped tunnel to only 50kbps and stuff it works. Total out-of-service time in one month? About one hour. Average router/AP uptime is about 50 days.
Why? It appears that some (expensive) routers are sensitive to high bandwidth fluctation at such low bandwidth conditions. Wireless software on the other hand is often made having that in mind. Somtimes less is more.
Doing a good job is like spilling coffee on a dark suit, you feel warm all over, but nobody notices.
Email attachment size limits probably came with the increase of virus and spam scrubbing. Software on the mail server would pattern match 100,000 possible threats against everything arriving. That made our own mail server delay incoming email for hours sometimes, so we stopped doing that (and went to a different system). There's no practical limit to an email attachment size within the bounds of file system rules (I've sent multi-gigabyte attachments to myself for grins).
Otherwise, ISPs are not that interested in storing more than they need to. I've actually seen people email files to themselves, and keep them in email, because they can find files easier in email than in their file system.
Most of the stuff on
A decade ago it was common where I worked for people to hand me paper output of their word processed original content to put on the web. Instead of teaching them how to put the file on a floppy or network drive (and I sure didn't want to retype it), I asked for and got a copy of OmniPage for my scanner.
I wonder what precentage of OmniPage sales has solved the same "problem".
On one job, we were required to have our PC's power switch on a keylock. So, they installed a little plastic box over the BRS (big red switch) - in the box was a key operated lever to operate the power switch. Problem was, all of the techs took a shortcut installing the things and it trivial to remove (too about 10 seconds). When I pointed this out to management, I was informed that "all security procedures are well thought out and would not be discussed with employees."
Not sure which was the bigger hack - the device or the policy.
I was incarcerated in my late teens and was given an opportunity to work as a teacher/technician inside after attending school for a while. Being a prison school, we were obviously way, way down the list of budgeting priorities for the state. We had a 30+ client network with every sort of OS running; Windows 95, 98, 2000, XP, Mac OS 8,9 and 10+, even a few Red Hat machines. My supervisor decided he wanted something better than buggy, half-functioning peer-to-peer and asked me to set up some servers. Partly he wanted to simulate the internet for the students, who had no access to outside networks, and there was also a security issue - a peer-to-peer network didn't provide good security to prevent tampering with publicly-shared files or to provide levels of access to shared data and student records.
So we requested some computers from the state to start making improvements. What we got was a pile of surplus Pentium 1's and 486's that the Judicial Branch and DMV had thrown away. It was a nightmare even getting them to work; half of the BIOS batteries were shot and we had to salvage new batteries from broken old Macs so we could reset the BIOS passwords and boot. Once they were working, we had to deal with the equipment's age. I ended up clearing out an old closet and setting up a whole bunch of sketchy, half-dead servers which were just barely capable of running Red Hat. One did nothing but run Apache, one did nothing but run MySQL, one just ran Samba. It was a nightmare to obtain outside software, so I had to write all the database software and our new online test-taking system in PHP as a server-side system, which was the only way it was guaranteed to work on every client in our crazy hodge-podge network (and there were still problems).
What made things worse was that we had only one UPS and that went on the file server, so I had to write backup scripts that distributed copies of the database and my source code to hidden places on client computers just in case we had a storm and a server's hard drive got toasted (this happened once or twice). An even weirder, and prison-specific problem, was that the administration of the facility was extremely distrustful of the school and our program in specific. They would periodically storm into the school claiming to have received an anonymous tip that we were engaged in some form of illicit activity and confiscate a computer without any regard for how this affected our operations. Seemingly harmless activities - playing games for a few minutes before knocking off, using your computer to type up a letter - were huge offenses inside that could get the culprit thrown into solitary and endanger the program. It was incredibly strange to be in prison and have the privilege of working in that environment, all the while that threat hanging over my head an reminding me of where I really was.
I worked at a place that had a NAT'd IP range. The range at work was one octet off from my home range. So one day at home when I thought I was ssh'ing to my server, I mistyped one octet and ssh'd to an invalid IP address. Except, I saw a lab-router> prompt. My heart jumped a beat and I pondered what to do next. I hit ^-] and typed quit and called my cable company. I escalated to a tech and told them that they have some lab network exposed that shouldn't be. I specifically asked them to call me back, they didn't but then the next day it wasn't available anymore and wasn't again.
It was clearly a cisco 2600 (or similar), I recognized the banner. I found it incredibly funny that their lab wasn't VLAN'd away from the customer network.
We had our air conditioning limited and thermostats locked up in the machine room and it was always sweltering. We defeated that system by hanging a drop light under the thermostat.
Most of the stuff on
Back in the Reagan years, when LANs were by no means ubiquitous, we had HP 2621 ASCII terminals on our desks, and worked on a VAX. Management was extremely cheap (how many of you old farts remember the AT&T "Low Cost Wins" campaign?), so we connected to the VAX by dialing into it over the building PBX using scavenged Bell 212A 1200bps modems.
One night, I found in a junk pile a hundred feet of 50-pin ribbon cable, so I crimped down two DB-25 connectors on each end, ran the cable back to the machine room which was in the next aisle over, connected that end to two spare serial ports on the VAX, and my end to two terminals: mine, and my officemate's. It was tricky keeping our 9600bps "high speed" lines secret, but it was really nice to be able to use Emacs instead of ed.
Sorry. My brain jumped on to the other.
:D
Actually, our own data center I'm kinda proud of, being that it is one large hack. We're in a mixed residential/commercial facility, and it happens that our unit was residential prior to our arrival. I converted the master bedroom to be our server room (holds 10 racks...not too shabby for a small operation!), have a 5 ton air conditioner piped in, picked up the racks themselves for about $100 each from a reseller here in town, and our original servers were Compaq desktop machines, all sharing NFS back to a RAID array, and using load balancers to make many of them appear as single servers (thanks for the idea, Google!)
Ah the joys of doing it yourself.
Karma: Chameleon (mostly due to the fact that you come and go).
I have already noticed a couple similar scnario's to the ones I have seen or created but here are the best of the worst. Found an old Novell server that was used to QA monitor and ship product from a foundry type industry, it ended up crashing and when it failed i located it under 4-5 feet of silica sand. A tow moter went through a wall in a crazy accident at a factory I was working at,it "lanced" our manufacturing systems server through the wall and went right through the middle of the server exiting out the front of the system, it missed 90% of the system's key components and was used in production for 6 months after the event!
Some years ago I was the computer guy for a fairly large network installation in a student dormatory. Because our 19in racks are (still) in the bicycle cellar, our equipment used to die of too much condensing wetness quite regularily.
So I ordered replacement parts and took the network switch out of the 19in rack into my apartment to replace all the components (thanks HP for their modular design).
On the way I encountered a panicking student, who knew me as the guy responsible for the internet, and asked me _WHILE_ I was carrying an insanely large box with a giant bunch of network connectors, "Do you know that Internet stopped working, like, 5 minutes ago?"
Maybe the storm water drain wasn't being used anymore. I guess as long as there isn't a risk of something cutting the wires down there (rats? unclogging service?) it wouldn't be a problem. The wires are insulated and waterproof after all. At the very least they should put the wires into their own conduit to protect them.
It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
In the Bahamas I witnessed a high school with multiple buildings being wired together by drilling holes through the brick in the side and literally running cat 5e across the ground - the school said they would take care of it later - In another school My team (Missionary team of IT personnel) Ran out of cabling while setting up a lab of 50 computers. As we were doing this free, and the school could in no way afford to have it done later - and anything we didn't do would probably not get done - we found some old telephone cable (8 wires not twisted) and used it for the last 10 computers - they dropped quite a few packets but did the job better than not.
In the Philippines I was part of a team that rewired the network for a seminary. When we got there the seminary was running its entire network over a series of D-Link routers - about 300 - 500 computers (its a small seminary). Unfortunately The seminary was running cat5e across the ground and through sewer pipes between the buildings. These pipes were exposed and every time it would storm these pips would attract lightning and the first router in the series would be fried through the Ethernet port.
That's excellent. Now I have to figure out what I can do with the setled call in AIX.
Intron: the portion of DNA which expresses nothing useful.
The phone lines in our office use RJ45 connectors. I expect they are intended to be used with the type of fancy phone system one would expect in a building with four whole rooms (2 of which actually seem intended to be used as offices)
After the person who hooked up our phone lines informed us that there was no way to use an old phone with these connectors, no converter, no nothing, we would need to buy new phones (which Bell South had a fine selection of, of course), and then each of the offices would need to be wired professionally (which Bell South could do!)...
I found that we still had: 1) A crimping tool 2) Leftover CAT5 cable 3) more splitters than could possibly be justified by a single DSL line.
I know nothing about phones, or electrical anything beyond my kindergarden science fair project on Conductivity (pennies DO conduct, oatmeal lids DON'T!). This is a piss-poor job.
Every phone in our office plugs into a splitter. An RJ11 crimped onto a CAT5 cable (two dangling wires) plugs into the same [read:wrong] end of the same splitter, which then plugs into the wall's RJ45 connector.
At the other end is a nest of wires in which many small wires pulled out of a CAT5 cable connect every line of the terminal to the same connection on the terminal as the only "real" phone line in the building (which is located in the 4'^2 "lobby")
I'm pretty certain the terminal is not intended to have four wires stacked onto the same connection. But as I said, I'm not an electrical anything. For all I know, this is how it was meant to be done. All I know is: It works.
-- 'The' Lord and Master Bitman On High, Master Of All
Problem is, if you leave a drain long enough without water passing through it, the water in the u-bend can evaporate, leaving an empty pipe and allowing the nasty sewer smell to escape. Thus, leave a faucet dripping to keep the U-Bend full!
This is ludicrous. The amount of water in a trap is *MASSIVE* compared to the amount that would evaporate in a day. It would take 6 months or more for all the water in a trap to evaporate. And if such a time period had passed, all the sewer gases in your home would have long since been evacuated from the piping via your stack anyway.
People please do not listen to this person's idiotic comment - leaky and dripping faucets are HORRIBLE WATER WASTERS - if you are on metered water they can cost you tens of dollars a month, and even if you are not, you're being very wasteful and not very environmentally conscious.
There was one incident for a large national credit card company. One day a guy in the IT dept decided to do some spring cleaning around his desk. So he was unpluging and plugging in some PCs he had under his desk to do some cleaning. Well they kept losing a call center (About 500 miles away) spuraticly that day and no one could figure out why. Then some smart guy noticed that every time the guy would go under his desk, the call center would either come back or go out. As it turns out, a big part of the call center's application was running on a PC under this guys desk. And ever time he unplugged it, the call center went down.
I used to work part time as a volunteer engineer at a community radio station. Aside from the digital audio playout system playing music off a Novell server, we had no money for IT. We occasionally experienced problems with stuttering music, caused by people transferring large files across the network which were causing packet collisions and interrupting the critical stream of audio data to the studio playout machine. This was prior to MP3 and the like, so the best audio file compression we got was 4:1 and the required bitrate was pretty hefty.
I was given a new 100Mb hub, the theory being that 100Mb is faster than the 10Mb we already had, so it would solve the problem. Not so! Those large files would still collide with the audio streaming because we had no intellient routing, traffic prioritising or cash to pay for a decent solution.
I discovered that the new hub would auto-sense the 10/100Mb speed from the NIC at the other end, but had no internal 10Mb100Mb switch capability. In other words it was effectively two hubs in one package with separate 10Mb and 100Mb buses inside. That turned out to be advantageous in the end. I set up all the audio workstations to run at 100Mb, and all the administrative workstations to run at 10Mb, so effectively we had two separate networks, one for audio and the other for admin. So the secretaries could continue sending their large files around, printing and so on, and it didn't affect the audio operations. There was only one PC that needed access to both audio and admin, and I solved that by simply giving it two NICs (making sure they weren't bridging).
It ran that way for several years. I believe more recently they employed the services of an IT contractor, who promptly saw fit to replace my old 10/100 hub with a fancy new switch. Almost immediately the stuttering problems returned. I don't know what they did to remedy that, but it seems to be better now. Presumably they have a bit more of an IT budget these days.
all drains will back up, it's just a question of when.
it is perfectly acceptable to put the chillers outside the server room on a lowered floor, folks. hire an architect.
if this is supposed to be a new economy, how come they still want my old fashioned money?
The problem isn't with running a low-voltage wire through a high-voltage conduit - the problem is that someone could later run a high-voltage wire through the same conduit.
Although that is a good reason to NOT use high-voltage conduit for low-voltage runs I do NOT believe it is a code violation (it is merely "not recommended"). In the interests of simplicity in large installations it might be easier to use the same kind of conduit for all runs. The remedy to the problem you describe is to MARK THE CONDUIT PROPERLY. I worked for a time in the Physical Plant dept. of a large medical facility and the same kind of conduit and cable trays were used for all sorts of wiring. To meet code/safety requirements the conduit was marked with standard coloured stripes and labeled at regular intervals ("comms", "low voltage", "120/208V", etc).
The electrician/inspector did not have to order the conduit removed--it probably would'vebeen fine to paint a stripe down the side and stencil "comms cable only" at regular intervals...thus it is a suitable candidate for "silly hacks".
I briefly worked for a dodgy website that was trying to be a sort of Yahoo! for music, but wanted to entice users into creating new content for free that then a couple of editors would tart up. Despite having seven marketing staff, I was the only techie (I subsequently found out that the three previous techies had left after disputes with the boss - not bad for a company less than a year old). The entire company was run off a single PC running RedHat Linux 5 - database, email, webserver and fileserver. They had installed everything, back when doing so meant it was enabled by default, and no firewall rules had been setup. No updates had been installed either. I pointed out that this was a disaster waiting to happen, but was told to forget about it, as updating and cleaning the machine would require a reboot and therefore an outage. No budget was available for a spare machine, as the bosses boyfriend kept buying stupid things for the office such as expensive modern art and uncomfortable designer chairs.
When the inevitable happened, and the server got hacked, the boss accused me of doing it. Once I stopped laughing, I asked her why I'd hack a machine I had the root password to. She then insisted it had to be an inside job, as hacking was such a superhuman feat in her opinion. I told her to stuff her opinion and quit. She then tried to hold me to my contracted notice period of three months, until I threatened to take her to court for constructive dismissal.
We had same problem with one MRI scanner I have used. Device room is so tiny that one cannot go around the unit and AC unit is cheapest I've ever seen. They punched some holes to the hallway but temperature in summer goes over 38 C (about 95F) and the machine (Sun inside) generally freezes after two hours of operation.
Same thing happened in other institution with more powerful MR scanner (also Sun inside) where AC unit was failing on a regulary basis, at least once a month. In one year they had that scanner fail so many times that they basically could buy a new one easily for the same money.
Thing is, computers in those scanners never failed, only the amplifilers.
Doing a good job is like spilling coffee on a dark suit, you feel warm all over, but nobody notices.
Comment removed based on user account deletion
about 3 years ago I interviewed with a fly by night Government contracting company. They showed me the server room which was in the kitchen (the kitchen sink was like 10 feet away from rack full of servers). I wanted to ask the guy what was their emergency backup plan if somebody spilled coffee the router. I bite my tongue instead thinking I didn't want to sound like an ass. Looking back on it I should have said something because the job I interviewed for wasn't that interesting anyways.
-Dipster
Back in the day, I used to have to go in late on Saturday nights and copy & compress the system disk on our PDP-11/70 system. It was located in the typical loud, cold computer room. THe copy process could take anywhere from 1 to 5 hours, depending on the amount of fragmentation on the system disk. I didn't want to wait in the loud room, and I didn't want to get up from my comfy chair in my office at the other end of the building to continually check on the progress. One night, tuning around on my FM radio in my office, I heard a funny sort of noise at 98.5 MHz. Its rythmic structure reminded me of the sound the disks made while they were seeking during this copy process. Sure enough, thise old school disk drives, with their Emitter Coupled Logic (which uses about a pound of electrons to do anything) were generating lots of EM noise, which was, I'm guessing, getting coupled to the power line and thence to my radio. After that, I could kick back and have a few beers, and listen to the radio to know when the copy was over, without going back and forth to check.
There is no God, and Dirac is his prophet.
So 10 years ago I worked at a small ISP where the owner lived in a small apartment. In his apartment, he had a couple bonded ISDN lines that he then resold bandwidth off to a smaller business. Ever time the power went out (and the UPS ran out of juice), the ISDN equipment had to be manually reset for his and the customers lines to come back up. He went on vacation and of course the power went out. When I tried to get access to the apartment, the leasing company told me no, not without a signed letter allowing me access. I thought a bit, then rummaged around in his desk at work, found a contract he had signed, cut his signature out, taped it to a hastily typed letter saying I was allowed access to the apartment, and faxed it over to the complex. I then waited 30 seconds and called them up, saying he had just faxed it over to them and did they see it. They said "Yep, come over and get the keys" Made a couple copies before I gave them back. It was too easy.
...let me introduce you to the Shark Tank.
Constitutionally Correct
Between the Plain, &, and % codes you really could get away with things like:
ATZ0%F0&UC&K2O1F&F which by the way is (IIRC) a perfectly acceptable init string to reset the modem to chipset default and then turn off some reporting that nobody uses anyway. It might also disable one of the 56K compression algorythms.
Which in turn is why Americans complain about how weedy electric cookers are, and don't generally boil water with electric kettles: I can have 7.5KW into an electric cooker (30A, 240V) or 3KW into a kettle (13A, 240V), while Americans can't get 70A into a cooker.
Actually in the USA most electric stoves (cookers) are supplied with 220V AC circuits, these consist of two 120V legs 180 degrees out of phase and a neutral wire. Household stove circuits are often of 40A or 50A capacity at 220V, which translates to 80-100A at 120V. People (particularly American consumers) wouldn't stand for stoves that didn't cook. For most of the USA, it's simply cheaper to use natural gas as a heat source, and it's easy for a manufacturer to increase the heat output of a stove burner by using larger gas jet orifices in the appliance since the gas pipe supplying the stove is often capable of over 100,000 BTU/hr - not so easy for electric stoves that are inherently limited by the wiring.
Being from the UK myself and living in the USA since 1982, I think the electric kettle thing is more of a cultural thing than a practical one. Americans don't drink much hot tea, preferring coffee. It's much easier to get your coffee from an appliance designed to make coffee, than a general purpose water heater like a kettle. Some people have hot water (about 190 degrees F) dispensers on their kitchen sinks for making things like tea. Kettles take up a lot of room on the kitchen counter surface for what they do.
Putting moderation advice in your
Your stories all involve corporations, IT firms, and other techie employers. Mine doesn't. It involves a tiny mom n' pop (literally) hardware store somewhere in the corporate wasteland of northern Delaware...
I worked there full time for three years and still work there one piddly day a week because I know what I'm doing and it's easier than hiring and training somebody else for that one pissant little day. I'm the resident bithead, so naturally all computer problems get routed directly to me. I have no idea what these people did before I worked there. (To be fair, I work for a boring old computer-laden engineering company by day.)
This place has a computerized point-of-sale system. It's called ABC, and I guess the right person with the right background might know that this is the "old" POS system used by TrueValue, formerly ServiStar - the two merged in the early 90's. The "new" system is Triad, and it runs in Windows in a semi-modern manner. No such luck for ABC; It runs in MS-DOS 6.0 (not 6.22) over Netware 3.1. All of the machines in this system (there are eight in total) are IBM GL300 workstations. Pentium 3's, with their vintage copies of Windows 98SE installed which are not used. Instead, every single machine in this system boots FROM A FLOPPY DISK. The hard drives remain untouched; Everything is pulled from a mapped network drive through DOS and Netware, which naturally resides on the server. And is read from floppy.
The mind reels.
So one of the first things I did upon discovering this was steal all the floppies and run them through my laptop, burning bootable CD's on halfway decent media and saving the floppy images on my hard drive, to be squirreled away on my file server at home. I loaded up every machine's CD's and set the boot order to go floppy -> CD -> hard disk. Already one of the bookkeeping machines upstairs and a POS terminal have had floppy failures and nobody even noticed until I pointed it out.
But it gets better.
It is of note that the ABC system runs in DOS and therefore requries very specific Netware compatible DOS drivers for its ethernet cards (IBM EtherJect 10/100 PCI cards, if you must know) and the sytem will only work with that specific type of card. Which is, helpfully, not made anymore. We've had multiple network card failures on one machine for reasons I'll get to in a moment, so my solution was to hie myself hence to eBay and purchase a lot of 25 of the stupid things for about as many dollars, which now reside in a file cabinet upstairs in a cardboard box maked with black sharpie: "Spare ABC parts network cards. Do not meddle with, mangle, or misplace, or we keel you." I now have some spare ABC boot disks and CD's in this box as well; If it is ever lost the entire business will surely go up in flames not long after.
I should explain the proprieters of this establishment. It's owned by a husband and wife couple, and their son. Mom, actualy grandmom, is pushing eighty. Her husband is in a similar situation but his health dictates that he isn't involved in the workings of the company much anymore. They, and their son (pushing 50) are not technically inclined in the least; Grandmom does all of the bookkeeping and paperwork upstairs and refuses to use a PC; Instead she uses a vintage IBM Selectric typewriter (I kid you not) and has somehow, somewhere, managed to scrounge up a source for ribbons for the silly thing.
All of our receipts and invoices are filed, by hand, in an imposing wall of file cabinets behind the office upstairs. They're in chronological order, every single one of them from about 1963, from when they bought the place from another owner. And grandmom can track down and pull any one of them inside of three minutes, and she's had to do it to settle disputes.
The nightly backup is put on an ancient parallel port Iomega Jaz drive on cartridges that I'm not entirely sure even still hold data. We've never had to restore a backup and I've never pressed the point. Some day I will.
But this is the crown jewe
Not that anyone will ever see this down in AC land, but what the hey...
/quite/ that as it was a segment offset below that, but I think the segment offsets must have been 4KB as I believe I remember 700KB being the max. Of course, one couldn't use the /entire/ 700KB, as DOS itself took up some of it, and its various memory managers (himem.sys to enable 640-1MB access, another one, 386enh or some such, that managed > 1MB by swapping 64KB segments into a 64KB window in the >640KB area) took up a bit more, and various hardware drivers required a few KB below 640KB or 1MB if they could go above it, and the order they were stuffed into the 640K-1MB area made a difference because some were big during init but then shrank down substantially so they had to go in first while there was still room or they'd end up taking up main memory 640KB area, and... .
/just/ right, often spending hours on hours doing so, only to have to do it again when a DOS or driver update changed how one or another piece fit or if one upgraded something, one could actually get a bit above 640KB free for use by a real program, tho just getting 640K free was considered quite a feat, almost a miracle. 654KB would have been possible -- just barely, and ONLY if one configured things /just/ right and didn't have any hardware with stubborn drivers. In theory, I think it was possible to do 680KB, but I don't believe I actually ever saw that available.
The 640KB limit wasn't exactly 640KB. You young whippersnappers are probably too young to know about it, and yes, some of the old guys around here outclass with their pdp-11 talk, but it was actually possible to run up nearly 64K beyond the 640K boundary... and to access slightly beyond the 1MB boundary as well -- the area called the HMA High Memory Area, if memory serves.
Remember, DOS used a segmented memory model at the time, two 16-bit words. The high word was the segment address, the low word the byte within the segment. IDR what the segment offset was (four sounds familiar, but IDR whether it was 4KB spacing, or four in the 16-bit=65535=64KB block, so 16KB spacing), but they overlapped, so different overlapping segments could reference the same physical memory location at different byte offset addresses relative to the segment.
The trick to addressing that last bit of memory was to reference the last possible segment in the allowed area (640k or 1M), and then use byte addresses that extended beyond the segment limit. Since the addresses were 16-bit the segments were 65535 bytes or 64KB, and one could access nearly that much space beyond the limit (minus one segment offset) -- with the limitation that it all had to be treated as a single memory block, used by the same memory resident program, since only that last segment could access the entire thing.
DOS had a couple config.sys options at the time, DOS=HMA, and DOS=UMB (which could be combined on a single line as DOS=UMB,HMA). UMB if I remember right allowed DOS to access the area between 640KB and 1MB for TSR (terminate and stay resident, the terminate referred to the initialization) programs, mainly device drivers and the like. HMA referred to the nearly 64KB above 1MB and told DOS to move as much of itself as possible to this segment, leaving only a stub, some 4-16KB IIRC, at the bottom of the 640KB area. The catch was that as I said, only one thing could be in that HMA area and DOS didn't take quite all of it, so if someone had something that could use the area a bit more efficiently, they'd not tell DOS to take the area, but tell this other thing to use it instead.
Anyway, back to the 640KB. 640+64=704KB. As I said, it wasn't
Anyway, if one were lucky, and had spent the time learning how to fit the puzzle pieces consisting of all the drivers and other bits in
Of course, back then, configuring the memory and drivers was the easy part. That was before PlugNPray, and just to get things working at all, one had often fought
Oh where to start...
I worked for a PHB a year ago who believed that she knew everything there was to know about computers, or at least more than me, despite not being able (literally) to tell the difference between a modem and a NIC.
When I was brought in as Network Admin, I was told that my first major job was to figure out "why the servers crashed so often". Not the best sign. I was then shown the server room. It was the furnace room, a tiny closet with no windows or a/c, piled high with every type of non-computer related trash (as in plastic wrappers and empty boxes, old food containers, empty coke cans etc) and 8 "servers" (some were Compaq ancient desktops, the ones used for the backup systems of course) running NT (a few were 2000) squashed together on tables. There was a server rack, but most of the servers didn't fit on it. On the ceiling a box fan had been hooked to the drop ceiling with coat hangers. As the cord did not reach the outlet, it was plugged into a power strip that hung in midair. The door to this room was always closed and locked tight.
The temp was regularly between 90-110 degrees. At least one server crashed every day. She asked me to get it down to once a week, thinking this was normal. I did a full report for her on what needed to be done, including actually creating a backup system that worked (they weren't actually backing up user data at the time, only the windows installations on the two least essential servers, on the same tape once a week which was never checked). She read it, decided that first we needed to be "more organized" and spend the entire budget on a new rack. When it arrived, it was discovered that, just like the old one, none of the servers actually worked with it. So, she begged *additional* money to buy new rails etc. Two of the servers *still* did not fit, so the tables stayed. She even told me that the trash needed to stay because she needed to "sort through it" and there was nowhere else to put it.
Incidently, when the CEO's computer crashed and he discovered that his data was not being backed up, my PHB blamed me, and the CEO *believed* her. I was officially reprimanded, and my protests ignored. This is just the first story of many, but let me sum up to say that at one point I actually requested an audience with the CEO (I was the Sr. Network Admin by that time (read: "only") but she was CIO) to talk about the problems with the network, how it could be fixed and why the CIO was frankly lying to him to cover her own ass. (I said it much more politely.) I was told that if I had a problem with her, I needed to change my mind or leave.
I got out of there as quickly as I could.
At my first ever consulting gig I started for a small company of about 9 office employees and workstations. The workstations ranged from a Pentium II 233MHz w/32MB RAM running Windows 98 to a 3.0GHz P4 Mobile laptop with XP (and 40 processes worth of bloatware). 3 - Windows 98 machines 3 - Windows ME machines 3 - Windows XP machines I immediately knew I was in trouble when I saw what their previous IT guy had done. They had a very nice HP Laser printer hooked up to the network and configured for IP printing. This guy had hooked up the 233MHz Pentium box (used by the accountant) to print to it via IP Printing and then shared it to the rest of the network. Everyone else printed through that share on the Win 98 box rather than IP printing themselves. Needless to say, her old system slowed down quite a bit every time someone decided to print something.
COmputerworld has a similar daily dose of funny IT cluelessness, not really development stuff, but more tech support and hardware inanity. It's called Shark Tank.
I once worked for a dot-bomb e-commerce company. We had a product that tied into several major credit card issuers (i.e. >40% worldwide market share for issued credit cards). As part of the installation and maintenance of the product, I got to spend many weeks in MAE East (perhaps the biggest data center in the world). From what I've read about the Baby Bells' special networking rooms when the NSA scandal broke last year, I wouldn't be surprised if these servers shared one of those special rooms with the NSA routers.
The data center was about 5 floors below ground level. No form of wireless communications worked whatsoever--cell phones, pagers, etc. Once I parked my car, I had to go to an unlabeled metal door with a tiny camera on the top. Security guards would buzz me in and require me to sign in at their station. Then I would get buzzed in to the main data center room that contained another room inside of it. From there, I had to enter a password into another security system and place my palm on a palm scanner. Inside this room was another security guard--I would have to sign in with them, too. Then I would enter a different password into another security system, and place my head in front of this retinal scanner. This would buzz me into another room with the cages for each of the clients. There was a padlock on the cage, behind which were our servers. The servers required two separate smart IDs to be placed into an external card reader so that there had to be at least 2 people there to perform any maintenance. The servers themselves were locked down pretty tightly, too. It all seemed pretty insane as far as security goes, but I understood--these computers contained every credit card for the credit card issuer.
Well, after about 3 days of going to this data center, everyone got to know me. They would sign in for me to speed up the process. The security guard behind the door with the palm scanner used to get very hot, so she would often block the door open, thus defeating the palm scanner. The retinal scanner also had problems, often requiring about 3 tries before it would read correctly, so that door was often blocked open, too. Then, one day one of us had forgotten our smart card. We started cursing, as the round trip to pick up the card was about 45 minutes, so we tried it with only one smart card. Bingo. It worked. So then we tried it with no card. Seems the card readers weren't functioning properly. So, overall, we were able to defeat all of the security measures except for the padlock, and all because the security staff (getting paid 2 bucks above minimum wage, no doubt) all "knew" us. In my humble opinion, it would have been far smarter to *not* have the security guard in the foyer behind the palm scanner. After all, social engineering is probably the most common form of circumventing security.
Another funny thing about this was that we had a rather difficult security audit for all code releases. We had a bunch of ex-NSA employees working for us that were rather good about it, too. We would also hire outside auditors to do reviews of major code releases. It was all fantastastic, except for one thing: code patches didn't get the same scrutiny as code releases. In fact, they got none. Well, in order to expedite the release of one particular feature (that required emailing confirmation to customers), we packaged it as a "patch". No security audits. And for something that required the installation of a mail server! Furthermore, the code base had access to the record-level encryption used to store the credit cards. So, basically, if I had wanted, I could have installed a bit of code that would have decrypted all of the credit cards of users of our software and emailed them to a third party. I could not believe it. It's a good thing I have what I consider to be high moral and ethical standards.
I realized through this ordeal that security measures are not put in place to ensure security. They are put in place to give people the perception of security. And, furthermore, automation and removing the human element are good things for security. People should be used to monitor and oversee automated security, not to be actively involved in that automated security.
--Be human.
what a hack.
I don't know if it counts as clever or ugly, but many years ago I was working with Amiga code to control theater lights. The interface basically involves repeatedly sending a frame of the lights' current levels over a serial line. Unfortunately the data rate is faster than the Amiga (1000) serial driver could manage. However the Amiga had additional graphics chips which could use the horizontal return to do other work, such as writing out to the serial port. The vertical refresh could then be used to hold the output high to signal the start of a new frame. The end result was that the CPU was doing nothing and the graphics chips were doing all the work.
The same place had a computer with so many peripherals that the power supply couldn't keep up, so there was a second power supply sitting on the floor with wires running into a hole in the side of the computer.
As for an ugly hack... Yesterday I was visiting my wife at her now job when everyone's version of MS Word crashed within a few minutes. Since I was there I did a little digging. The IT folks had decided that to make backups easier they would store all documents on the main server, including Word's temporary files. So if the network goes down not only can you not open or save docs, but Word dies. Who do you call when this happens? No one, the IP phones run over the same network. I didn't look to see where the swap space was, I was too scared.
I used to work for Canadian Natural Resources in Calgary. Other than running >100 Oracle databases off of a single Oracle instance, the most foolhardy setup that they had was placing an on/off switch BETWEEN the server racks and the UPS. Seemed like a funny setup to me, but it wasn't very funny when the ENTIRE organization's computing infrastructure (another dumb-ass design decision) went dark when someone said 'hey, what does THIS switch do?' I'm not sure if they ever did a calculation of the cost of the lost productivity for ~1500 people for 1.5 hours was. At a minimum it is over a year's work hours for a single person. Obviously someone in charge there doesn't understand what the U in UPS means.
One memorable IT setup I've seen: In the early 90's an old NT paging software application would stop paging people over the weekend. Someone figured out that the system was hibernating due to inactivity. So on Friday nights as we left work we'd crank up a plastic, spring-loaded "micro-go-round" microwave oven turn-table, set the mouse on it (the mouse and turn-table were placed at the length of the mouse cable so the mouse couldn't move away from the PC) and released it. The turntable clicked and rotated for most of the weekend (releasing the spring tension) and kept the PC from going to sleep until Monday morning.
I've got several bad IT scenarios that I could share, but this one is the best I've personally seen.
The building my company is in was never designed to house IT infrastructure - even new newer additions are not build to house IT infrastructure. But we've made do and put infrastructure where we can, even if it means sharing a closet with the cleaning crew. One such closet had and old catalyst 2900 24 port switch in it, with the ports facing upwards. The connection are nice and hardwired into the switch, and generally looks like a rats nest (it pre-dates my employment). Recently in the renovation of the floor above it, the construction crew needed to poor a new floor. It didn't occur to them to plug the holes that had been drilled through the old floor over time, or cap the stubbed in conduit that had been put in. As a result of this the switch with it's ports facing upwards was filled with concrete. Completely and utterly filled. The cleaning crew reported that the fans made an awful noise for days. The switch never failed and is still in operation, although we can't unplug any cables or plug any new wiring into it.
At a different company, the server administrator was deeply convinced that rack mount server cases were evil and caused problem. He didn't want to use tower or desktop cases either, his reasoning? They were ugly (his words). The solution? He ordered servers, removed the motherboards from them, zip tied them into the rack via their screw-down holes, and placed the hard drives directly on top with cardboard "protecting" them from the motherboard. 5 racks of servers mounted just like this. I quite after a week of working for them, as I didn't want to be anywhere near that place when a zip-tie failed.
http://www.accelerateglobalwarming.com
Marvelously effective technique...I've actually had a facilities guy tell me he couldn't make a fix, then whisper in my ear to file a hazard report. OSHA may be a pain in the ass, but sometimes you can direct it to somebody else's ass.
rj
How about running around 30 large hubs daisy chained together and calling it a network? That is what my current employer had until 5 years ago when we finally but some cisco gear and made a real network. Because of the patchwork nature it was all but impossible to push anything out to the user pcs. Any software upgrade took days since we had to physically go to each and every machine to install by cd. Biggest pain and most bizarre setup I have come across. Not a hack, but bizarre.
WTF?
Ding. Ding. Ding. Give the man a prize! Yes, we had to run a various set of hacks to get where we needed to be. I remeber, we had to first makes sure that all drivers were loaded to upper memory with himem. This had to be done in a perticular order, because if loaded in the wrong order, they would take an extra k or two of the himem. We would also have to run some of the tools in the qemm386 package to, as you said, free the video memory. For those that don't know, there was a small amount of the upper memory reserved for text mode video. All of this had to be in just the right order, with just the right number of buffers.
A good deal of the time, we just had to accept that we could not have CD drivers loaded if we wanted to compile. Of course the guy writing the compiler, and in control of our paychecks did not actually run the compiler himself, so while he was unwilling to fix the problem, he could not actually get enough free memory to run it either. We were basically just told, figure out a way to run it.
Then we found out that the cleaning crew was using portable "backpack" vacuum cleaners. They would plug into a wall outlet that was on the same breaker as our lab. When the breaker popped, they would reset it, and continue cleaning.
At my last job, one of the sites got a nice, new network printer/copier. They were a Mac shop, but their printer didn't support Macs natively. Solution: a small parallel port print server that did AppleTalk. When you plugged it in, a phantom printer appeared on the network for five minutes. You had to "print" a specially-formatted text file to that phantom to configure the server (this was in 1999 or 2000, so browser-based UIs had not yet become widespread). When AC power was lost, the print server was set back to is defaults.
We kept getting calls from those guys that they couldn't print to their new printer. We'd investigate, find out that the server had lost its config, reconfigure it, and move on-- they didn't want to put a UPS there just for the print server, so this continued for a while. I think we even had an electrician in to check that outlet. Finally one morning we got another call, and I went over to find the AC adapter for the print server unplugged. It was then that we realized that it was the cleaning crew unplugging the stuff for their vacuum, and they had forgotten to plug it back in this time.
We posted a small sign next to the outlet that said (in English and Spanish) "Do not unplug these!" They still unplugged them. So we ended up buying a lockbox that screwed onto the outlet so the outlets could not be unplugged unless you had the key. That finally fixed the problem.
~Philly
I started as a systems admin at one place about 6 months after they had started their move into a new data center. While the new center was being built, they moved into "temporary" quarters. The "temporary" turned out to be for another 18 months. The server room was a converted office space, which was carpeted. There wasn't much room to move around in, because of the number of servers that had been shoved into the room. That doesn't even include the number of cables running around, the "temporary" wiring, the use of small UPC's to try to minimize some of the power issues. Let's not forget the large number of static pads we put in, to try to minimize the inevitable static charges.
My "favorite" hack was the clear plastic boxes we duct-taped over the power switches to two of the servers I had. That was to stop the rest of the IT staff from turning off the servers when they bumped into them - which happened about once a day until we did that. About six months after I started, I discovered a server in the room that no one had remembered. It was a print server, which had the simple job of determining which printer a given document was supposed to be sent to, and routing it accordingly. One of our vendors had made some requested changes in document formats, and we'd added in a new printer, which caused all printing to come to a screeching halt. It didn't take long to figure out that the print server was locked up. The problem was that no one seemed to know where it was. After searching through the server room for almost an hour, I happened to notice an old PC stuck way back in a corner on the floor under a desk - which turned out to be the print server.
I had worked for a company, which shall remain nameless, that insisted on using 1 Cat5 cable for 2 jacks and put dialup modem banks to use as T1 routers as they were "cheaper" than regular T1 routers and all purchased on ebay for next to nothing... Needless to say, there were nothing but problems and I made my leave as quickly as I could.
About 10 years ago I was working for the local branch of a large medical insurance company. We were going to move to a brand new 3 store office and my boss asked me to design the new network. We had about 150 PCs and a few servers. Most of the PCs would have their places at the first floor, we (IT) would be on the second and the rest of the PCs on the 3rd floor. So I went for something easy and simple: a rack with patch panels and switches on each floor and a redundant fiber backbone connecting them. Easy to keep, fix and expand.
When I presented it to my boss he said, fine but those switches you're asking (good manageable Cisco's) are quite expensive, can we do with others? I said "Yes but we are going to loose some ability to diagnose and expand, split lans, etc". New design and again my boss asked about the fiber backbone because would be expensive, etc. Well, to make a long story short, I endded up with just one rack on the second floor with el cheapo switches, all the CAT5s coming directly from the PCs to this rack so, instead of having just a few fibers from one floor to another we had a large chunk of cables. Of course that any idea of mainintance or expansion would be a nightmare. Luckly, I left this job some time later.
Scientia est Potentia
A month after installing an almost $1 million dollar server / SAN setup in our eastern office they discovered that it was directly underneath a water pipe.. of course they only discovered that when it burst, destroying all of the still new equipment overnight.
:P Reading your story reminded me of that.. ahh good times!
Our office sent them an umbrella as a gift as our "splash guard"
As a traveling school district tech, I once visited a school where the local yahoos had installed a 48 port hub under a sink in the kid's bathroom. To keep the little kiddies from pulling the patch cables out, someone had very carefully cut the release tabs off of all of the RJ45 ends.
Virtual Memory and Pagefile set to 0 MB.
"Why does it keep crashing?" - Sysadmin
Our old Cabletron hubs (the name escapes me now it was awhile ago... man I'm getting old) had the enviromental module with a nice LCD display that showed, amongst other things, temperature of the unit. The thing was quite large, larger then a Cisco 6509 we have now.
One of the times our AC died, the commisionaires that were looking at it said our temperature monitoring unit was obviously wrong since it showed the wrong temperature. I went to look at our thermastat and it was fine... then he pointed out what our "temperature monitoring unit" was.... our central cabletron hub that managed the network for like 500 users. Had a bit of trouble explaining that it's primary purpose was not just to show the damn temperature!
Probably a you had to be there, but got a few good chuckles over it.
Past few years I've worked for a few different datacenters across my state, mostly night shifts, as either a NOC Tech....or Admin. Here's a quick list of some of the monstrosities I've seen...ill try to keep the worst for last. - Bread racks for Server Racks - cat5 'spliced' together...then electrical taped....ran across a DC. - A/C and Ventilation for an entire phase of a dC by having a door open...unprotected....street level. - had a office that used alot of a old CompUSA brand switch...and when you'd plug both ends of a patch cable in to 2 of its ports....it would flood the network beyond believe and render it useless. a Contract IT guy did this occasionally to get calls and make money. I have one...'Switch of Death' displayed in my office. - (this is one of my favorites) found a dead rat in what we called 'rat's nests'. They are the unorganized bundles of dusty cable/wire found behind racks...typically in corners. - an entire 'zone' of a DC that never used racks.....servers on the floor....stacked on others...etc etc. QUALITY CONTROLLLLLLLLLLLL - we've all seen data ports in wall plates right? Well what about one that had 2 ports...and for some reason the top was made 'Live' when the cat cable came out of a hole in the wall next to it...and plugged in to the bottom port. : D ill add more as I remember. I'll come back and add more when I remember them.
The worst situation that I have seen was a wiring closet full of Cisco gear that had an air handler mounted in the ceiling, the drain plugged up and the water started raining down onto the Cisco racks on the Friday before a long weekend.
I also worked in a Bell telco office that had a water boiler type humidifier mounted in the ceiling over a #1 4W ESS AUTOVON switch (cold war era military telephone network). The humidifier tank rusted out and scalding, crusty water ran all over several racks carrying military telco traffic. The worst part was that the humidifier was over 20 years old and replacement parts were unavailable. Luckily the flood didn't cause a switch outage. Almost all of the AUTOVON #1 4W ESS's ran for thirty years with ZERO down time. There were three controllers in the switch, any two controllers could declare a third controller to be insane and cut it off.
The AT&T AUTOVON offices had WECo built switches and were located in underground buildings. The non-AT&T AUTOVON switches were made by Automatic Electric and many were located in above ground buildings. The AT&T switches passed the government EMP tests and the buildings were shock mounted heavy duty underground fallout shelters that were designed to survive a nuclear detonation that wasn't all that far away. Everything was shock mounted, even the toilets were mounted on rubber shock mounts. The non-AT&T switches usually did not pass the government EMP tests and required very high speed tape drives to reload the controller memory in the event of an EMP caused by a nuclear weapon detonation. The above ground buildings had huge sprinkler systems that were intended to wash the fallout off of the roof of the building. Luckily the nuclear resistance of any of these buildings was never tested.
I was at Carnegie Mellon University back in '98 when this happened (give or take a year). This is an unconfirmed story so I'm not sure the exact details. In the main server room for the campus there was a break room. Someone had put a toaster in there so people could eat "on the go." This one particular day someone made dark toast. When the timer ended and popped the food unit, it was out of alignment and the toast got stuck on the lip of the toaster. It kept the heating elements on and eventually it started to burn. Smoke propogated and triggered the halon system in the computer room. Thirty seconds later the room was evacuated and all the machines were automatically shut down. It took two weeks to bring all the interdependent systems back online. I don't remember exactly but it took at least 24 hours to get the email systems back up. Toaster wins.
I worked for a health clinic for a few years that had the most ridiculous setup I have ever seen: On a 533mhz server with 1gb memory and two raid 1 disk arrays we have: a windows 2000 server acting as a domain controller for a little over 50 users, running SQL server, also as an application server for the finance department software (peachtree, then MAS) also serving a proprietary data analytics program which ran a SQL scrub operation nightly on about 2.5 GB data; exchange 2000 server with a third party POP collector (I still don't get what those are for); as a file server and to top it off it was licensed as a small business and back office server, so it was also limited to 50 users (yeah, I needed to learn to count my users... duh) I had a routine that had me going into work at least two hours before anyone else was there in order to do a system reset because one of the applications that interacted with SQL had a flaw that would not release the memory it took up in its night operation without a reboot. This system lived in a closet with an NT server running a single program that operated in a DOS shell and ran velocis database software and a SQL server to provide data to the application server. Also in the closet was a little remote service box (a networked computer running PC anywhere over a modem). The server closet had a slotted door for ventilation, and a small fan to draw air into it. Needless to say that the ambient room temperature was always above 80F. So I show up, the only man on staff, and on a $50,000 grant I completely renovate their small little network into an innovative gigabit modern data crunching machine complete with money saving devices like managed switches, thin clients, open source network service management (nessus, squid, mail gateway, firewall/port forwards, spam assasin)... and then I got outsourced to a 'consulting firm' for the expressed intent of saving money... and has no *nix support, experience or knowledge. I'm sure they'll figure out that was not a wise move. Disgruntled? Maybe a little, but I took the high road and gave them explicit instructions to change all their passwords upon my exit. They never did
I was working for an IT consulting company and my contract with a client had just ended. They had no new contracts available so they offered me 2 weeks severance as long as I agreed to hang out at their main office and be the office computer guy for a few hours a day. One of the first things they had me do was to look at a PC in a conference room that they said they thought had a virus. So I fired it up and found Sub 7, back orifice, some sort of IRC bot and numerous other trojans running. For whatever reason I pulled up ipconfig and saw that the IP address of this computer was a 4.x.x.x which I immediately recognized as a Verizon IP address. Mind you this was plugged into a wall jack in the building. So I ran across the hall to the PC on my desk and sure enough it had a 4.x.x.x IP. I checked about 3 other computers and they did as well. I went back into the room that had their networking equipment and saw that they had plugged a Fujitsu DSL "modem" directly into their switch in the office and it was handing out IPs to the computers on their network. I told the VP out there that their entire network, servers and printers were publicly accessible from the internet and they should assume their entire network had been compromised at this point. I asked him why they (meaning him since he set it up) did this and he basically said "Well....It worked..." So I had them pick up a cheapo Linksys router and threw it in place and told them to buy updated copies of Norton Antivirus and load it on all their computers. By that time, my time was up and I left them to their own devices.
The next place I worked as was as an on-call part time network admin. They would just call me at home whenever they needed something done. Their main file/print/exchange server was fairly decent looking but inside was a different story. It had a 1.2GB IDE drive, 20GB drive and a 10GB drive combined into a volume set, and then that volume set was partitioned into 3 equally sized drives. Most of their exchange stuff was spread across those 3 partitions. The file data was on a 36GB IBM 10k drive. My first order of business was to get rid of that 1.2Gb drive that must have been inherited from another computer. Luckily they had a unopened 40GB drive floating around that I used to move all data to and if they didn't need it for another purpose I was going to get rid of all the drives in the volume set, but sadly they needed it for another computer so I broke the volume set and removed the 1.2GB drive and then created single partitions on each drive and redistributed the exchange data. They had recently purchased a MaxAttach NAS box which I configured to make differential backups of the server. Other things I had to deal with were the 5$ network card in the server failing and dumping garbage onto the network (confirmed by Ethereal). I told the head guy to buy an Intel Pro/100 card to replace it. He went down to Fry's and came back with a 10$ Realtek card that the guy told him was "just as good". That card lasted about a week and started having autonegotiate problems with the switch. I brought it back up to the head guy there that he went against my recommendation. So he took it back and got the Intel Pro/100 card that I had asked for initially and the network performance nearly doubled. A few weeks later the server went down again this time the 10K drive had overheated and started filling the event log with disk errors. I told him to buy a pair of 10K 36GB drives so I could set up mirroring as well as 2 front mounted drive fan kits. After that the server was running like a champ. I was going to throw out the old 36Gb drive but I took it home to use as a place to temporarily hold data. I ran IBM diagnostics on it and it completed a wipe and was able to remap the bad sectors and worked great as the main drive in my old SGI Indy. There were some other things I had to do there as well such as convert their 486 server with a 540MB drive that ran Novel 3.x to a new NT server after the drive failed. This server held all the data from their Accp
10+ years ago, I used to work for a Networking Software company called Artisoft in their Tech Support office. There were two notable hacks that come to mind. The first one I only heard about (file this one under Urban Legend if you please), but someone needed to network their farm house to their barn. Their solution: Barbed wire that was twisted just enough to simulate CAT 3 wiring. Worked ok, except for the dropout when the cows would lean against the fence.... The other one was a case I worked on. Gent in Florida or Louisiana had lost conectivity between two machines on a coaxial network. Now, as 80% of existing network failure can be attributed to cabling, I had him check the span.... He had to look under the house (the structure turned out to be on stilts in the swamps) and lo and behold the cable was broken. By an alligator/crocodile living under the house.... I wished him luck and closed the case....
...'nuff said. Every day is a non-stop WTF? and head scratching experience. When I was hired, the VP of IT left due to the fact that he knew I actually knew what I was doing and it was only a matter of time before he was found out. (he had been there over 8 years) Then the promotion to his spot went to someone who has zero computer experience, and a B.S. degree in HR. She was without a PC for over a week due to a hard drive crash and I was on vacation... this is the manager of IT.
You can imagine the lovely surprises I still find to this day some 2 years later.
* Who needs switches, hubs are cheaper... but why is the bandwidth so crappy?
* We are going to run 28 VOIP phones on a single 64k Frame relay that also supplies internet and connectivity... what? Thats a problem? make it happen!
* I had over 6 in person meeting to explain that we could upgrade from 64k to full T1's for a *third* of what we were paying for 64k... YES we save money and get 20+ times the speed... NO no extra costs... denied. another meeting... denied. another meeting, etc. 6 times until it was still begrudgingly accepted!
* We have fiber links! (after seeing speeds around 2mbps and investigating) No, we have fiber connecting the buildings via 10mbps ports on hubs.
* We don't have that T1. Yes we do. No we don't. In a basement, *behind* the current wooden panel that houses the current telco equipment, sure enough an old T1 channel bank that still was working. How could I have missed it?
* We need 4 custom servers, a rack, a switch, 60 patch cords of different lengths, 4 PC's, and some cable management ordered. OK. We need it in 4 days. I don't think so. They are custom servers... 2 weeks later I get it all (still damn near a miracle) and have it up and running only to get shit about how long it took and told that I should have shipped it overnight next time (which I did of course). They must have missed the part where building custom servers takes a vendor more than 10 minutes let alone 4. Yeah, the shipping held it up.
* APC units that are installed, and working, but everything plugged into the outlet next to it.
* And best of all... our annual IT budget? $500.00 yep. 500 bucks.
http://teasphere.wordpress.com - A little spot of tea
I worked for a company that did robotic inspections of heat exchangers. We had portable data acquisition units with metal cases and of course the metal case was connected to the ground wire on the three pronged plug. We kept blowing power supplies and communication interfaces on those boxes. One day we decided to check the metal grating the boxes were sitting on inside the reactor building with a volt meter. It measured over 120 volts to ground. So we clipped the ground wires on everything and wrapped the boxes in plastic and wouldn't you know it no more problems.
Telecommuting! What about socialization?
Its well known that freezing or cooling failing hard drives can get them working, usually temporariy. At one place I worked we had a PC that was near permanent fixture in the freezer. The company was so cheap that they wouldn't replace it, but they also couldn't lose the data! Gah! Regardless, it was very common to walk into the breakroom, see a nest of cables coming from the freezer and see a developer using a keyboard, mouse and monitor on a table.
At a web hosting company I used to work for in Orlando, our main datacenter was in Fremont, CA, and we had servers hosted in a few other datacenters around the country, but we also had a number of older servers hosted in our Orlando office, a small office/industrial building in an industrial park. When I first started working there, the "server room" there was a tiny supply closet with wire bookshelves holding the servers. Cooling was provided by going into the ceiling, ripping the flexible AC ductwork off of other outlets, and dropping it into the closet.
At one point, we got rid of one of our data centers and moved several dozen old BSDi web servers from it to the Orlando office. There was no room for them in our server closet, so we stuck them in an uncooled storage area in the warehouse section in back. Now, this was in Florida, so that room would routinely be well over 100 degrees during the summer. We usually had to reboot a dozen of those servers a day when they'd overheat. Sometimes they wouldn't recognize the hot-swappable drives upon rebooting. When that happened, we'd just yank the drives out and throw them in the break room freezer for five minutes, then plug them back in.
At some point, they moved the server "rack" (a wire shelf with wheels) out to the garage area and left the garage door open to provide some ventilation. Of course, this meant that there was a rack of computers on wheels, not secured in any manner, sitting inside an open garage in the middle of an industrial ghetto (our cars were regularly broken into while parked at the office). I'm amazed no one simply walked in and wheeled them away.
Despite all that, those servers continued to run just fine and almost never had any hardware failures. They eventually built a real server room with its own climate control and moved everything there, and those things kept right on going up until the time the company that bought them out closed that office.
I had a rookie desktop support guy working for me in a small San Jose office whom i though could handle the cabling for a new furniture purchase. After he quit for greener pastures i had to pick up the support role myself and to my surprise discovered that he wired the entire office in Cross-over cables. From the patch panel to the switch then at the users desk another crossover cable from the wall to the computer. It all worked if you stuck to crossover cables but was a real pain to work with...
In my small server room that I inherited, I was sitting working one day at the terminal when I heard a drip-drip-drip. I looked up to see a warped ceiling panel and water dripping down.
I was a little terrified but I got up on a ladder, removed the ceiling tile, and found the evaporation bucket the size of the server room itself. Some fool had put the drain for this large (200 Gallon) tray at the lip of the tray instead of the bottom. Thus 200 gallons, a 'youre fired' amount of water was hanging over my servers at all times.
It took myself and another engineer hours to siphon all of the water out into large plastic tanks using a garden hose. That's when the building maintenance people hit me with this gem: "We can't fix it. But we'll put an alarm up there to go and empty it when its full". A ladder and broom to tap the bucket are now standard issue for that room. Lame...
But you got out right? I'm in university IT hell right now... rogue administrative departments deploy their own solutions, central IT doesn't have any clue (why yes, we are still running NT4 systems. It's what the sysadmins are comfortable with.), and there's always money for new capital expenses but not salaries or training.
Just gotta hang on for a few (12 is a "few" right?) more months... Please tell me that you got out to a job that has proper hardware support and requires specifications for projects before they begin.
"temperature in summer goes over 38 C (about 95F) and the machine (Sun inside) generally freezes after two hours of operation"
The Sun freezes because it gets too hot?
Many moons ago I...er...I mean "a friend" worked at Cabletron in Durham, NH. There was a glass walled server room that had a badge lock, and all the 'important' server machines went in there. After a couple of power blinkages, someone got the go-ahead to install a massive diesel UPS unit outside the building. There were a few (noisy) test firings of the diesel, and all seemed to be working as planned.
Then we had an ice storm which caused a power outage one weekend...
The diesel fired up...the power flowed...and all the servers in the server room stayed up. Two problems though:
1) The air conditioning was not on the UPS. The machines continued to run...without A/C...for hours. The machines started overheating and crashing. Some poor schmuck IT guy drove in from home...through the ice storm...to investigate why he couldn't contact one of the servers. There he found:
2) The badge-access door lock system was not on the UPS either. He managed to get into the building somehow, but when he got to the server room and discovered the A/C problem, he couldn't open the server room door to let some cooler air in and/or shut down the machines gracefully. Without a key to the lock, he wasn't getting in. He sat and waited for the key to arrive, as more systems baked and died.
Several machines and disks were lost in that exercise...and the UPS was rewired shortly afterwards. A liquid crystal thermometer got taped up on the inside of one of the windows too.
Most sites seem to use it because they're addicted to the calendaring, but there were better apps for that before that were cross platform (unlike that one). And there are better calendaring apps, now too, just different from a few years ago. Otherwise, it'd be gone in a heartbeat and replaced with a real mail transfer agent.
... AFAICT it's fine for use as a groupware serverI used to believe that, in absence of other data, until I had more and more contact with people actually using it. Even people on the same physical server. It'd be funny to see if it weren't such a sabotage of productivity. Losing an important mail is not just losing 5 minutes: there's waiting for the important mail, figuring out that it really is gone and not at the destination, the phone calls, the stress, calming from the stress and then figuring out what you were doing before you figured out the error and then getting back to where you left off.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
I once had a 486 Linux box I set up for myself and roommates to use. We had a couple spare serial terminals, so I put two serial cards in the machine so I'd have enough serial ports to cover a mouse, two terminals and a modem. With the serial cards came two printer ports, only one of which I needed. IRQs started to become a problem. Fortunately, one the I/O cards was flexible enough it allowed assigning IRQs 5 and 7 to the COM ports. These IRQs were normally used by the printer port.
Later, I added an NE2K clone card to the machine along with a sound card. One of the serial terminals was retired and replaced with my roommate's PC connected by Ethernet. The sound card insisted on IRQ 5, and the NE2K card only allowed selecting between IRQ 2 and IRQ 5. IRQ 2 was already taken--it's the "cascade" interrupt that IRQ 9 and up map to--leaving me with only IRQ 5. (I forget what else I had munching IRQs. I believe my Adaptec 2740 ate one of the upper IRQs, which would explain why I couldn't use the cascade interrupt.) What to do, what to do...
I took a jumper wire intended to hook a CD-ROM drive to the sound card (as I didn't have a CD-ROM drive at the time), and used it to jump the IRQ 7 post on one of the I/O boards to the IRQ select line on the NE2K card. Voila! The NE2K was now on IRQ 7.
I ran that way for a good year or so until I could retire the extra serial ports. I didn't retire the machine for another few years.
--JoeProgram Intellivision!
I use to work for one of the biggest banks in my country. Datacenter was on the 5th floor, and suposedly the room was a bunker, concrete and steel walls. Access door was like a vault. Had to pass 3 doors, buzzed open with a smartcard. All servers containing customer accounts, money, credit card data.. everything a bank manages where in here. One day the bank execs where bragging about the security of the datacenter, that it would stand an airplane crashing in to it and blah blah.. The next week a large Fishtank they had on the 6th floor offices broke, spilling gallons and gallons of water (not to say unlucky goldfish) ...Water leaked down, all hell broke loose in the "bunker" it was literally raining inside. Had to put all plastic covers over servers, open all vault doors, shut down the bank's operation to clean up and save servers without loosing data. Execs didn't comment about the incident.
Oh, and I should point out that while the printer ports *could* use IRQs, the Linux printer driver at the time operated the ports in polling mode, so thankfully they didn't need to eat up my precious IRQs. I didn't connect either to an IRQ. The fact that the printer ports, when run in interrupt driven mode, used IRQs 5 and 7 is the reason the boards even cared to think about these IRQs.
Program Intellivision!
OK, maybe this isn't bizarre, but it's not something you see every day. When I joined my current school, most of the computers were from a single company, except about ~250. Most had XP Pro, except the 250, which were Windows 98 First Edition. The network (had about ~1250 computers at that time) worked. Mostly. Then, all at once, they installed service pack two on some of the computers but not others (god knows why), installed XP Pro onto the Windows 98 computers (without updating the hardware), and converted all of the ~100 laptops from wired to wireless. Aswell as that, they upgraded some of the servers to the latest OS. Needless to say, this was a major series of upgrades. To make matters worse, none of the equipment was from the same manufacturer. I know that shouldn't be a problem, but it certainly doesn't help matters. The network ground to a halt. If you could get logged on, you were lucky. The internet was virtually non-existant, and the intranet was slow at best. Printouts had to be copied out manually, or printed at home. Now, all of the formerly Windows 98 computers have been replaced by new(er) ones, with TFT monitors. Thank god. Most of the problems have been fixed, but during those few weeks when virtually nothing at all worked, you had to wonder what was going on with the admin staff.
The best story I can come up with is this. At the private school I'm a techie at, they don't have the resources to get Windows Server proper, so I had to set up Windows 2000 as a server for Windows 98 clients. It wasn't so bad, as the Windows 98 would look to the Windows 2000. But then all the machines were upgraded to Windows 2000, and so I have a small network (10 computers) with Windows 2000 clients looking to a Windows 2000 (standard) server. So if I want to change anything about a user, I have to change it on each computer, and Active Directory doesn't, of course, exist, so many things don't work. But the server is working as a file server with separate folders for each user.
You all have Oo.o and Firefox, so get World Wind.
I was doing some consulting a long time ago, and went into a little publishing company to set up a new computer.
I was checking out the network (AppleTalk), and noticed the print server.
What print server? I asked them about it, and they didn't know they had one. So I traced the cables, and found (on the other side of a partition) a door with cables running underneath it. So I moved the partition, opened the door, and found an original Macintosh II, plugged into some ancient UPS, chugging away, with (literally) a half inch of dust on top of it. It had been sitting in the room, doing nothing but print server stuff, for over seven years (I checked - no reboots - seven years of continuous uptime).
It had been in the closet two years before anyone who was currently working there had been hired...
...There is esmtpverbs control parameter in AD which by default does not play very nice with some other smtp servers(its arguable whose fault is that - every party claims its other party to blame) .Easy enough to deduce: On the one side we have all the other MTAs which, unsurprisingly, are able to communicate well with each other. That leaves, on the other side, one system the odd man out which does not play very nice with the others.
What's wrong with admitting that it's still broken or don't normal rules of any kind apply to that one vendor ?
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
A company I previously worked for had employees on the road use free dial-up Juno internet access to send company mail. That's right, pop-up supported and all. I guess they didn't understand that another 3 bucks a night to stay in a hotel with free wireless high speed would have been a better investment.
Yes it's an anecdote! Were you expecting original research in a Slashdot comment?
A few interetsing tales: The miracle network: A customer complains that a couple of their machines are very slow on the network. Even internet (via cable modem patched into the network) is slow for those machine. I go over there and start checking things... both machines are generating packet loss to anywhere else on the network, and the network drops which run all the way across the building are direct runs. No wall jacks or anything. I grab our cable tester and hook up to the first cable ends here and in the wireing closet (everything is labeled). It shows no connection. None. I plug it back in and realize I get no light on the switch for that jack. Mark down everything, and do the 'connect-disconnect' light shuffle... the switches lights don't change. Property owner assures me that the only network equipment is in this room, and the runs are all straight, no additional switches. We begin tracing the cables through his dropped ceilings, all nicely zip tied together and of course all the same color cables. We find nothing out of the ordinary, and even peer down the wall space where the drop comes down with a flashlight... maybe there is a hub or switch? Get back in and trace the cable which runs into a hole drilled into a desk, out another hole in the desk, under a several hundered pound file drawer... what's this.. 3 FEET of cable covered by what looks like a whole roll of black electrical tape. So figuring we have a bad splice, we trim that out, use some punchdown splice blocks we had to add in a segment to replace the taped mess, and bingo, it all works. The punch line? After pulling all that tape off, one of the other techs discover the splice was made by stripping the outer jacket off of 2 feet on each end of each cable, and wraping the still insulated wires together. All their network connectivitiy for that computer was by inductive coupling. Computer 2 had a similar problem, having also been moved. There however they were a bit more professional. They went out and purchased a 10' cat5 patch cable, and sheared most of the plastic off the connector with a knife, then taped the two connectors together pin to pin. The exploding server: We work on a lot of machines. A server comes in from someone who is not normally a customer, but needs the machine back pronto as it has all their customer data on it. They shut it off last night and now it won't power on. The tech wipes it off with a dry cloth before putting it on the server desk because it is filthy with crud. Plugs it up, and turns it on about the time I'm going back there to work on another machine we have back there. It looked like something out of a hollywood film. Sparks flew out of the power supply, things snapped, awful smells came forth, then the circuit breaker on the UPS tripped. The server came from a machine shop and had been out on the floor. The entire machine, inside and out, was covered with dust sized flecks of metal. When they brought it here, the metal shifted around and formed shorts when we powered it on. They also needed it back up and running within 3-5 hours. Their hard drive was fortunatly not a casualty, and we were able to move it to another box. You give us 5% packet loss like everyone else, we go elsewhere again: Local business has two offices. One served by the cable company, one for 6 months by us via dsl (cable company who set up network in first place doesn't reach them). They have a business app that is not networked, they simply run a VNC session from the remote office. VNC appears to be unhappy with 5% packet loss. Customer came to us because their previous provider was handing them constant 5% packet loss on their DSL circuit. They assured us it was the provider's fault and not hardware or the other end, and with the standard disclaimers (the phone company is notoriously reluctant to fix dsl lines that work 'mostly right' around here) we set up their internet. Less than a week later they call, VNC won't stay connected. So I start nosing around from their router. Router-> us, ping for 24 hours, o
One of the stranger ones that i've seen wasone building where certain computers were getting sporatic connections. After following many many meters of cabling i came to notice that in one room where all the cables went through, the tech that had installed it was tieing knots in the cables to identify what cable went where. 1 knot, computer 1, 2 knots, computer 2, etc. Needless to say pretty much all the computers over #5 had some issues, and all the cable had to be replaced. And at the original technicians expense, of course.
I remember many years ago, when my buddy and I both owned Tandy Color Computers and 300 baud external modems. His modem was some weird Hayes-compatible model with the serial cable permanently attached to it, and it died. He borrowed my modem to get online, but I forgot to bring my serial cable - and it was too late at night to go buy one.
We ended up soldering wires to partially straightened paper-clips, and sticking the straightened ends of the paperclips in the proper holes on the RS-232C ports to serve as our data cable!
Fixing it would have cost time and money to have it done the correct way only to serve 1 or 2 machines that would probably be replaced in a couple of weeks anyway (everyone else in the building had upgraded to the "new" ethernet connections except for the holdouts). One of our network guys climbed a nearby tree, took some rope and tied it to a convenient tree branch then went across the street and was again roped to a convenient light pole, angled down to the top of a fence, along the fence on the ground and finally into the building. Some years later I noticed that someone had moved the now long defunct cable to run along the top of the fence via zip-ties (groundskeepers?). The rope (amazingly) was still holding the line to the tree branch.
-One computer room in a pork plant had its air intake on the roof... right next to the exhaust from the rendering exhaust. Now, while the smell itself was awful, the worst part was that the sulphur and other noxious chemicals would eat the computers alive. The IT group had to install a special device that monitored the decay levels of samples of metals such as copper, silver, and gold to see how long the machines would last.
-One computer room was made out of a new office on the third floor of one plant. But there were no elevators to get the hardware up into the room. So they cut a hole in the floor, used a gigantic crane, and lifted two guys with each fully-populated rack up to the room. And the crane was still three inches short, so the two guys would have to do a wheelie with 1000+ lbs. of equipment to get it in the room.
-The same plant with the hole in the floor was also keen on bringing in electricians who were severely brain damaged. How bad? Imagine a 110V cord strung across the air with no support. And the other end of the cord with the three prongs was "hot". And someone once plugged it into the wall socket. I hear it doubles as a cattle prod now.
-Two computer rooms needed air conditioning, so they simply carved holes in the walls and hung out Wal-Mart brand home air conditioners. With no other insulation. In the winter sometimes frost appeared on the machines.
-Many of the computer functions were relegated to plant employees with their own unique vocabulary. I had the privilege of speaking with one woman in Texas who sounded exactly like Boomhauer on "King of The Hill." Honest. Another one always tried one-upping the "smart" IT guys back at corporate. On one occasion, she said that her monitor was "tricating". This was the word she used to mean that the emulation was off. I informed her that there is no such word as "tricating," at which point she told me that since I was young I likely hadn't heard of that word before. Funny, neither had Merriam-Webster.
--Chag
I'm currently the webmaster/IT person in a very small company (about 10 people maximum in the office). All the computers in the office, and our server, are wired up via 10BaseT to a 3Com rackmounted hub in a closet where our phone line and PBX are. We have DSL service to the office, and it works alright, but a faster connection would be nice.
The most interesting part of our network is the Netgear Cable/DSL router that connects our entire network to the DSL modem. At least once a day, and more often upwards of twice, the router appears to stop routing traffic -- nobody in the office can use any internet services until the router is hard reset.
After a blackout a few months ago, the router shut itself off. Checking the power connections, it seems the former IT person had plugged the router into a mechanical timer, set to turn it off between 3:00-3:30AM, but the blackout had caused the timer to stop keeping time just long enough to cause the switch to power down our network during business hours.
I assume they thought resetting the router overnight could alleviate the problem? I've been after the company to purchase a new router, but since this still works it's not going to happen anytime soon.
My first PC (a 286/12 IIRC) had dual 1.2MB floppy drives, no hard drive. Yet I got almost everything to run that way, even if it offcially required a hard drive.
A later PC I had was a mini-tower, but I had 2 hand-me-down full-height hard drives I wanted to put in it. My solution was to stack the drives on the table beside the case and run the power and SCSI cables out the back of the case. Worked for years that way.
I've also been known to use a cardboard box that used to contain 10 5 1/4" floppies to put a 3.5 inch drive into a 5.25 inch bay.
We ran our company for years on a ~35 mile custom wireless Internet link. Our end had a ~4 foot dish on a 100 foot tower, the other end was a taller tower on a moutain.
We burn CDs 4 at a time at work, but I don't always hear the CDs when they are done and pop out of the drives. So I just prop a empty jewel case up in front of the lowest drive, it gets knocked over when the CD is ejected. Time to put in a new batch!
1. A "server room" that was essentially the most worthless room in the entire building, a long skinny room with four windows (perfect for keeping an uneven temperature!). Rather than buy 19" racks or even wire racks, they found a bunch of tables and put one server on each all the way around the edge of the room.
1.a. All of the servers were in fact desktop systems; an Ultra 1 was the mail server, a SPARCstation 5 the print server, a Gateway Pentium Pro 200 desktop the web server, etc.
Damn, dude!!!!!! Did you work with me at the FlashNet ISP in Ft. Worth in the late 1990's?
Because that describes EXACTLY the hardware situation there, down to the brands and models.... except the "desktop" web server was not a Pentium Pro 200, it was a Pentium Pro 166...but it did run FreeBSD extremely well and had a 7200 rpm SCSI hard drive in it.
Not a huge hack, but when I was working on my first network management system (windows 3.1) I went on a trip to see how it was being used by some customers. One of them (The city of Olympia) had required that we have the ability to send each alarm to a serial port--when I got there I found out why.
.wav files upon an alarm, but since nobody had speakers back then (windows 3.0/early 3.1 timeframe) I bet they ran that printer until the 2K bug we had in there made them find a new solution.
They had a very old, slow serial dot-matrix line printer hooked up to the management system. Each alarm set it off with a loud Bzzzzzzzz so they could hear it if they were in another portion of the (loud) shop. The hard-record of alarms was secondary.
After that I implemented a system to allow playing of
I had an ancient computer when I first got on the 'net (8Mhz 8088). It ran DOS, and had all of the wonderful problems associated with running out of "conventional" RAM. I had a modem and some 'net experience gained through shell accounts on some local BBSes. Trying to coax a PPP connection onto that machine was a hack-job. Since DOS wasn't designed as a network-capable OS, pretty much any attempt to route IP packets to the modem required a network stack, 64Kb of conventional memory down the tubes. So much for running DOSLynx or anything that required any sort of RAM.
Novell seems to be involved in a lot of these old-timer stories, and it is here too. It turns out that Novell had a great packet driver available for DOS -- but it was for an IPX network. Someone released an interesting TSR which used the Novell IPX packet driver to transfer TCP/IP. It saved an enormous 40Kb of RAM, allowing all of those apps that wouldn't start before to run. I'm sure that the shortened stack had all kinds of vulnerabilities, but thankfully there just weren't enough people around to exploit them.
mandelbr0t
"Please describe the scientific nature of the 'whammy'" - Agent Scully
A company which shall remain nameless currently has half of the building (i.e. 7 workstations and 2 printers) jerry-rigged together with three workgroup hubs connected to a single drop. For good measure, the cables used are all at least 25 feet too long and wound under desks and along walk ways.
Oh you mean like the snarl of thicknet behind a rack that wasn't connected to anything but if you touched it at all half of the campus would go down?
:)
Or do you mean a pentium 90 running LRP handling all the routing for the QA lab of a major software vendor?
Or the drip pans over the server racks to catch the run-off from the swamp coolers (which were just as good as air conditioning and a lot cheaper).
The list just goes on and on
Hopefully this is buried enough that none of my former coworkers will find it ...
... ehehehe).
Around 10 years ago, I wrote a system to do online registration for some continuing education courses at my university. It was a series of horrible, horrible Perl scripts that kept its data in a dbm hash until it was manually entered into Paradox by the conference registration staff. This worked splendidly for several months until it was decided, for efficiency's sake, that we should upload the data directly into a Paradox holding table. Good idea, but Paradox ran on our Novell network while our web server was an SGI running IRIX.
We batted around a few ideas until it hit me - we could make it work by sticking a PC in between the two. The first pass at it had the SGI box FTPing completed registration records to the Novell server. Once there, "Reggie", the PC, had a special Paradox script that polled the FTP directory, sucked in any new data and wrote it out to the holding table in the master registration database.
As far as I know, that system is still in place (at least judging by the IM I got from a fellow a couple of years ago that started out "YOU ARE THE AUTHOR OF MY PAIN!"
James
During the 1994 monsoon I was consulting to a large manufacturer in Taiwan. They were having problems with the control systems losing network connectivity when they started rolling steel. Followed the cable and it was draped over the 4kV busbars running to the motors. Needless to say we were all standing in knee-high water at the time. I exited slowly.....
Our site had 2 groups managing computers. My group supported the R&D group on Unix boxes. No PCs. The other group was IT supporting PCs. And a 3rd guy doing the networking stuff.
The R&D group developed network hardware. So naturally our network consisted of anything that could be scrounged from prototypes and refurbished systems returned from customers. Lots of blue wire. Some of the engineers would plug their prototype into the network to see what it would do with real traffic.....
Our group was originally 2 guys that we took over from as they went back to engineering. They had external SCSI drives with user home directories attached to people's desktops. And 3 NetApps in the server room that "were full". 25% of the storage on the NetApps was taken up by copies of the free Sun CDs with samples of vendor software.
The IT group never asked for budget. They didn't know they had to. They bought computers from the consumer part of gateway/compaq/etc. We regularly used thier budget and they never knew until one of our guys was made the boss.
100T was just coming in. We had just shipped our switch so the prototypes were available to our net. We converted the unix servers to it. Then we asked the PC guys. The building was all cat 3 so we had to run cat 5 to their server room. 2 months later, I went to their server room because users were complaining the Notes server was slow and the PC guys were out. I found 5 of the servers connected into a 10T hub that uplinked via the old cat3 wire to the network closet. All of the new (expensive) cat5 jacks were unused. We had the servers hooked up to them last month.
Finally, the we gave the PC guys a tape stacker to automate backups. After 6 months we found out he was coming in every Saturday to swap the tape (the backup took 2 tapes). He never setup the tape backup software we gave him. You couldn't call him lazy!
At a large Telco in Canada, a rural town's CO was renovated with new siding. All was well and good until a large storm took the power out. The backup generator for the site was a diesel turbine. Turns out, the new siding was not properly insulated around the generator's exhaust. The building caught fire. The worst part was, the generator could not be shut down for long enough to properly fix the problem without the phones going down. I think the building caught fire 3 times before the power came back on and repairs made.
Or sometimes an "S-trap" depending on the application. You are correct in its designed purpose, but incorrect in its maintenance.
There shouldn't really be a large problem with evaporation provided that there is proper building pressurization/envelope control (remember, changes in building pressure will affect evaporation) and a working Trap Primer system (usually required by Plumbing Code). It is a misconception that floor drains should just "dry out" if the facility is being maintained and was constructed properly. A trap primer operating intermittently to "re-wet" a trap uses a lot less water than even a small trickle from a faucet.
In short, if you're smelling that nasty odor, there's something wrong that requires a fix beyond letting a tap drip, as this merely treats the symptom and not the disease.
I once knew a PHB at a large financial services company that brought in Microsoft consultants to re-write the GINA.
This was so a user could press a single key after system startup to present Windows logon credentials instead of the default CTRL+ALT+DEL.
In his yearly "state of the company" bulletin he proudly proclaimed that he tripled worker productivity. I know this sounds like Homer realizing he could press "Y" instead of "YES", but this really happened. Sad but true.
A company I worked for just bought the entire 5th floor of a new building, although we only occupied half of it. When we moved in, the management insisted that we put all our servers in the "server room". Everyone knew it was the "server room" because it had a picture of a computer on a plate on the outside of the door.
The room was somewhat adequate space-wise although the shape of the room really made it interesting to get anything installed. What really got me were the wet pipe fire sprinklers. When I mentioned that water + expensive servers = BAD, their solution was to hang plastic sheeting between the servers and sprinklers.
In the quest for Internet service and TV in our barracks area in Iraq (a tent, mind you), we did silly, silly things.
- Cat 5e running out in all directions from my tent - some partially buried in gravel/sand, some over roads, etc. At 1000ft per box, I must have strung nearly a mile of cat5e over there. (FYI, cat5e will survive a run over a moderately trafficked gravel road for about a month and a half.)
- Three different Wi-Fi access points of different models / configurations. Having no external antennas for the "primary" access point, improved signal and range was accomplished by shimmying up to the top of a tent pole and mounting the access point to a platform hung from the pole. The whole mess was then covered up with a cardboard box with vent holes cut in the side to try and keep dust from clogging the unit.
- Tons of splitters and hundreds of feet of network cable to share the few TV antennas available. Starting at the cable in my friend's trailer, I once attempted to determine which antenna he was actually hooked up to. A complete loop and a half around the entire housing complex later, I still hadn't found the actual antenna.
three comments to offer:
1) Another option is to reduce the 3 PCs to 1 PC running multi-head multi-user:
ex. http://linuxgazette.net/124/smith.html if you NEED windows then run inside of free vmware server.
If you don't want a DIY solution, there's a commercial version at http://userful.com/
2) I'm guessing your fans blow in since you said the cable hole is at the top. The cable hole should be more than just and inch or so in diameter and probably atleast the size of one of the fan holes.
3) to reduce the fan noise, mount the fans inside the box and run curved(noise baffle) ducting to the holes.
LoB
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
The company I work for (remaining nameless) recently switched over from Oracle to Datasweep - NO BACKUP. Productivity went from 1,000+ units a day to ~700 units a day, and with no way to switch back to their original database. Needless to say, my first thought was the whole IT department was literally nothing but one grand hack-job!
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
If you aren't talking out your ass and you actually have more bathrooms in your house than you use in a month, than you should seriously be thinking about retrofitting your plumbing since it is not properly constructed for your dwelling habits.
If a bathroom fixture is going to often not be in use for a long period of time, the correct solution IS NOT "oh drip water in it all the time", it is to install a proper trap primer. This will use much less water than a dripping tap and ensure the trap is sealed.
There are purists who will always say $OS is the One True Way. I've found more often than not, that when coming into a shop where one person is responsible for a given piece of the System without any real oversight, what happens is that they will sometimes do some really weird shit. BTW 65 to 85 F is a REALLY big deal as that's the air temp in the place, think about the volume of the room and how much energy it takes to to heat a space like that in such a short period of time. You'll fry some stuff in a server room at those temps. If they'd let it go for much longer they'd have fried some hardware without fail.
"Chinese Amazons, power armor, laser swords.... things just meant to be." - Shampoo, A Very Scary Bet
He comes in and saved do-it-them-selfers from themselves. And there was one couple who hired a "contractor" to remodel their basement. The electrical wiring job Holmes found almost made everyone gape in amazement the house wasn't a pile of cinders. And people wonder why I insist on hiring people to do work I feel I'm not competent enough to do, and don't mind paying more than that lovely man who advertised in the Weekly Shopper who only wants cash, no cheques please and who changes his phone # every couple of weeks.
The bitter lessons of a veteran coder: http://bitterprogrammer.blogspot.com
I work at Intel, and I saw an email thread once for one of our datacenters in Asia (Malaysia, I think). There was an Emergency Response Process initiated. The cause? Snake in the datacenter. Not just any snake. A cobra. Apparently, it had crawled in through an open (?!) window. They evacuated the place, and called the local snake control guy to come and catch it. It was released outside. I don't know if I'm more bothered by the fact that a cobra got into the datacenter, or the fact that they just released it outside. I'd be worried walking to my car...
At an ISP where I used to work, management decided we wanted to start offering managed networks for business customers. The demark between what we managed and what was the customers' problem was a Cisco Pix router/firewall, which we supplied. One night while I was on shift, I noticed an alarm on one of our customer's networks. SSH to the Pix didn't respond, so I dialed into the OOB modem to find the Pix perpetually rebooting. I dispatched one of our field techs to investigate, and what he found suprised even this jaded sys admin:
The customer had mounted the Pix in a corner of the hottest room in the building (no AC). Unfortunately, there was a mouse problem in the building, so the customer had placed the Pix in metal box to keep the mice out. Unfortunately, while the box was very good at keeping cooling air from reaching the Pix, somehow the mice were apparently able to get in, based upon the quantity of mouse poop and urine stains covering the Pix.
While I'm not a huge fan of Cisco, they certainly deserve kudos for the fact that Pix worked *at all* IMHO.
MCSE? No, sir...I don't do Windows. Yes, I am an idealist. What's your point?
Slashdot: Where anecdotes and generalizations can be freely substituted for facts, logic, or intelligence
Slashdot: Where anecdotes and generalizations can be freely substituted for facts, logic, or intelligence
My former employer was an ISP service, we were in the 8th floor of a building. We ran a fiber cable out our server room, through the drop ceiling into the elevator shaft, down the shaft and to the 2nd floor, drilled a hole in the wall ran the cable through accross an alley way ~ 16-20foot off the ground accross to the AT&T building next door for our "Primary" internet connection directly connected into the Local HUB, "we" did this on a Saturday. Best internet connection I have ever seen.... Nothing like P2P'in a complete 650mb ISO in less than 5 seconds.
The longest "lost server" tale I ever heard:
Seems someone had stuck the company file server, running Netware v2, into a closet -- which during later renovations wound up being sealed and forgotten. I forget the circumstances by which the server was rediscovered, but at the time it had been cranking along, without a reboot, for about 12 years.
~REZ~ #43301. Who'd fake being me anyway?
I worked at a place called MEMEC, before they had tax problems and were forced into being bought by AVNET. Management decided to use Cisco Network Registrar, and that hosed up the entire internal DNS setup, so I installed a stealth DNS subsystem that was literally 100x more efficient, and reliable. All the savvy IT people were on board with it, and when customers complained about slow resolution when using Cisco Network Registrar, I would throw up my hands and say "well, I keep telling them about it, but someone bought it -- you may as well keep using it."
We had a worksite setup in a conference room. By the time we were done we had a dozen dual CPU machines, tube monitors, and employees in there elbow to elbow. The worst came when they began adding servers and SANS :-) Anyway, prior to our moving in this conference room was hated for it's sheer cold, not anymore! It got pretty bad. to the point that we had both an indoor and outdoor thermometer posted up. Why? Well it seems that in our explorations of the ceiling we had found a vent to the outside. We constructed ventilation ducting out of cardboard leading to this vent and attached a LARGE box fan who's power wire dangled near someone's desk outlet. When temp outside was less than temp inside the employee was to plug in the fan, when the situation changed we unplugged the fan. At one point we actually tried to get them to buy expensive (at that time) LCD monitors just because of the heat savings - we were denied. That interior thermometer served another purpose too - when it hit 90 we got to go home for the day - paid :-P
:-) By the time we were done we were runing at least 4 different big fans, it was pretty sad and quite noisy too.
:-)
As you might imagine power became a problem since most of this was run through just a few sockets and the panels couldn't take it. The solution? It seems that breakers trip due to heat melting some sort of wax inside so to prevent this we added yet another cooling fan - this one directed into the open door of the circuit breakers in the closet!
Yeah, it was nutz and it was loud but it was a critical function so we did it. Our final fix for this was shutting down all of our rigged cooling, turning on ALL of our equipment, having every employee at their desk, and having the higher ups come tour one day. It didn't take them long to figure out that this was a fire hazard and a lawsuit waiting to happen and we got better digs.
Oh, we did find one other source for cooling but it was short lived. We explored the ceiling space and found an access panel intop the main A/C trunk leading to the bank next door. Upon opening it we were greeted with glorious cool air by the bucket full! Sadly when the bank employees noticed that it was becoming quite warm and complained the maintenance guys knew exactly where to go to fnid the source of the missing airflow - right to our office! It was good while it lasted though
Build it, Drive it, Improve it! Hybridz.org
Recently I worked at a US subsidiary of a worldwide electronics firm where the 1980ish server room air conditioner couldn't cool the 2004ish intel hardware and legacy system. IT had been requesting funding for addressing the issue for years but the executives kept cutting it out of the budget. The president of North America operations thought it was too expensive to fix the air conditioning or build a capable server room. So he refused to do anything.... then after the disk drives started cooking in the racks and impacting business operations he arranged a compromise. He instructed IT to put the AS400 running the entire ERP system in the hallway! In addition IT was instructed to move the Oracle/Linux systems that ran the websites, ecommerce, and firewalls in the same hallway! (note - same executive also felt that disaster recovery was expensive and over-rated so there was no redundancy, spare systems, etc. The place was one gigantic single point of failure). This "hallway datacenter" setup was in place for about a year.
And it would have continued had the SOX auditors not asked why the systems were in the hallway on their first day..... Funding for a new server room was approved in a day. Even Enron didn't put the ERP in the hallway.
I once consulted for a place that seriously must have had a complete psychotic running their cables. Cat-3 cables went from the computers to the wall jack, but only had RJ-45s on the computer end. The wall end of the cable used RJ-11s. All of these jacks ended up coming out to a punchdown block. More Cat-3 was connected to that, and then ran to the switch, at which point each cable was split so that two pairs ended up in one RJ-45 and two in another, and they were both plugged into the switch. Trying to track down a problem there was heinous in the extreme
We had just the opposite - we had a FREEZING cold office. I hate cold so this pissed me off and I went around to figure out WHY it was so cold. Seems some dumbass had setup a coffee put under the thermostat! I used a can of freezespray to get temporary relief and taped a cold soda to the thing too. eventually I bitched enough that they moved the coffee pot to a more reasonable position. I have no idea what they were thinking when they put it there...
Build it, Drive it, Improve it! Hybridz.org
I work for CAM Commerce Solutions, and we provide point of sale/inventory software to medium and small businesses. One of our customers had the free version of our software which limits you to having one physical store. To get around this, the woman (who was a very elderly lady in her 70s) would backup their database every night, bring it home on a disc, and restore it on her home computer to continue work. Then, do the same thing the next morning to get it back to their store. She actually called our support line to try to get help whenever she had restore problems. She's probably restored her database hundreds of times since they opened.
I just found the box to change my sig. Um.... [timeless witticism].
When I first started working here the server "racks" were a series of old green shelving like you would see at a mini-mart. They had 1/5 inch plywood bolted underneath the shelves so they could support the weight of our equipment. Behind this was the worst jumbled up mess of cables I had ever seen. It was an 8 foot long cascade of Cat V cables pouring down the wall into a pile a foot thick where it then rebounded back up to the patch panels. The patch panels themselves were barely visible from all the extra wire poking out of the punch downs. I found out later that Dave(the guy who was here before me but got fired after I started) didn't believe in using punch down tools because he thought they would damage the cables. He "punched" all of the runs down with the back of his pocket knife. If you so much as breathed heavy behind these patch panels at least 6 users would lose connectivity. Shortly after I started working here we bought some APC racks and me and another guy "uninstalled" the green shelves. With claw hammers. We then spent the rest of the night with a cable toner and a pair of walkie talkies re-terminating the entire network. After we cleaned up the cables we discovered a cast iron "cable guide" Dave had bolted to the wall behind the wall'O'cables. The bolts were all rounded off and I had to cut the dang thing off the wall with a hacksaw. I've been here 5 years now and I still stumble across some of Dave's handy work from time to time. Not too long ago a user asked me if she could stop holding down the f4+shift+tab+ctrl+L keys when she booted up her workstation in the mornings. Come to find out her machine blue screened a couple of times before I started working here. Dave told her if she pressed that finger cramping combination of keys when she booted up it would stop. She's been working here 7 years. Sometimes I kind of miss Dave. Then I'll discover something else he left for me and I'll get over it.
I helped start a computer leasing company, and was tasked as the network manager for the majority of my stay. This was about 7 years ago and we needed to create a database of all inbound leads (calls) We put in about 800-1200 records a day with about 40 fields (non-realtional.) I pushed for either filemaker or sql database. Unfortunatly I was turned down and told we need to use access as all the computers in the office had an office suite that included access. They did not want to spend the extra cash for filemaker lics, or an sql programmer. About 1.5 years later low and behold the access database goes to hell with records being lost and performance of the db tanking.
I get called into a board meeting and proceed to get chewed out for allowing this to happen. After about 10 minutes of being yelled luckily I came prepared with a copy of the email from a year and a half ago stating that access was not a solution for our scale of business and we would have problem after 1 year of use. Walked out without saying a word.
About a week later I was informed that we had a contractor to move the access of to sql, unfortunatly I was not allowed to interview mostly due to the previously mentioned board meeting...
Turns out the contractor was a bro and thought you could just migrate an older access db stright to sql... after monkeying around homeboy worked on the live DB and corrupted the whole thing. Apperently his expertise did not expand to working on a copy of the database. I get a call saying that we lost the whole databse and that it was my fault. Apperently Joe Newb had also never heard of backups and tried to blame me for the fact that he tried to migrate (the wrong way) the live database and blamed me for the whole thing.
Came in loaded the backup and kicked the guy out of the building... I left soon after this, as the company was rapidly going downhill for a multitude of other reasons.
---In a time of Chimpanzees I was a Monkey.
That reminds me...
15 years ago (wow I feel old now) a sprinkler main burst where I worked. It was on the 2nd floor of the offices and it dumped 20,000 gallons. The stairway was a cool waterfall, 3-5 inches of water were standing on the 1st floor and it was raining throughout the entire 1st floor as water soaked through the ceiling tiles.
This was in February and the temperature outside was around -10 F (-23 C) so opening the outer doors to let the water out also let freezing air in which instantly formed a fog you could barely see five feet in.
Luckily the downpour in the server room fell about 1 inch away from all of the equipment. All of the PCs on the 1st floor weren't so lucky. With the help of bunch of towels, rubbing alcohol, canned air and hair dryers we managed to save everything but 1 keyboard, one UPS and two CRTs.
That reminds me of my server set up as well. Up until just a couple of months ago all of our servers were in the kitchen closet of my home. Being a cheap bastard, I didn't want to pay for co-lo space and I liked running OS/2 systems. We had originally tried to find a better space in the house to do it, but the house had been built in the late 50's and the only functional spare pair of copper for DSL terminated in the wall next to the kitchen closet... which also seemed to have an electrical outlet on the same wall which had it's own circuit. So that's where the servers went. We cut a hole in the drywall and ran an extension cord through the wall and into the outlet on the other side and wa'la! To manage heat I parked a small desk fan in the base of the door opening and left it adjar for air to blow in the bottom and out the top. In a fit of self proclaimed cleverness a few years later I registered kitchencloset.com as our primary domain at one point as well. It made for great social conversations.
...@kitchencloset.com ...(pause)... ?!? WTF!!!
Q: What's your e-mail?
A:
Q: kitchencloset?
A: Yeah, that's where the server lives (said nonchalantly)
Q:
Then I just go on with the conversation like nothing unusual happened.
We've recently moved though and all of the systems have moved as well to a new location where just the equipment will live, and I'm also getting ready to decomission the WSeB (OS/2 4.5) server in a matter of days. I guess the only constant is change eh?
Funniest post on slashdot for weeks. Thanks. :)
The best I have seen was with the Arizona Department of Transportation (ADOT).
ADOT inherited a building in the easment next to the I-10/I-17 intersection near the AZ Fairgrounds that formerly contained some heavy cranes. These cranes were meant to lift heavy equipment onto trucks and were suspended on rails som 30 feet from the warehouse floor.
When the state inherited the building they decided to lease the downstairs to the Arizona Magazine for printing and assembling that fine pictoral magazine. The area in the rafters where the aforementioned crane resided was useless to them... so along comes the genius.
They used the heavy beams meant to support the crane as the basis for hanging a plywood floor. On this hanging / suspended plywood floor they would put in "Office Space" and lease that to ADOT for their IT development group.
It gets better.
Yes, all the electrisity and data wiring came into the building and was drapped across the gap between the wall and the suspended floor.
Yes the floor moved noticably, although it was too big for a single person to shift it on their own. After all there were 10 cubicles, 2 offices and a conference room on the floor.
Yes they put up pseudo walls separating areas of the room.
Yes, it was a warehouse so the ceiling and walls were just corregated steel... Yes it was in Phoenix Arizona... a dessert.
They did provide air conditioning so it was warm, but not unreasonably.
Another nice feature of the buiding was that it was partially beneath the I-17 N to I-10 E ramp that was about 60 feet off the ground. Every once in a while you would hear the clunk of someone Super Big Gulp hitting the metal roof, or the lite tap of a cigarette butt. At one point an ADOT truck in the parking lot was crushed by a truck tire that came off and went over the railing. Another truck was damaged by a water tank that came off another vehicle.
The best part about the IT solution was that they also put the servers up there. We had a separate room where all the servers were. It had extra air conditioning blowing on it, but it was not a contained room, it was firly open with walls that went part way to the cieling and a gap between the floor and the print shop below.
So the server room was in a metal building with no insulation in the dessert on a plywood floor suspended above a print shop under a highway were large things occassionally rain down. I shudder to think what would happen during a power outage.
Slowly waving my hand - "This is not the sig you are looking for."
I have at least 2 customers with Server/Bathrooms.....
The California-based parent company of a production facility I worked at in Ohio decided the best way to handle the network for Ohio, was to load all 35 PCs with (can't remember the actual software) a remote-desktop program. All 35 machines went through a pair of switch boxes, to a central PC running Win NT, it in turn was connected to a phone line via 56K modem.
All 35 users in Ohio were running remote desktops on a PC in California that was logged in to the parent company's network.
About half the time, the "system" would be down on Monday morning; it wasn't until 8:00-9:00 CALIFORNIA time that they even realized there was a problem; by the time someone at the parent company figured out that they needed to reboot the "Ohio PC", it was almost time to go home!
I set up a computer at the company in Ohio with Win 2K; plugged it into one of the switches, and had it continuously download engineering files during the off-hours.
I put everything into the same directory structure it came from in California, then set up file sharing for the Ohio users.
It took one day for every engineer to set drive letters to the now-local files and begin using that system exclusively.
It took the company president two weeks to figure out what I had done, and then it took him about 30 seconds to realize how much faster things were running.
And even though we had our own design department, and NONE of the files on our new network was used outside of our four walls, it only took the parent company an hour to fax the Ohio Company president a nasty-gram, demanding that we adhere to an obscure document control policy... and everything got put back like it was.
I have my own IT firm and our main business is network and system administration. One day we get a call from a callcenter. Apperantly their terminals are slow in getting the data from the server. Me and a collegue go over there to fix it. We get there and troubleshoot the network for a while and indeed, the network is blimey slow. We check the server, runs smooth (this is a 5000$ server anno 06/2006). Cables look okay will we get to the main switch setup, which looked like this: Switch>hub>hub>hub>hub. Anyway ALL the network equipment is X-brand junk. In the server room there is also a stange plastic smell. We open the switch and we fin the inner casing molten onto the pcb. After we changed everything to proper 48 port Cisco switches the system ran fine. Another good one is the time we got a call from a decent size warehouse. Their network was completely wireless and slow as hell, loss of connectivity, bad signal. Turns out they had 30+ computers hooked up to a 20$ 11mbit accesspoint which was located in a closed in the.......basement. Good times.
At an old workplace, there was a server (ok, a Sun Ultra 1, but it was running Oracle) which no-one seemed to know where it was, but it was on the network, running OK. I resolved to track it down...
First plan was to have it write something on the screen asking whoever saw it to call me. No joy; guess no-one went there.
Then I figured that it had a sound card & speaker - I also knew it would play .au files natively so went a searching and found a line from Monty Python's Holy Grail: specifically, "Help, Help! I'm being repressed". I then set up a cron job to cat this file to /dev/audio every 15 minutes. Unfortunately, all someone could hear was "help, help" from outside the comms room it was in and assumed someone was trapped inside. Security guard looks around and eventually finds the server with my name on the monitor.
At least we found out where the damn thing was, which was useful when some numpty builder cut the ethernet cable while working in the room.
A group within Franklin Templeton Financial actually archived unique data to JAZZ cartriges!!! And I dont mean old (we will likely never use that again) data, but important stuff that was just "filling" up the server! On a side note, they had an entire floor on a large building dedicated to Lotus Notes "development" I allmost peed my pants.
"This message was sent from an Apple
1. During the .Bomb Days I was the resident IT guru at an E-commerce developer. One Monday morning in October, one of our clients (a large department store) was complaining about a bug in their system. Usually the orders were issued sequential order numbers, but a bunch of orders were out of sequence and they were worried that the DB had gotten corrupted. After a few hours of debugging by our DBAs, they asked me to investigate. "Is is happening intermittantly?, or is it confined to a specific time period?" - before they could finish confirming that it only happend over an hour-long span, I prophesized that all the bad orders were between 1-2am. Stunned they asked how I knew - I quickly pointed out that we had changed our clocks back that sunday, and that the 1am-2am hour was repeated. ...which reminds me of the time we ran web use reports with odd results, until we found out our logs were logging everything in GMT (we were in NY). .... and my most recent hack: We were setting up a conference call with high-end conference equipment, but one of the C-level officers was going to be on vacation. We offered to remotely install our VC company's pc software, but he insisted we patch him in using his favorite Instant messaging client. Thankfully the High-end VC equipment had an SVGA in port so you could show a powerpoint on a pc to all participants, so we could show his video to everyone else, but not vice versa. But then it dawned on me that there were video out connections for videotaping a VC. So I went to Best Buy, bought a USB TV receiver and connected the Video out from the Video Conferencing system to the input of the TV tuner, and used the TV tuner as my own webcam. With some creative audio cabling, the problem was solved :)
Yes, the condenser sits outside the building. However the "condensing" in the word "condensor" refers to the condensing of the refrigerant under pressure within the outdoor refrigerant tubing, not the condensing of water on the coils. Water condensation occurs on the evaporator coils. (The Evaporation referred to in the name refers to the evaporation of the refrigerant within the evaporator coils once it passes through the restrictive orifice.) The evaporator is usually located inside the building, or in the case of a data center, on the raised floor itself.
This is the case even with a standard home split A/C, which has a drain coming out of the cabinet where the indoor coils reside to drain off evap condensate.
Before you insult other posters, please obtain a clue first.
SirWired
Sounds like 10Base5 or "Thicknet", which was the original Ethernet cabling spec.
-A Cabletron core router (a company that had long since been out of business even at the time) that required a $10 Walmart fan to be pointed at it 24x7, even though the room had AC. If someone moved the fan, or shut it off, the network would come crashing down.
-Backups for a GIS department done on a firewire drive, because no one wanted to pay for proper backup equipment or servers.
-Using used Flowpoints to connect buildings, long after they had stopped being manufactured. (Flowpoints used a copper pair, for distances *up to* several thousand yards, with speeds *up to* 2 Mpbs... if the stars were aligned, and if you could get them configured.)
-Old desktop PCs being used as Domain Controllers (hand-me-downs from users!)
-Office in an old jail, complete with asbestos, which made cabling an adventure.
-A 40 year old back up generator, which made power outages a crap shoot.
-Phone equipment in basement that had experienced a sewer back-up... the cleaners that were used caused corrosion on any exposed contacts.
-Responding to a remote site not having internet connectivity, we found that they had hired a contractor who had bypassed our firewall and replaced it with a D-link home router!
...///...
Nah, it was one of those corrugated plastic sheets that is quite stiff. It went across the top of the rack posts and there was no front, back or sides on this rack. Think poles, not doors. Plenty of air flow. It was like a sheet of plywood lying on the top of some poles that had stuff bolted to them, just made of light plastic. So the water dripped onto my head as I made sure the servers were healthy. 8)
You know, this bugs me - what's up with the whole unwashed nerd stereotype? I will occasionally put off showers if I'm just messing around at home, but I always bathe before running into other people - I hate feeling sweaty and gross, and most women appreciate a clean guy.
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
A few interetsing tales:
The miracle network: A customer complains that a couple of their machines are very slow on the network. Even internet (via cable modem patched into the network) is slow for those machine. I go over there and start checking things... both machines are generating packet loss to anywhere else on the network, and the network drops which run all the way across the building are direct runs. No wall jacks or anything. I grab our cable tester and hook up to the first cable ends here and in the wireing closet (everything is labeled). It shows no connection. None.
I plug it back in and realize I get no light on the switch for that jack. Mark down everything, and do the 'connect-disconnect' light shuffle... the switches lights don't change. Property owner assures me that the only network equipment is in this room, and the runs are all straight, no additional switches.
We begin tracing the cables through his dropped ceilings, all nicely zip tied together and of course all the same color cables. We find nothing out of the ordinary, and even peer down the wall space where the drop comes down with a flashlight... maybe there is a hub or switch?
Get back in and trace the cable which runs into a hole drilled into a desk, out another hole in the desk, under a several hundered pound file drawer... what's this.. 3 FEET of cable covered by what looks like a whole roll of black electrical tape.
So figuring we have a bad splice, we trim that out, use some punchdown splice blocks we had to add in a segment to replace the taped mess, and bingo, it all works.
The punch line? After pulling all that tape off, one of the other techs discover the splice was made by stripping the outer jacket off of 2 feet on each end of each cable, and wraping the still insulated wires together. All their network connectivitiy for that computer was by inductive coupling.
Computer 2 had a similar problem, having also been moved. There however they were a bit more professional. They went out and purchased a 10' cat5 patch cable, and sheared most of the plastic off the connector with a knife, then taped the two connectors together pin to pin.
The exploding server: We work on a lot of machines. A server comes in from someone who is not normally a customer, but needs the machine back pronto as it has all their customer data on it. They shut it off last night and now it won't power on. The tech wipes it off with a dry cloth before putting it on the server desk because it is filthy with crud. Plugs it up, and turns it on about the time I'm going back there to work on another machine we have back there. It looked like something out of a hollywood film. Sparks flew out of the power supply, things snapped, awful smells came forth, then the circuit breaker on the UPS tripped.
The server came from a machine shop and had been out on the floor. The entire machine, inside and out, was covered with dust sized flecks of metal. When they brought it here, the metal shifted around and formed shorts when we powered it on. They also needed it back up and running within 3-5 hours.
Their hard drive was fortunatly not a casualty, and we were able to move it to another box.
You give us 5% packet loss like everyone else, we go elsewhere again: Local business has two offices. One served by the cable company, one for 6 months by us via dsl (cable company who set up network in first place doesn't reach them). They have a business app that is not networked, they simply run a VNC session from the remote office. VNC appears to be unhappy with 5% packet loss.
Customer came to us because their previous provider was handing them constant 5% packet loss on their DSL circuit. They assured us it was the provider's fault and not hardware or the other end, and with the standard disclaimers (the phone company is notoriously reluctant to fix dsl lines that work 'mostly right' around here) we set up their internet. Less than a week later they call, VNC won't stay con
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
Back in 1992, I had a short job with a company that made window blinds. They had been around for long enough that the first computer they used was an Apple II. The guy they had doing their software tended to write everything himself, which was in a lot of cases necessary because what they were asking him to do just couldn't be done with any off-the-shelf packages. So, he wrote The Blinds Program. Originally written on some sort of UCSD p-code (remember that?) implementation of Pascal, The Blinds Program eventually got moved through a couple of other Apples, some sort of SmallTalk environment, and a couple of other things until it ended up on a trio of Macintosh II servers and a bunch of Macintosh workstations strung together with their AppleTalk connectors.
However, nothing was off-the-shelf. Literally, nothing. I'm pretty sure the guy wrote most of his own networking protocols, wrote his own database engine, and then wrote the guts of The Blinds Program within it. The data -- ALL data, from details about blinds orders to supplies purchasing to payroll records to scheduling to who knows what -- was in this mysterious custom database that consisted of pointers-to-pointers-to-pointers-of-data. You literally couldn't find anything unless you already knew where it was likely to be. The problem is, the thing was so dependent on its rat's-nest of pointers that if one was corrupted, the system would happily go writing new data to entirely unexpected places, including places that older data might already live.
Whenever anybody needed a new tool, it got written into The Blinds Program. Simple calculator? Put it in The Blinds Program. EDI interface to talk to your customers' systems? Put it in The Blinds Program. Need to dial up a BBS? Let's not use Red Ryder, let's write it into The Blinds Program (with hard-coded phone numbers, of course).
Well, eventually the guy decided he wanted to do something else with his life, and he left, and they handed the thing over to me. After about four weeks of exploring this monster, I realized that (a) the only person in the entire world who could grok it had already left and (b) I didn't wanna be around when (not if) it exploded and took the entire company with it. I went into the general manager's office and laid it out for him: go hire a consultant, have them design you a new system from supportable, off-the-shelf components, pay them to test and maintain it, or this thing is gonna eat your company. And then I resigned. No sense having a digital Hiroshima on my conscience. The GM hemmed and hawed and I think eventually hired the original guy back as a consultant and never did replace The Blinds Program.
The company went out of business in 1999 or so. I was amazed they lasted that long. My old boss got a good deal on a pool table at their bankruptcy sale.
---------------------------------------
Rotate the pod, please, HAL....
At one point over the summer I had finished building my new desktop system (previously I only had a laptop at my disposal), this desktop was set up at my desk which is about as far away from my router as possible. Having not thought ahead about the connectivity problem, I didn't have a wireless card installed on my desktop, so instead of going out and buying one (all that time going to the store and back, spending money) I instead hooked my laptop into my desktop through a crossover cable and shared the wireless connection on my laptop. Nothing particularly brilliant, but quite wasteful in the fact that I was essentially using the laptop as nothing more than a wireless card.
A nearby hospital called the consulting joint I worked for about 12 years ago because their network was down. I met the new IT guy in the lobby, and as he's walking me back towards the networking room he explains that the old IT guy was an ex-Bellhead. He's going on and on about the old guy's various problems, and I'm like, what's the point here? Then we get to the networking room, which is in a small cinderblock bunker in the parking lot between several of the main buildings. When he opens the door, there's no data networking gear... instead, there's a two wall collection of punchdown blocks, and a drift of red sticky dots all over the floor. Seems that the guy had wired 10Base-T in with his phone system in order to save money, and marked the data ports with a red dot. Worked okay until the A/C failed, and he was promptly fired.
Later on, I was consulting with a company that sold devices which you downloaded content to. Each user would log into the Apache-powered website, where a WebLogic app generated 4 to 10 MB of sorta-custom data and shoved it over the net into the device. Unfortunately, some bright fellow came up with the idea of also sticking this same data into the Oracle database as a BLOB... Site went down Christmas day and didn't come back until January.
Same company had a DS-3 from their corporate offices to their data center, and complained about performance. A little bing testing showed it was pulling 14 mbps rather than 45, so I spent a day with the datacenter and ISP verifying all their stuff. Finally, I ended up standing in their telco closet and tracing the cables... turns out it doesn't go CSU/DSU > Sun firewall > Catalyst 5500, it actually goes CSU/DSU > Sun firewall > dusty 16-port NetGear switch under the rack > Catalyst 5500. Shockingly, removing the NetGear made the problem go away, though I had to argue pretty hard for at least trying it.
Then there was the time that Dell sent some free Ethernet switches for testing. I accidentally knocked the power cord out of one, but when I tried to plug it back in, the power supply's little plastic collar snapped off and the whole thing fell into the switch.
"Nothing was broken, and it's been fixed." -- Jon Carroll
I know I'm late to this thread, but here's my submission anyway.
The names have been changed to protect the (criminally insane) innocent.
Working for a Large Systems Integrator, on a multimillion dollar contract to provide a very highly customized, specialized - we'll call it an ERP system - that's about as close as I want to come to describing what it actually does. This contract started in about 1996 (I've only been involved for 2 years).
There was a competing vendor supplying the personnel who would actually be RUNNING the system. While they were waiting for my company to finish the much more expensive, much more broadly scoped project, they hobbled together a little web-server-based system on their own to take care of the more mundane subset of tasks in a very basic fashion, out of their own personel budget.
Well, long story short, we ran into some major scope-creep, and the product we were supposed to deliver in 2000, ran over schedule. Here we are, 7 years later, and one of the components that's deeply integrated into the system, the customer LOVES. But the other component, which handles OUR system's mundane task, the customer's operators (the ones who work for the other vendor) HATE it. (surprise). All along, they had bitched and moaned, and were the primary drivers behind the scope-creep. And we faithfully complied with contract mods and retooling, etc. Now we're ready to start installing, and they pull a last ditch "kill it now!" move.
The compromise they ended up with was;
They continue to use their system for the mundane "inventory tracking, and resource scheduling" functions, and they hire an additional operator whose sole task is to manually copy data from their system (which, for security reasons, has no physical network connection to ours, and some of the data is display-only, and we are not on contract to provide an export/import capability), to the console of our system, where that data is used to drive the automatic resource configuration tool we provided them with (which is the component they can't do without).
Then, if there's any change to the actual data used by their operations based on what our system did with the numbers, this guy copies the data back into their system, to make sure both systems are on the same page.
Our system; about 10 computers, Oracle-based database servers, custom-written front-ends, full-on network management (we bundled Tivoli with it), security, failover, redundancy, centralized management, multiplatform (Windows/AIX/Linux), tightly integrated with a system of probably 100 other computers, including truckloads of documentation, manuals, procedures, etc. including routers, switches, and some very specialized hardware.
Their system; A pair of clustered Windows IIS boxes.
Up until last year, they (the other contractor's people, the USERS of our system) were BEGGING us to add this and that and the other feature to our system. Then they pulled this political stunt at the last minute, to try to kill the whole project. Then the compromise; the customer is stuck with owning, operating, and maintaining BOTH systems in parallel, with the weaknesses inherent in both, and the strengths of either system completely erased.
It's a living.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
I heard some stories of the early days of Voice over IP being deployed at Sprint. Here in KC Sprint is a major employer. I was at lunch at a restaurant when several Sprint guys (apparently from IT) came in and started telling war stories. One told of the time an entire building phone system failed at the flip of a single switch. The techs had just deployed a trial of VoIP for the entire building. They had, of course, placed the new VoIP server int eh data center witht he rest of the servers. Being new, they hadn't completed all the "official" labeling, and it was a trial run anyway. After a while one of the PHBs of the center was walking by and noticed an unlabeled machine in the rack. Policy said that nothing unofficial ran there, and if it wasn't labeled it was official. He powered it down and went to the phone to call some techs to get rid of the offending machine. Of course, the phone was dead. Next room, next phone. Dead. Continue ad nauseaum across the whole building with people trying to find a functional phone. Cell phones work, but nobody can call in to the building. Eventually the techs realize the VoIP server is messing up, go the the center and discover the dead phone server. While they mill around waiting for the server to come back up the PHB comes back by to tell them about the strange machine he found. He notices it back on and promptly shuts it back off. Needless the say a heated discussion of PHBs, power switches, and corporate policy and common sense ensued.
This is reminding me of a temp gig as a graphic designer in a London property firm. It's probably not that far out there, but it was just a few years ago, in 2001. For such recent history, it wasn't pretty.
The cosy little nest of Macs that was the design department wasn't hooked up to the rest of the (PC) network, because the rest of the network was Token Ring and the Macs had some kind of new-fangled networking plug that Wasn't Token Ring. Besides, there was some kind of non-Mac-friendly software to run email, calendaring or some kind of stuff that wasn't my problem as a temp.
Every Mac therefore had a PC next to it for these crucial office tasks. Of course, we often needed to get data from the PCs to the Macs and back again. The Macs weren't remotely networked, to each other, to the PC-based Token Ring network, or to the internet.
So, Mac-PC or Mac-Mac data transfer was purely sneakernet. This happened by putting a PC-formatted Zip disk into a Zip drive connected to the PC, copying files onto it, ejecting it, putting it into the Zip drive connected to the Mac, then copying files off it. When a Zip disk or drive failed (click-of-death how I miss thee) a working drive got passed around the office to move files about.
Of course, nothing a cheap hub and some ethernet cables wouldn't have fixed. But it worked! Why fix it?
it's not about the karma, it's about the whuffie
Most interesting, informative post I have run into in this thread... really, you shouldn't have posted this AC
Had a friend who had a box colo'd for free at a fairly run down ISP. It was due to some contractual obligation they had to live up to, so they were not happy about it. Was supposed to have a 10 mbit connection to the switch, and pretty much unmetered out over the backbone, but it never saw more than 2 mbit. Turns out, the ISP put an extra long ethernet line place, 500 feet of signal killing goodness. Just to make his service suck, in a way not easy to see unless you pulled up the raised floor, and noticed that all that wire was in fact, one single wire, going back, and forth, and back and forth.
Another friend has the dilbert boss, decided that the router _needs_ to be at one end of the building, far away from the office. A distance of 350 yards, add even more when you take into account the fact that the wiring goes from the ground floor at one end, up above the second floor, spans the building, and comes back down to the ground. Then, he decides that 10 base T is too slow, and he demands gigE. My friend tries explaining to him why all of this is nuts of course, but the man never listens, and says, "just buy a bunch of repeaters." So, this gigE run of nearly 400 yards, is daisy chained together, every 100 yards or so, by a 4 port gigE switch. Most of them live up in the drop ceiling above the second floor, and have some ludicrous power lines run to them. Store and forward nightmare. Boss is currently pissed because it is slow, and because the gigE switches didn't make his 3 mbit cable modem go any faster.
--Nuintari
slashdot : where an opinion can be wrong.
Nope, different buildings, different locations. Both examples were separate instances
of the 'creative' ideas in my company. BTW there are others, but I fear if I mention
too many some people might guess, (and consequently come back to haunt me). Needless
to say they all worked very well for many years and I've only had the good fortune to
banish some of them forever.
BTW, the computer room actually was built over a part of a factory floor that... er...
used the drains extensively.
-- main(s){printf(s="main(s){printf(s=%c%s%c,34,s,34
The one where ambient humidity serves as an insulator and thereby prevents the aggregation of static charge, which is a tremendous risk in a room with that many highly charged moving parts.
I realize this is nitpicking (and the rest of your post was right on) but I think you meant to say "where ambient humidity serves as a conductor and thereby prevents the aggregation of static charge..."
Higher humidity makes the air a poorer dielectric, meaning that static charges dissipate before they can build up to significant voltages. With dry air, the air is a better insulator, hence higher-voltage static charges. (This is why the kids' trick where you scuff your feet on the carpet, particularly while wearing rubber-footed one-piece jammies, and then shock the beejesus out of someone, only works in the winter.) Naturally, anything that produces sparks -- particularly my favorite, Van de Graaf generators -- work far better in dry air than wet.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
I swear this is true. At a company I worked for about a decade ago, they had a network switch (or maybe it was a modem) in the toilets. Not actually "in" the toilets themselves you understand, they weren't quite that daft. Imagine the conversation. "Where should we put this switch? Next to the server perhaps?", and the answer would be "Nah, that's too inconvenient. Let's put it on a shelf in the lavatories instead".
Once, about three years ago, I was part of a team fixing the datacentres for a big mobile phone company. When we got to the the centre, it was a mess, dead, unused boxes everywhere drawing power, racks with only one server in them, you name it. The peach though, was the power distribution board for the biggest of the centres. This thing supplied power for every box in the place, and was running so hot that you could actually fry an egg on the top (we measured it at about 82 C). But what was absolute genius was how they kept it ticking over. They had a portable air-conditioning unit shoved up against it. Which was plugged in to the same board it was cooling. Oh, yeah, this was the main datacentre for over ten million customers, which was 100 metres away from the 'disaster recovery' backup sites. Both were on a floodplain.
The now-defunct Tower Air, a charter (mainly) airline that operated out of JFK, set the standard for airline IT hacks, IMHO. Their datacenter consisted of a closet (literally) with no ventilation that housed one rack containing a couple low-end Unix servers. The network cables between the incoming switch and the servers were laid on the floor, so that you tripped over them whenever you went into the room.
A previous employer's datacenter was very state-of-the-art, with the exception of the amount of available common sense. There was some need to drill a hole in the concrete floor underneath the raised flooring, so they brought in a freaking jackhammer and went at it for three days. Rows of rack-mounted Unix servers were shaking all day long, as if in a major earthquake. What a surprise when some of the servers started to go belly-up, resulting in application outages that were costing my employer lots of money. The doofus VP of engineering was yelling at the vendor about the quality of their servers, when I pointed-out to him that no server was built to withstand the amount of vibration to which these servers had just been subjected.
Comment removed based on user account deletion
Hotmail runs on solaris Boxen. :)
Originally, MS was to buy AT&T, not Cingular. But AT&T backed out when microsoft couldnt migrate to their own domain controllers and had to go back to the Suns.
A friend of mine does networks for a major oil company. He brought back pictures of an installation in (I think) Angola.
The first notable point was that the outside of the building seemed to be polka-dotted. Those were the bullet holes. Nobody was firing at the building per se, it was just downrange from one of the local hot spots.
The telecom room for the building was on the second floor. My friend's company rented the fifth floor, and some other company rented the floor above that. There were no data risers in the building. Instead, there was a hole about a foot across that had been knocked through the wall on the second floor, and a couple of six-inch holes on the fifth and sixth floors. The data cables ran out through the wall on the second floor, up the outside of the building (among all the previously mentioned bullet holes), and back in through the walls on the fifth and sixth floors. My friend's company had gone to the extra time and expense of running conduit, but the company above just had a 40' swag of cable hanging there in the breeze.
The telecom room itself looked like the aftermath of a "will it blend" episode. Take 20 drums of assorted wire product and unspool it all, wad it all up into a 10'x12'x8' snarl, then start grabbing random segments and pulling until you can nail that particular chunk of wire to some point on the wall. That's what it looked like. When they needed to fix something, they put one guy upstairs with a handset on the line and a radio, then sent another guy with a radio down to the telecom room to wave a toner around until the guy upstairs started to hear noise.
My friend also told a story about renting a phone line from the Chadian national government. That's all it was.. the same kind of POTS line you have to the phone on your wall, for a cost of something like $10k per month. One of the things he had to do was install a modem on the line (which saw about 95% use since it served a whole office), but the line itself was so unbalanced that he couldn't get a decent signal. He mentioned this to the local telecoms expat, who said, "oh yeah.. come on. You'll enjoy this."
They got their security people and drove over to the phone substation (which was run by the Chadian military and had armed guards outside), and before the truck even stopped, the expat was out and stomping his way past the guards, through the door, and into the wiring frame. As my friend came in behind him (pretty much thinking, "okay, we're dead"), he realized that he could hear voices speaking English througout the building. The military had tapped their line, but since it was used so much, they were running the sound through a loudspeaker rather than just listening to it on a handset.
Halfway down the row of wires, the expat stopped, pointed at the wire that dropped down from the celing and tapped into the frame, and shouted, "IS THIS A WIRETAP? IS IT?" By that time, *everyone* in the facility was there watching, and the colonel who ran the place was saying, "no no! Is not a tap," despite the fact that everyone could hear the voices of the people using the line over the loudspeaker.
The expat yanked the wire loose and said, "damn right it isn't a wiretap.. it's rubbish. We spotted it two seconds after you put it in. Here.. let me show you what to do," then proceeded to do a quick lecture-demo on professional wiretapping techniques.
The following computer is real. The names have been changed, to protect the innocent.
So, my friend Jake, in high school, was running a combined FTP and HTTP server/firewall, in addition to a gaming/school desktop and laptop. His standard desktop was a something Athlon, as was the laptop, and both were in fine condition (well, the desktop ran Windows ME, but the hardware was unfscked). The server, on the other hand...
He called it Frankenstein.
The system was a 350 Pentium II, on a motherboard from an old Aptiva. Gross, but not too strange. What was strange, however, was the case he had it in. From the outside, it looked like a normal old tower case. Internally, though, it was a monster. The power supply from the Aptiva was a stupid custom job, and hadn't fit the brackets in the case, so, of course, he had duct taped it to the case's ceiling, avoiding the vents. The drive bay carrier was long since gone, so the optical drive was taped likewise to the top of the case. The hard drive was balanced on its side near the front of the case, under the motherboard's bottom edge.
It was the motherboard that was the worst. The mount points on the motherboard and case failed to agree, not only in all particulars, but in any. So his solution was to suspend the motherboard inside the case... with twine run through the four corners of the board. The motion of the fan would set the board rocking back and forth, in a sickening manner.
Well, this one might not be entirely in the spirit of the original question since it's not a cool "hack" so much as it is just an amusing error in planning, but here we go anyway:
Back in '95, my father was a VP of research for a large manufacturer of transmissive and reflective coatings for various glass applications (think insulated windows for the simplest example of said product) in Palo Alto, California. I was 15 and in highschool at the time, and having spent many a year trying like hell to keep a series of shitty no-name x86 computers up and running well enough to play the latest games, I had a sufficient skillset (and my dad had sufficient clout) to get me a job in their IT department. I did pretty well, and quickly found that users generally only got mean-spirited when made to look stupid, so a small dose of humility coupled with an interest in details on their primary task - "While I fix these printer drives you accidentally deleted, I was curious, what does a spectral photometer do?" - kept me out of trouble. Long story short, next year when I switched to full-time for the summer break, my boss actually brought me for a one-day business trip to our plant in Tempe, Arizona.
Now, you've got to realize, a business trip for a 16 year old (this was '96 now) is freakin' AWESOME. I was nervous as hell, had been up since the crack of dawn to take a red-eye with my boss out to the plant, and was deathly afraid I'd do something to embarrass not just me, but my father for having recommended me. So it was pretty unnerving to learn that my first job involved going into a large clean room production area, kept free from particles that could settle on the film during that specific type of sputtering process. We're talking the full disposable "bunny suit" that covered everything but the eyes, even with little slippers, and an airlock-type blower to clean you of all particles before entering.
The problem was a simple fix, really. The brand of 486 motherboard we were using at the time had a tendency, in about 1 out of every 3 units, to burn out the CMOS battery much earlier than you'd expect. And for a manufacturing-floor computer, not having a correct internal clock was a bad thing, not to mention that the lab techs had to go through some errors at startup with BIOS setings no longer being saved. So I suited up, cleaned off the replacement part and my tools as ordered, and went to find the bad machine.
That took some doing, oddly enough, since these computers were rarely shut down due to a 24/7 production schedule, so I had to go through back records on hand to find the lab techs' notes during the last power cycle on which computer had the boot errors. But, once located, the terminal was taken offline and I was able - after being told I had 20 minutes for the repair, tops, before the company would start to lose money as they needed that terminal again - to drag it off to a quiet, out of the way corner for the swap.
But see, there was a problem in the planning stages when this plant was set up. The PCs they used to control the machines were pretty complicated to configure, and the machines run in the clean room were just slightly modified versions of those used in the full-on manufacturing area in the main plant in Palo Alto. It was actually only a pretty small fraction of these production machines that had to operate in a clean environment. So when it came time to set these terminals up, they carefully washed off the outside of the older computers - computers, mind you, that have been sitting on a 24/7 PRODUCTION FLOOR with 10+ lab techs nearby at all times and various debris kicked up from the manufacturing process - and shuffled them off into the clean room.
So picture the scene: our hero, an extremely nervous 16-year-old on his first business trip in full head-to-toe bunny suit gear in the corner of a white, immaculately clean production floor opens his target computer to find a system so full of dust that he can't even SEE the goddamn cards inside. We're talking full-
In a similar vein, I once cooked a microwave burrito on one of those 4G Seagate barracudas (the ones with the big metal flange with holes on the front). It was plugged into my workstation, an (even at the time) old Sun Sparc 5. I had the cover off the external housing, and I noticed that it was getting *really* hot, so I stuck the burrito in the front for an hour or two. After an hour, I flipped it around to cook the other side. Steam came out of the package when I opened it -- that sucker was HOT!
Zhrodague.net - I do projects and stuff too.
The Sony Singstar audio adapter is a class-compliant USB audio device, but better yet, each individual device has a unique embedded serial number for a name, so you can aggregate multiple units via a USB hub.
I've been using a rig of 3 Singstar adapters sourced off eBay for about $20 each, a Griffin iMic I had lying around, some old XLR connectors from junkbox terminated to 1/8th inch plugs to attach the mics and splits, aggregated via a cheap hub and AudioMidi Setup, then fed to Audacity on my iBook to record my band live for about a month now.
Eventually, when we've made a bit more money, I'll probably by a Presonus Firepod, but for now, this does the job almost as well for about a tenth of the cost. Level calibration's a bit of a pain, but the results have been pretty good so far.
"I hope you like Guinness, Sir. I find it a refreshing substitute for, er... food." Col. Jack O'Neil, SG-1
Here's one worth mentioning...
I was working for a small start-up around 1995. We had a PC running in a remote facility. We administered it remotely using Norton's PC Anywhere. Problem is, the machine usually crashed after a few days of uptime. The only way to recover it was to physically hit the reset switch.
I hacked around this by putting together a small circuit card that connected to the phone line and the reset line inside the PC case, with a simple Z80 microcontroller in-between. What it did was simply count the number of rings within a 30 second span. If the count reached 4, it would strobe the reset. Since we had PC Anywhere set to answer after 2 rings, whenever we dialed into the machine, it would either answer (because it was up-and-running) or it would reboot (because it had crashed, and PC Anywhere no longer answered the phone).
You may be interested to know that Schneier devoted a blog entry to linking to this description:
u rity_theate.html
http://www.schneier.com/blog/archives/2007/01/sec
Xenu loves you!