Slashdot Mirror

← Back to Stories (view on slashdot.org)

AOL Creates Fully Automated Data Center

Posted by Unknown on from the tomorrow-system-architect-automates-himself dept.

miller60 writes with an except from a Data Center Knowledge article: "AOL has begun operations at a new data center that will be completely unmanned, with all monitoring and management being handled remotely. The new 'lights out' facility is part of a broader updating of AOL infrastructure that leverages virtualization and modular design to quickly deploy and manage server capacity. 'These changes have not been easy,' AOL's Mike Manos writes in a blog post about the new facility. 'It's always culturally tough being open to fundamentally changing business as usual.'" Mike Manos's weblog post provides a look into AOL's internal infrastructure. It's easy to forget that AOL had to tackle scaling to tens of thousands of servers over a decade before the term Cloud was even coined.

105 of 123 comments (clear)

  1. So it will take ages for a fix by Anonymous Coward · · Score: 1

    How long will it take for an engineer to get there to replace a card or server?

    1. Re:So it will take ages for a fix by errandum · · Score: 1

      About as much time as it takes on most datacenters that already are monitored remotely. With news like this some would think Nagios or Ganglia did not provide the admins with a web interface.

      PS: They might want to, at least, man it with a security guard to sound the alarm in case of fire or robbery

    2. Re:So it will take ages for a fix by PTBarnum · · Score: 2

      The article states "failed equipment is addressed in a scheduled way using outsourced or vendor partners". They don't care if an individual server is down, they just move the workload elsewhere, and wait for a repair. So there actually will be people in their data center doing repairs, they just aren't AOL employees and aren't based in the data center. I could see making a decision that a longer wait time for repairs is justified by labor savings, but it isn't really obvious where those savings come from. There is a suggestion in the article that they want the flexibility to increase or decrease the number of workers as needed, which is somewhat easier with contractors than regular employees, but with regular employees you can get a similar effect from part time or overtime work.

    3. Re:So it will take ages for a fix by Martin+Blank · · Score: 4, Interesting

      One of the major backbone providers has a lights-out data center not far from my work. I know a guy who has a hosting business there, and he's shown me around to the limits of his access. There is no one on-site from the company or its contractors--not even a security guard. They have biometrics plus PINs for access; it's laced with low-light/IR cameras (it wouldn't surprise me to learn they have microphones); it has motion detectors in case the cameras miss something; and the redundancy is incredible. They maintain contracts with local electricians, plumbers, and a few technical companies should a blade burn out. They manage the entire thing from a few states over, and as of a couple of years ago almost all of their data centers had been converted to run this way. Savings were good, something like a million dollars per DC per year even as unanticipated downtime decreased.

      I looked at it and saw the future of IT. I wasn't sure if I was more impressed or scared.

      --
      You can never go home again... but I guess you can shop there.
    4. Re:So it will take ages for a fix by arbiter1 · · Score: 1

      What it sounds like everything hosted there will be a cloud type system, so if 1 machine dies you won't even notice.

    5. Re:So it will take ages for a fix by Zocalo · · Score: 3, Insightful

      Who cares? I'm guessing you don't have much experience of server clusters but generally, long before you get to the kind of scale we are talking about here, you start treating servers in the same way you might treat HDDs in a RAID array. When one fails, other servers in the cluster pick up the slack until you can either repair the broken unit or you simply remote install the appropriate image onto a standby server and bring that up until an engineer physically goes to site. Handling of the data is somewhat critical though; should a server die you ideally need to be able to resume what it was working on seemlessly and without causing any data corruption; think transaction based DB queries and timeout/retry.

      If you have enough spare servers and you can easily get by with engineers only needing to go on site once a month or so, assuming you get your MTBF calculations right that is. There's a good white paper by Google on how 200,000 hr MTBF hard drive failure rates equate to drive failures every few hours when you have a few 100k HDs.

      --
      UNIX? They're not even circumcised! Savages!
    6. Re:So it will take ages for a fix by mikael · · Score: 1

      It's more scary - every field of technology evolves that way.

      Early valve computers used to require technicians to replace burnt out valves on a daily basis. Each morning of the day, the technicians would go round and replace any that had burnt out or were about to burn out. Now your PC has about 2 billion transistors or more (CPU +GPU), and not one will burn out.

      100 years ago, it would take 25 minutes to make a long-distance call between San Francisco and New York due to all the operators involved. Now, it's all automated.

      200 year ago, it took four people to operate a single loom to make a shirt. Now, one technician can supervise fifteen industrial carpet making looms that reload automatically.

      We've actually got a global clothing surplus due to all the designer and brand name labels to the extent that local makers in developing countries are put out of business.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    7. Re:So it will take ages for a fix by Grishnakh · · Score: 1

      How long will it take for an engineer to get there to replace a card or server?

      Much less time than it'll take them to get a user.

      Honestly, I was surprised by this article; I thought AOL had already folded.

    8. Re:So it will take ages for a fix by X0563511 · · Score: 1

      Those must be some fancy microphones to be of any use inside a DC...

      --
      For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
    9. Re:So it will take ages for a fix by X0563511 · · Score: 1

      No, but if a whole cabinet or row goes out because someone wasn't around to notice the funny smell or magic smoke coming out of the power equipment, or hear that ACU fan belt starting to come loose, you just might notice...

      --
      For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
    10. Re:So it will take ages for a fix by trickyD1ck · · Score: 1

      This isn't scary. This is things getting better.

    11. Re:So it will take ages for a fix by kmoser · · Score: 1

      One word: RoboCop.

    12. Re:So it will take ages for a fix by tehcyder · · Score: 2

      This isn't scary. This is things getting better.

      It's scary if your job is manually maintaining servers.

      --
      To have a right to do a thing is not at all the same as to be right in doing it
    13. Re:So it will take ages for a fix by Chapter80 · · Score: 1

      ...over a decade before the term Cloud was even coined.

      You mean over a decade before you heard the term?

      Cmon! HP was using the term Cloud five years before "America Online" existed in 1991.

      Just because your expertise doesn't extend back before you got that first AOL floppy and went online to type "a/s/l?", it doesn't mean it didn't happen.

    14. Re:So it will take ages for a fix by Grave · · Score: 1

      I'm not so sure. While individual reliability has increased dramatically, the shear number of systems in use around the world has increased as well, probably along a similar rate. Will we eventually reach a point at which computer hardware simply does not fail without an external event (power surge, physical damage, etc)? Maybe. But I don't see that happening until performance plateaus.

    15. Re:So it will take ages for a fix by Critical+Facilities · · Score: 1

      You know, I've seen this idea floated time and time again, and I still have to say that I think this is all smoke and mirrors, at least from an infrastructure perspective. My opinion is that you will never see an industry-wide move toward Data Centers without onsite staffing simply because the risk is far too great. Even in the example you cite above, there is huge liability and potential for disaster. Biometric security devices can fail or be defeated, and all I can say about infrared cameras, microphones, or any other monitoring systems is that while they are an important component to a successful operation, they're not going to do you a whole lot of good if you don't have a skilled engineering presence in the building who is capable of responding quickly when needed. If you have a chiller pipe burst with no one in the building, all you're going to have is some great infrared footage, and perhaps some nice audio of a friggin' electrical disaster while your mechanical contractor races to roll trucks to the site.

      Disasters aside, a properly maintained, fully redundant site has so much preventive and predictive maintenance that has to take place in a year, you're going to need an onsite staff to coordinate and facilitate those operations. Even if you have service contracts set up for various systems (i.e. Chillers, UPS, Generators, ATS, STS, etc), you're still going to need an in-house staff to migrate loads as needed to allow various components to be de-energized for service/testing. Someone has to be there to allow vendors access to various areas of the DC and to ensure that everyone is doing what they're saying that they're doing. That is, someone needs to be there to ensure that someone isn't just "pencil whipping" the PM visits on your CRACs, or that someone actually went around and tested ALL of your smoke detector heads above and below the raised floor.....you know, someone to watch the watcher.

      I guess my feeling is, while it's tempting for companies to get all excited and get themselves convinced that they're going to save tons of money by running their DCs with no onsite staff, an it just sounds so futuristic and science fiction-ey to imagine a facility being run by remote control, but if you really think about the potential for disaster and the liability, I don't think it's worth it. I think that if anything, someone will try it and will push the issue for long enough, and someone will have a big enough issue that has a big enough impact that makes a big enough story that we can all read about and say "shit, I'm glad that wasn't me", and suddenly, these non-manned DCs will be a thing of the past. I admit, these are just my opinions, and this being Slashdot, you are all of course welcome to disagree to your hearts' content.

    16. Re:So it will take ages for a fix by mikael · · Score: 1

      Reliable will become nearly 100% if everything moves to solid state. How many electric motors are there in a laptop these days? Cooling fans, hard disk drives, CD-drives (auto-eject, play motor) - must be around five or six.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    17. Re:So it will take ages for a fix by Lashat · · Score: 1

      In the your second paragraph, you detail PERFECTLY the reasons this move is taking place. Actually, the first two words of that paragraph state the entire goal. "Disasters Aside." I believe that the goal is predictable realiability and uptime for the entire operation. I agree that there must be a staff of engineers for maintainence, but all the maintainence can be put in the schedule, plan, and budget. Even the supervisor/inspectors/watchers fall into the "expected cost" column.

      If a data center spends more on prevention vs. recovery and yields near zero down time the profits come in the form of a better service offering than your competition. System failure for any reason short of acts of God should not interrupt your operation to the point of affecting the year's profits.

      --
      For every benefit you receive a tax is levied. - Ralph Waldo Emerson
    18. Re:So it will take ages for a fix by Martin+Blank · · Score: 1

      It depends on the noise level of the DC. Where I work, microphones would be useless, but some of the computer rooms in other buildings are relatively quiet and we've used microphones on NetBotz devices when people have been in the room and we're monitoring what they're talking about while working. (It has sometimes saved a phone call when a configuration looked odd momentarily but they were doing it for a reason.)

      --
      You can never go home again... but I guess you can shop there.
    19. Re:So it will take ages for a fix by Tomato42 · · Score: 1

      electron migration -- electrons moving metal atoms -- is a problem when your transistor consists of a dozen or so atoms...

    20. Re:So it will take ages for a fix by Critical+Facilities · · Score: 1

      I'm not sure you're following me. You're saying "even the supervisor/inspectors/watchers fall into the 'expected cost' column" as if a contingent of dedicated, on-site Data Center engineers is no big deal. However, the whole boast of this ideology is that they're promoting the idea of NO ONE being in the Data Center, and I'm countering that I do not believe it's do-able on a long enough timeline. I think that there are some aspects that can be done remotely, and I think that there are some services that can be made redundant through the network. I think the model of making the Data Centers themselves redundant to one another makes sense as Google does in their operational model, but even in those cases, I do not think that the idea of operating a Data Center with literally no on-site engineers is a wise idea.

      I agree with you that a properly built and maintained DC should be pretty fault tolerant, and in all likelihood, there probably won't be any major issues. But, we all know you have to plan for all those nice little curve balls that just have a way of coming at you. No one expects that one-in-a-million harmonic on the B phase of your incoming Utility feed that takes out your TVSS, which blows the capacitor bank inside it which (unbeknownst to you at the time) was recently changed to a different manufacturer's type which is a MOV type which when presented with just such a situation tends to fail phase-to-phase as opposed to failing open which creates a nasty little arc flash event spraying all kinds of molten copper around the inside of switchgear cubicles, or some other bizarre situation. The point is, weird stuff happens, and when it does, brother you better have people in the building who know what the hell they're doing, and even more importantly, people who know what not to do.

      Even if you want to write off those types of situations as the "one in a million" types, you still can't just casually throw in the engineers into the model as if that's insignificant. The whole crux of this idea that the DC of the future is going to be truly un-manned is ridiculous. How in the world can you maintain a Data Center and keep it truly redundant without onsite staff? Anyone who thinks that you can just set up contracts with Liebert, GE and Caterpillar and call it a day either needs some counseling, or they've never tried to manage a DC before. I'm not saying you can't operate a DC like this, you can....for a while. You can also go out and buy a new car, and never do any maintenance on it, and just keep driving it, and everything will be fine.....for a while. (sorry, had to throw in the mandatory car reference there for good measure)

    21. Re:So it will take ages for a fix by Lashat · · Score: 1

      Well maybe the pendulum will swing the other way. After that "for a while" time is up the DC owner will rebuild from scratch. They will off-load to those other DCs they own for the duration. During that rebuild the cost-effectiveness of running with "No one" on location (which I translate as minimal staff. Be they employees or contractors.) will be evaluated.

      The question will be asked. Is it cheaper to re-build the entire place with updated equipment, processes, and possible wire and ducts too. Or, is it cheaper to go the maintainence route with a full cadre of staff and update a little at a time.

      I don't know for certain, but most cases in history support the theory of machine trumps manpower.

      --
      For every benefit you receive a tax is levied. - Ralph Waldo Emerson
  2. Manos, Hands of Fate? by Anonymous Coward · · Score: 1

    Is now hands-off?

    1. Re:Manos, Hands of Fate? by oodaloop · · Score: 1

      I guess they won't need Torgo to look after the place.

      --
      Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
  3. Uh.... by Anonymous Coward · · Score: 1

    So they have a fully automated unmanned data center... For their fully unused unpopulated services?

    WIN!

    1. Re:Uh.... by silverglade00 · · Score: 3, Funny

      Nobody will be there to see Skynet become self-aware. What... you thought the end of humanity wouldn't come from AOL?

    2. Re:Uh.... by crafty.munchkin · · Score: 1

      Well played, sir!

      --
      ... wait, what?
  4. Wow .. how '2000'ish by johnlcallaway · · Score: 3, Informative

    Wow ... we were doing this 10 years ago before virtual systems were commonplace, 'computers on a card' where just coming out. Data center was 90 miles away. All monitoring and managing was done remotely. The only time we ever went to physical data center was if a physical piece of hardware had to be swapped out. Multiple IP addresses were configured per server so any single server one one tier could act as a fail over for another one on the same tier. We used firewalls to automate failovers, hardware failures were too infrequent to spend money on other methods. We could rebuild Sun servers in 10 minutes from saved images. All software updates were scripted and automated. A separate maintenance network was maintained. Logins were not allowed except on the maintenance network, and all ports where shutdown except for ssh. A remote serial interface provided hard-console access to each machine if the networks to a system wasn't available.

    Yawn ......

    --
    I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
    1. Re:Wow .. how '2000'ish by johnlcallaway · · Score: 3, Informative

      Thanks for not pointing to the actual blog in the original article. So what they are really blogging is their ability to move an entire DATA CENTER without having to send people to do it. Other than .. you know .. install the hardware to start with.

      Never mind........

      --
      I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
    2. Re:Wow .. how '2000'ish by rubycodez · · Score: 1

      virtual systems were commonplace in the 1960s. But finally these bus-oriented microcomputers, and PC wintel type "servers" have gotten into it. Young 'uns.......

    3. Re:Wow .. how '2000'ish by ebunga · · Score: 1

      Eh, machines of that era required constant manual supervision, and uptime was measured in hours, not months or years. That doesn't negate the fact that many new tech fads are poor reimplementations of technology that died for very good reasons.

    4. Re:Wow .. how '2000'ish by timeOday · · Score: 2

      And other new tech fads are good reimplementations of ideas that didn't pan out in the past but are now feasible due to advances in technology. You really can't generalize without looking at specifics - "somebody tried that a long time ago and it wasn't worth it" doesn't necessarily prove anything.

    5. Re:Wow .. how '2000'ish by rubycodez · · Score: 1

      depends what model you bought, the redundant, fault-tolerant systems stayed up while components replaced

    6. Re:Wow .. how '2000'ish by rednip · · Score: 2

      "somebody tried that a long time ago and it wasn't worth it" doesn't necessarily prove anything.

      Unless there is some change in technology or technique, past failures are a good indicator of continued inability.

      --
      The force that blew the Big Bang continues to accelerate.
    7. Re:Wow .. how '2000'ish by mikael · · Score: 1

      Telephone exchanges in rural areas are like that. The only time a technician had to enter the premises was to clear out old equipment. There was enough spare capacity in the exchanges that the only work required was to open the local cabinets on the street and pair up a new telephone line.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    8. Re:Wow .. how '2000'ish by dwreid · · Score: 1

      Actually, while 60s era mainframes did require significant maintenance by the time the late 70s came around up-time was much better. I still have a late 70s mini-computer that I keep around for laughs that routinely gets about a year and a half between reboots, running 11 users and multi-tasking for each user. As for features that come and go, the IBM 7030 had instruction pipe-lining and look-ahead (what Intel calls hyper-threading) way back in the 60s. In fact it could have as many as 11 instructions in the pipeline at any time. (Though 4 was typical) That went away in the era of the microprocessor, not because it was a bad idea, but because it wasn't possible to implement in early primitive Intel processors. Only after technology caught up again did it reappear as hyper-threading.

    9. Re:Wow .. how '2000'ish by timeOday · · Score: 1

      The tradeoff between centralized and decentralized computing is a perfect example of a situation where the technology is constantly evolving at a rapid pace. Whether it's better to have a mainframe, a cluster, a distributed cluster (cloud), or fully decentralized (peer-to-peer) varies from application to application and from year-to-year. None of those options can be ruled in or out by making generalizations from the year 2000, let alone the 1960's.

    10. Re:Wow .. how '2000'ish by afabbro · · Score: 1

      Eh, machines of that era required constant manual supervision, and uptime was measured in hours, not months or years.

      I'm not sure what datacenter you were working in, but in general that is quite untrue.

      --
      Advice: on VPS providers
    11. Re:Wow .. how '2000'ish by R-ballbat · · Score: 1

      Which ones were those? Everyone I've seen has some requirements for preventive and corrective maintenance. Minor troubles can be ignored and they don't need someone on site full time or even regularly scheduled but the so did a community dial office from the '50's.

    12. Re:Wow .. how '2000'ish by Geminii · · Score: 1

      Of course, in the IT industry, 'some change in technology' comes along every week.

  5. I luv AOL! by jimpop · · Score: 1

    Seriously. AOL keeps my relative's PC experience safe; which, generally, keeps them from bugging me for help. :-)

  6. Who? by Jailbrekr · · Score: 3, Insightful

    Seriously though, most telcomm operations operate like this. Their switching centers are all fully automated and unmanned, and usually in the basement of some non descript building. This is nothing new.

    --
    Feed the need: Digitaladdiction.net
    1. Re:Who? by rickb928 · · Score: 2

      Um, I wouldn't be comfortable my telcomm's switching centers in basements. These are moct commonly the first room to flood when the water comes, and telcomm, switches are everywhere their users are.

      I see telcomm switches housed above ground, in plain, sometimes unmarked buildings. There's one a quarter mile from my house, and I drive by two others to go to work. If they have basements, I bet that's where they keep stuff that doesn't matter as much.

      And the huge switch that used to work in my old hometown, one of the last crossbar switches in the U.S. to convert to ESS. It was deafening in there, and the basement was empty. Six floors of relays going constantly. The mice ate the insulation like it was licorice. Putting any of that in the basement would be wrong, even if it was built on a hill.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    2. Re:Who? by h4rr4r · · Score: 1

      The building I am in hosts one such setup in the basement. It never floods at my location.

    3. Re:Who? by dwreid · · Score: 1

      Actually that's not true. Equipment of this type was and is routinely stored in basements as well as entire buildings. I know because I've worked on them for years.

  7. It was to be staffed... by haus · · Score: 1

    .. but there last geek quite, so now the data center must fend for itself.

    1. Re:It was to be staffed... by Anonymous Coward · · Score: 1

      Spelling. You fail it.

    2. Re:It was to be staffed... by haus · · Score: 2

      You do realize that this story is about AOL, correct spelling would simply be out of plase.

  8. What by Dunbal · · Score: 3, Funny

    AOL still exists? Wow. Yeah ok I guess this is the result of years of beancounter thinking - the expensive part of running the service and the reason they were losing money was the IT staff, huh? Glad I closed my CompuServe account before giving these guys any money.

    --
    Seven puppies were harmed during the making of this post.
    1. Re:What by jgotts · · Score: 1

      Instead of $15/hour techs working for AOL doing regular maintenance they've switched to outside contractors billing at $100-200/hr when the shit hits the fan. I don't think this idea is going to work very well.

    2. Re:What by Synerg1y · · Score: 2

      The contractors warranty their work :) Sometimes makes all the difference, the $15/h tech is just miserable usually.

    3. Re:What by billcopc · · Score: 4, Informative

      How often does shit hit the fan in that sort of environment ?

      As a hybrid techie who does a lot of hardware work, I would much rather go in once a month, fix a batch of issues in one visit, collect my fat cheque and go back to the pub, than spend 40+ hours a week playing Bejeweled, waiting for stuff to break.

      I would expect AOL's strategy to greatly reduce costs, because that $15/hr rack monkey costs a lot more than $15/hr in the end. They have benefits, you have to "manage" them, they need human comforts like bathrooms, cleaning, seating, heating/air, lunch room. From an efficiency standpoint, the contractor route is more efficient in both money and time.

      --
      -Billco, Fnarg.com
    4. Re:What by hedwards · · Score: 1

      Depends, how confident are you that every eventuality has been planned for and provided for by the system? A significant outage can easily eat up an entire years worth of $15 an hour salaries if you hit an unforeseen condition which causes the whole data center to go down. Sure it's unlikely if the people doing the planning know what they're doing, but I'm sure that the folks in the WTC weren't expecting their records to be destroyed by a terrorist attack taking the entire building down.

    5. Re:What by maxwell+demon · · Score: 1

      Depends, how confident are you that every eventuality has been planned for and provided for by the system? A significant outage can easily eat up an entire years worth of $15 an hour salaries if you hit an unforeseen condition which causes the whole data center to go down. Sure it's unlikely if the people doing the planning know what they're doing, but I'm sure that the folks in the WTC weren't expecting their records to be destroyed by a terrorist attack taking the entire building down.

      Of course any number of $15/h techs in the WTC wouldn't have helped them with this problem anyway.

      --
      The Tao of math: The numbers you can count are not the real numbers.
    6. Re:What by Snoggle · · Score: 1

      AOL still exists? Wow.

      AOL has been riding the profits from its dying dialup business to relaunch itself as an internet content and advertising business. They actually have a fairly large stable of brands besides their own Autos or Shopping channels. Poking around their corporate site, here is a list of sites they operate: 5min, advertising, about.me, adtech, AIM, autoblog, cambio, citysbest, comicsalliance, dailyfinance, engadget, everydayhealth, gadling, games, giseleandthegreenteam, goodnewsnetwork, goviral, holidash,huffington post, joystiq, jsyk, kitchendaily, mapquest, mmafighting, moviefone, noisecreep, patch, pawnation, pictela, seed, shortcuts, shoutcast, slashcontrol, smckids, spinner, streampad, studionow, stylist, theboombox, theboot, techcrunch, tuaw, tuvozentuvida, winamp, wow and probably a bunch of others I couldn't find. While some of these might not be top of mind, combined they represent a huge pile of pageviews and traffic. There is also a nice set of investments in new sties tracked on crunchbase: http://www.crunchbase.com/financial-organization/aol-ventures

    7. Re:What by billcopc · · Score: 1

      So, how do you suggest one should plan against supposed terrorists razing the whole building ?

      More to the point: how is the $15 lackey going to make a difference in that scenario ? If nothing else, NOT having the lackey there saves the company from paying out death benefits :D

      --
      -Billco, Fnarg.com
  9. no security or maintenance? by Joe_Dragon · · Score: 1

    Seem like it may take time for any one to come to the site for any thing vs have a few people on site to get to stuff quicker.

    1. Re:no security or maintenance? by EdIII · · Score: 3, Insightful

      The whole idea is not to need to get to stuff quicker at all.

      If you are:

      1) Completely virtualized.
      2) Use power circuits that are monitored for load, on a battery back up, power conditioners, and diesel fuel generators for local utility backup.
      3) Use management devices to control all your bare metal as if you are standing there, complete with USB connected storage per device that you can swap out the iso for.
      4) Have redundancy in your virtualization setup that allows you to have high availability, live migration, automated backups, etc.

      What you get is an infrastructure that allows you to route around failures and schedule hardware swap outs on your own timetable, which can be far more economical.

      If you don't have that then it does involve costly emergency response at 2am to replace a bare metal server that went down. You either pay somebody you have retained locally to do it, or you are the one driving down to the datacenter at 2am to do the replacement yourself with who-the-heck-knows how long it will take with uptime monitoring solutions sending out emails like crazy to the rest of the admin staff, and heavens help you, some execs that demanded to be in the loop from now on due to an "incident".

      Don't know about you..... but I would rather be able to relax at 10pm and have a few beers once awhile (to the point I can't drive) without worrying about bare metal servers going down all the time, or who is on call, etc.

  10. What is AOL again. ..? by SplatMan_DK · · Score: 1


    I'm from Europe. What is AOL again? And what is its/their significance in 2011/2012 anyway?

    - Jesper

    --
    My security clearance is so high I have to kill myself if I remember I have it...
    1. Re:What is AOL again. ..? by SwedishChef · · Score: 3, Funny

      I thought everyone knew... AOL is the Internet.

      --
      No one ever had to evacuate a city because the solar panels broke!
    2. Re:What is AOL again. ..? by Moridineas · · Score: 2

      They suck. They just suck differently now. They've switched from being an ISP to being a content company (and most of their content creators seems rather disgruntled). Mostly US-based, but most slashdotters should recognize names like TechCrunch or primarily HuffPo...the rest, not so much.

    3. Re:What is AOL again. ..? by Osgeld · · Score: 1

      They have decent TV program listings

    4. Re:What is AOL again. ..? by maxwell+demon · · Score: 1

      AOL is a service which was providing you with free CDs for decorative purposes. It was, however, a bad idea to put them into you computer's CD drive.
      And yes, they also operated in Europe.

      --
      The Tao of math: The numbers you can count are not the real numbers.
    5. Re:What is AOL again. ..? by burris · · Score: 1

      They are formerly America's premier manufacturer and distributor of coasters.

  11. In other news by mccrew · · Score: 2

    In other news, the rest of AOL is expected to go "lights out" any time now.

    --
    Hey, Windows users, there is no such thing as "forward" slash, there is only slash and backslash.
  12. Huh? by frisket · · Score: 1

    AOL? Who they?

  13. Re:Offtopic, but IT workers? by Synerg1y · · Score: 1

    Really? I'm a bit of a hybrid in terms of tasks, but I've gotten...

    1. a lot more offers for admin positions (might have more to do w/ my presentation though)
    2. better salary offers on coding positions

    Thinking of just me, it seems to be better to stay w the code, especially web development, those are always in demand.

    I'd take being a part of an admin team over a coding team anyway though, prolly need more experience before I start getting offered those w/o actively seeking them and getting no reply :)

  14. I apologize in advance, by The+Yuckinator · · Score: 1

    But I can't resist.
     
    ...In Soviet Russia, remote hands are YOURS!

  15. Pretty easy. by ewhenn · · Score: 1

    It's pretty easy to automate a bunch of off switches. ;)

  16. Works very well. by 140Mandak262Jamuna · · Score: 2

    The new data center with 0 head count matches nicely the AOL user base with 0 head count!

    --
    sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
  17. Two points. by rickb928 · · Score: 3, Insightful

    One - If there is redundancy and virtualization, AOL can certainly keep services running while a tech goes in, maybe once a week, and swaps out the failed blades that have already beeen remotely disabled and their usual services relocated. this is not a problem. Our outfit here has a lights-out facility that sees a tech maybe every few weeks, and other than that a janitor keeps the dust bunnies at bay and makes sure the locks work daily. And yes, they've asked him to flip power switches and tell them what color the lights were. He's gotten used to this. that center doesn't have state-of-the-art stuff in it, either.

    Two - Didn't AOL run on a mainframe (or more than one) in the 90s? It predated anything useful, even the Web I think. Netscape was being launched in 1998, Berners-Lee was making a NeXT browser in 1990, and AOL for Windows existed in 1991. Mosaic and Lynx were out in 1993. AOL sure didn't need any PC infrastructure, it predated even Trumpet Winsock, I think, and Linux. I don't think I could have surfed the Web in 1991 with a Windows machine, but I could use AOL.

    --
    deleting the extra space after periods so i can stay relevant, yeah.
    1. Re:Two points. by laffer1 · · Score: 1

      Netscape was founded in 1994. http://en.wikipedia.org/wiki/Netscape

      --
      MidnightBSD: The BSD for Everyone
    2. Re:Two points. by lakeland · · Score: 1

      AOL was already famous for being a good source of free floppies in the early 90s, and a search on wikipedia confirms they were renamed to AOL and expanded in '89.

      They were doing graphical forums in '86, almost 10 years before Netscape.

    3. Re:Two points. by evilviper · · Score: 1

      It predated anything useful, even the Web I think. Netscape was being launched in 1998, Berners-Lee was making a NeXT browser in 1990, and AOL for Windows existed in 1991.

      The web was around, and in-force MUCH earlier than you would imagine. Windows 98 had Internet Explorer version 4 inextricably linked to the OS. Not version 1, but version 4. Internet Explorer was concieved as a weapon against Netscape, so there's no way IEv4 predated Netscape...

      And before the WWW, the internet was quite useful. Newsgroups, FTP sites, and Gopher sites contained a lot. Many people here were downloading floppies of Slackware Linux back then...

        I don't think I could have surfed the Web in 1991 with a Windows machine, but I could use AOL.

      No, you couldn't because NOBODY had Windows in 91. Everyone was running on MS-DOS. I still remember the Compuserve and Prodigy login-screens from their old DOS apps. Trumpet Winsock is irrelevant in the DOS days.

       

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
    4. Re:Two points. by bill_mcgonigle · · Score: 1

      No, you couldn't because NOBODY had Windows in 91.

      What in the world are you talking about?

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
    5. Re:Two points. by Jay+L · · Score: 2

      AOL initially ran on a network of Stratus fault-tolerant minicomputers, each running two to eight 680x0 CPUs. Later we added unix boxen, some beefy SGIs and HPs for servers, and Suns for front-end telco interfacing IIRC. By the mid-90s we grew a Tandem fault-tolerant cluster for our critical databases; it did hot component failover, multimaster replication, all
      the stuff that's common today, but
      with SQL down in the drive controller for blazing speeds. We didn't really
      start moving to a PC-based architecture until the late '90s, when
      Linux provided cheap, reliable enough workhorses, and helped drive the
      big Iron prices down too

    6. Re:Two points. by Jay+L · · Score: 1

      Wow. I will never post from an iPhone again...

    7. Re:Two points. by rickb928 · · Score: 1

      I was thinking of the browser, not the company.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    8. Re:Two points. by rickb928 · · Score: 1

      Wow. We're still two years from decomissioning our Stratus servers. We're still 6 months from decom of SNA. I gotta talk to the other team about stepping it up.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    9. Re:Two points. by dkf · · Score: 1

      No, you couldn't because NOBODY had Windows in 91.

      '91 was when Win 3.1 came out, and that was when it was becoming obvious that Win really was evolving to becoming a full-time OS. (It wasn't there yet at the time, oh boy it wasn't there, but it was clear that was the way things were going.) Surfing the web at that time (well, info services like gopher) required third-party software, but it definitely existed. I remember using it.

      --
      "Little does he know, but there is no 'I' in 'Idiot'!"
    10. Re:Two points. by laffer1 · · Score: 1

      Netscape didn't come out in 1998. Netscape Navigator 3 was out in 1997 for instance http://sillydog.org/narchive/full123.php

      I was using Netscape Navigator 2.x with AOL in 1996. I remember because it was a big deal that AOL finally got 32bit winsock support for windows 95. Netscape was definitely out in 1995 as well. I remember "best vieweed with netscape" buttons on websites when I first got on AOL in 1995.

      Are you talking about a specific browser version? Like Netscape Communicator 4.0 ?

      Both Internet Explorer and Netscape were available in 1996. IE was based on NCSA Mosaic.

      --
      MidnightBSD: The BSD for Everyone
    11. Re:Two points. by rickb928 · · Score: 1

      I'm thinking of Mosaic browsers. Before Windows 3.11.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    12. Re:Two points. by Jay+L · · Score: 1

      Are you running VOS or FTX? I don't know about FTX, but if you're running VOS, and you're (at least) two years out, I highly recommend upgrading to the V-Series. Stuff that used to compile overnight now takes seconds; we stopped building an inverted index of our source code because "display *.pl1 -match x" was instant. More on the port:

      http://newsgroups.derkeiler.com/Archive/Comp/comp.sys.stratus/2007-11/msg00005.html

    13. Re:Two points. by rickb928 · · Score: 1

      We're killing all of them. They don't fit into the new software models, and are actually 3 years overdue for decommissioning. We have no redundancy on 75% of them, and their replacements are already online and on production. It's our users who are holding this up, some have put off their work for 5-6 years now, and we don't have the power to compel them to do it. Yet.

      Good while they worked, still there, but doomed. They mostly do file transforms and routing, much better on the RHEL system replacing them.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    14. Re:Two points. by evilviper · · Score: 1

      '91 was when Win 3.1 came out

      Nope, March of '92. Others have pointed out that Windows 3.0 was out at that time, but I still maintain practically nobody was running it. In 91 it was very much a DOS world.

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  18. Re:Offtopic, but IT workers? by aix+tom · · Score: 3, Insightful

    The software still needs to be written. The programs still need to be run somewhere.

    Technically not much has changed. The "Cloud" is still made up of servers that have to be administered. The main effect is that the IT and network admins will have to keep up with technology, especially the new virtualization layers between the hardware and the running application. But keeping up to date has always been a part of working in IT.

  19. Re:I'm still expecting their datacenters by 517714 · · Score: 1

    Zero electricity would have been achieved if the last technician had turned the lights out when he left.

    --
    The US government have made it clear that we have no inalienable rights; any we do not defend vigorously will be taken.
  20. AOL Needs a Data Center? by Trip6 · · Score: 2

    Oh yeah, to house all the dial-up modems...

    --
    I hate being bipolar; it's awesome!
  21. Amazing! by MAXOMENOS · · Score: 1

    I didn't know AOL even still existed!

    --
    Finding God in a Dog
  22. really? by geekoid · · Score: 1

    AOWho?

    --
    The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
  23. Me too by Pezbian · · Score: 1

    n/t /obligatory

    --
    In a world of the blind, the one-eyed man is king--and the two-eyed man is a heretic.
  24. how does redundancy help you when the main power by Joe_Dragon · · Score: 1

    how does redundancy help you when the main power switch goes down / on fire and there is no one there. Let's see firemen make a big mess and no is there to start the rebuild or it may just do a safe shutdown just to send some out just to find out you need to call in this other guy to fix the switch or generator.

  25. Re:how does redundancy help you when the main powe by ToddDTaft · · Score: 1

    how does redundancy help you when the main power switch goes down / on fire and there is no one there

    If you are a big enough operation, you have redundancy at the data center level. i.e. you can lose an entire data center and have no loss of service on your production applications. Other than a possible speed/performance degradation, your average customer has no knowledge that anything bad has happened.

  26. Datacenter in a box by lucm · · Score: 1

    At least that way they won't need "heroic support"

    --
    lucm, indeed.
  27. Wow, is AOL still around? by QuietLagoon · · Score: 2

    What are they doing nowadays that requires multiple servers?

    1. Re:Wow, is AOL still around? by archen · · Score: 1

      They still serve email. My boss (and much of his family) still use (and pay) for AOL even though they have broadband and AOL provides them with nothing but an email address as far as I can figure. It's apparently hilariously bad, as he's always talking about how the website doesn't work much of the time, and the connection simply times out. I think they also distribute some software that goes with "AOL" but I have no idea what it does. I hear it still crashes a lot though.

    2. Re:Wow, is AOL still around? by Geminii · · Score: 1

      Counting all the money they make off suckers?

  28. As an Operator currently working inside a DC... by Datamonstar · · Score: 1

    ... I say FUUUUUUUUUUUUUUUuuuuuu...

    --
    The eternal struggle of good vs. evil begins within one's self.
  29. And they named it.... by Gimbal · · Score: 1

    ....wait for it .... Smynet! (Someone typoed)

  30. They're on the Time Warner life support by Gimbal · · Score: 1

    ...and Daddy Warbucks got some dough - in a manner of speaking, as it were, etc und so weiter.

  31. that's what geographic redundancy is for by Chirs · · Score: 1

    This is why you have a duplicate data center in another city that is kept in standby and is just sitting there ready to take over. (Actually, you normally have a mix of services active at either location.)

    The company I work for makes telecom equipment, and supporting geo redundancy is a fairly key requirement for some major customers.

  32. Hope they don't have rats by bryan1945 · · Score: 1

    To start chewing through wires, causing power outages, starting fires, pooping in the mailbox, that kind of stuff.

    --
    Vote monkeys into Congress. They are cheaper and more trustworthy.
  33. One of the early search engines did this. by Animats · · Score: 1

    One of the early search engines, I think Infoseek, worked this way. Machines were installed in blocks of 100 (this was before 1U servers) and never replaced individually. Failed machines were powered off remotely. When some fraction of the block had failed, about 20%, the whole cluster was replaced.

    There's a lot to be said for this. You have less maintenance-induced failure. Operating costs are low.

  34. Grid by 1s44c · · Score: 1

    ...over a decade before the term Cloud was even coined.

    You mean back when it was called 'grid'?

  35. Not much to see here by LordFolken · · Score: 1

    What they did:
    * Modularize/Standardize Infrastructure, e.g. storage & computing power
    * Build provisioning systems
    * Virtualize everything

    When they say that they are flexible, they mean that they have a lot of dark hardware lying around.