Slashdot Mirror

← Back to Stories (view on slashdot.org)

Patch the Linux Kernel Without Reboots

Posted by kdawson on from the click-n-go dept.

evanbro writes "ZDNet is reporting on ksplice, a system for applying patches to the Linux kernel without rebooting. ksplice requires no kernel modifications, just the source, the config files, and a patch. Author Jeff Arnold discusses the system in a technical overview paper (PDF). Ted Ts'o comments, 'Users in the carrier grade linux space have been clamoring for this for a while. If you are a carrier in telephony and don't want downtime, this stuff is pure gold.'" Update: 04/24 10:04 GMT by KD : Tomasz Chmielewsk writes on LKML that the idea seems to be patented by Microsoft.

69 of 286 comments (clear)

  1. Needed that bad? by MetalliQaZ · · Score: 5, Insightful

    If you are a carrier in telephony, you should have many load-balanced servers that can be taken offline one at a time and restored after patching. They probably would be taken out of the loop for the in-place patching anyway. So who is "clamoring"?

    --
    "Here Lies Philip J. Fry, named for his uncle, to carry on his spirit"
    1. Re:Needed that bad? by tgatliff · · Score: 2, Funny

      I guess a better way to put it would be "oh... Way Cool!!!!"... :)

      Meaning, yes I agree that in most cases it is not needed, but I have internal processing servers that have up times of over 3 years, so if I had something like this probably all my servers would have up times of this long..

    2. Re:Needed that bad? by Chris+Burke · · Score: 3, Interesting

      If you are a carrier in telephony, you should have many load-balanced servers that can be taken offline one at a time and restored after patching.

      Two things:

      The very fact that there is load balancing means that every server is likely to have active connections going through it. If you currently have connections going through a specific server, you don't want to drop those connections in order to reboot that particular machine. This allows updates to a live machine.

      Second, this is telephony, meaning it is the infrastructure on which the internet is based. There's no dns tricks or tcp/ip you can use to send people to a different "server" if that server is the switch connected to your fiber backbone. Basically, there are points in the infrastructure where there are by necessity a single chokepoint.

      As to how often these things collide, and how much of a pain it is to actually stop a server for some amount of time, I can't say. But I can see situations where being able to hot-swap a kernel would be useful.

      --

      The enemies of Democracy are
    3. Re:Needed that bad? by garlicbready · · Score: 3, Interesting

      I was about to say another idea might be virtulisation via xen for example
      start up a new virtual machine with the new kernel, then when your sure it's working, just switch everything across from the old to the new, and shut down the old virtual instance

    4. Re:Needed that bad? by Iphtashu+Fitz · · Score: 4, Informative

      The very fact that there is load balancing means that every server is likely to have active connections going through it. If you currently have connections going through a specific server, you don't want to drop those connections in order to reboot that particular machine. This allows updates to a live machine.

      If you have a load balanced environment then you have the ability to redirect new connections away from a given server. Then it's just a matter of waiting for the active connections to terminate before the machine ends up in an idle state where you can safely apply patches offline. I've worked in a number of telephony environments and this was always the way we would patch systems. Stop accepting new connections, wait for existing ones to end, then perform the patch, reboot, verify, and start accepting connections again.

      Second, this is telephony, meaning it is the infrastructure on which the internet is based. There's no dns tricks or tcp/ip you can use to send people to a different "server" if that server is the switch connected to your fiber backbone. Basically, there are points in the infrastructure where there are by necessity a single chokepoint.

      Any mission critical hardware, switches, routers, servers, etc. should be set up in redundant pairs (or triplets, ...) so that if a hardware failure occurs the remaining hardware can keep the service up. Single points of failure are avoided like the plague in datacenters that require 100% uptime. Part of that is to deal with hardware failures but part is also to provide an ability to perform software/firmware upgrades when necessary. Once again, you migrate all traffic off the system you're upgrading then apply the upgrades offline. Upgrading a kernel, especially, in an online environment, is something virtually any sysadmin would want to avoid if at all possible.

      Redundancy is key, and any commercial datacenter will offer it all the way from their connections to the outside world to the connections they provide their customers. Every datacenter used by every company I ever worked for (about 10) offered redundant power and redundant network drops (using HSRP, VRRP, etc) for our equipment. If the datacenter needed to upgrade a router they'd move all traffic off one router so they could upgrade and test it, then move traffic off the other and repeat the process. Similarly if we needed to upgrade our firewalls, switches, etc. we'd fail over to the second redundant device first. In some cases we had bonded interfaces right on the end servers so as long as one path remained active we could power down an entire switch, router, firewall, etc. In other cases we relied on load balancing across servers that were alternately connected to one or another switch.

    5. Re:Needed that bad? by Paul+Carver · · Score: 3, Insightful

      If your load balancer can't take a server out of the pool while allowing current sessions to finish cleanly then you need to shop for a new load balancer.

      A decent load balancer will obviously give you the choice of whether to take a server out of service immediately disrupting existing sessions or simply stop sending new sessions to it while allowing existing sessions to continue.

      As for your comment about physical connections, that's what portchannels and multilink trunks are for. Or VRRP and HSRP depending on which level of "connected to" you mean.

    6. Re:Needed that bad? by QuantumRiff · · Score: 2, Interesting

      But what about the servers that are placed in remote sites like small cell towers, where space, and backup power are critical issues.

      --

      What are we going to do tonight Brain?
    7. Re:Needed that bad? by jelle · · Score: 5, Insightful

      So you take it out of rotation on the load balancer and give it a few minutes to complete all its active connections. Patch/reboot whatever. Bring it back into rotation, and repeat with the other box.

      Methods like that usually suck in real-life, because right the day before you want to 'take it out of rotation', a circuit is opened through it that requires five nines (so you can't drop it), and it will remain open for months...

      You will end up with 99 boxes waiting to 'get out of rotation' for every
      single box that you don't need to update...

      Murphy will make sure of that.

      --
      --- Hindsight is 20/20, but walking backwards is not the answer.
    8. Re:Needed that bad? by Colin+Smith · · Score: 2, Interesting

      The very fact that there is load balancing means that every server is likely to have active connections going through it http://conntrack-tools.netfilter.org/

      I hot-swap whole networks.

      HTH.

      --
      Deleted
    9. Re:Needed that bad? by Anonymous Coward · · Score: 5, Insightful

      I have internal processing servers that have up times of over 3 years

      I've never understood this boasting about uptime. Long uptimes are a bad thing! How do you know a configuration change hasn't rendered one of your startup scripts ineffective? If you have to reboot for some unexpected reason, you could be stuck debugging unrelated problems at very inopportune moments.

      You need to schedule regular reboots so that you can test that your servers can start up fine at a moment's notice. Long uptimes are a sign a sysadmin hasn't been doing his job.

    10. Re:Needed that bad? by Nkwe · · Score: 3, Informative
      Then it's just a matter of waiting for the active connections to terminate before the machine ends up in an idle state where you can safely apply patches offline.

      This assumes that active connections will terminate in a timely fashion. I used to have internet service via an ISDN via a connection to my office. My ISDN calls would connected for a couple of months at a time. Yes, one connection lasting multiple months. There are other cases where a connection, context, or state between two systems would need to be maintained for extended periods of time. Many of these situations can not be solved by load balancing and would benefit greatly by the ability to make kernel changes without interrupting current work, or waiting for it to complete.

    11. Re:Needed that bad? by mr_mischief · · Score: 4, Informative

      If you change something in a configuration that requires a change to the startup script, then you also change the startup script.

      A patch to the kernel almost never requires changes to startup scripts. They're not talking about adding new functionality with user-space-addressable interfaces with this tool. They're talking about being able to install about 84% of security hotfixes in a hurry outside your scheduled reboots then rebooting on your regular maintenance schedule.

    12. Re:Needed that bad? by mr_mischief · · Score: 3, Interesting

      Yes, but you end up taking one machine out at a time, giving it time to simmer down, and then bringing it back up. If you have 100 boxes and it takes 30 minutes to simmer down to idle, you have 30 * 100, or 3,000 minutes to do the upgrades. On average, your boxes go unpatched for 1500 minutes.

      So you have this security hotfix you really want to apply, but it's going to 25 hours on average to fix a box and 50 hours to fix them all.

      You could, with ksplice and a good concurrent control system, make your average time to fix 5 minutes in over 80% of kernel upgrade scenarios rated "Critical". Your boxes could still be rebooted on a regular basis later.

      Which do you prefer?

    13. Re:Needed that bad? by Kookus · · Score: 5, Insightful

      Production systems are not for testing purposes. You want to test rebooting? Do it on a test box.

    14. Re:Needed that bad? by mopower70 · · Score: 2, Informative

      A configuration change that renders a start up script ineffective is a sign that your sysadmin hasn't been doing his job, and that you have no concept of change control in your environment.

      We have systems that run for years at a time because we have change management tools that guarantee that those systems are in the exact state of configuration they should be in, and these tools run every night. If you're running around making undocumented configuration changes that even have a ghost of a chance of affecting server operation, anyone that gave you root access needs to have their fingers shortened.

    15. Re:Needed that bad? by Kymermosst · · Score: 4, Insightful

      How do you know a configuration change hasn't rendered one of your startup scripts ineffective?

      Isn't that what QA systems and effective approaches to change management are supposed to handle?

      If I am planning a change, I should discover problems with the startup scripts in QA, not in production, especially if a production reboot is not required to implement the change.

      --
      "Alcohol, Tobacco, Firearms, and Explosives" should be a convenience store, not a government agency.
    16. Re:Needed that bad? by adrianbaugh · · Score: 4, Insightful

      How do you know that your test boxes are configured precisely identically to the production boxes?

      dd your production box's system filesystems to another hard drive, put in an identically specced machine, boot that?

      --
      "'I pass the test,' she said. 'I will diminish, and go into the West, and remain Galadriel.'"
      - JRR Tolkien.
    17. Re:Needed that bad? by mOdQuArK! · · Score: 4, Informative

      There's a difference between what YOU as an end user consider to be an open connection, and what the telecom equipment consider as a connection.

      For all you know, your apparent always-on connection was actually a virtual connection being frequently switched & reswitched over many different real physical connections. That would be a fairly standard architecture for having a network infrastructure which can have components being worked on while data is still flowing through the network.

      When the telecom provider is "waiting for active connections to go away" on a particular device only means that all of the virtual connections that are momentarily being switched that device have been successfully switched to another device. It doesn't mean that any of those virtual connections have to be terminated.

    18. Re:Needed that bad? by Anonymous Coward · · Score: 2, Interesting

      Like that. Kookus' point applies to the Telco environment quite well. The grandparent's post asks:

      >How do you know a configuration change hasn't rendered one of your startup scripts ineffective?

      And the answer is: Because every configuration change that happens in the production environment happened many times in the lab on an identical machine, and if there was any danger of this occurring then an image of the production machine would be made "hot" during the maintenance window, applied to the lab system, and the config change applied there, and the system thoroughly tested, including reboots.

      For those of us with a dozen boot options on our grub screen the danger of a "change that makes the system reboot into an odd state." May seem very real, but in production Telco environments it's been factored out of the equation.

    19. Re:Needed that bad? by smallfries · · Score: 2, Informative

      True, but I've been standing in switch rooms watching operators manually kill those circuits because they wanted to reboot a box. 5x 9s doesn't mean perfect service, and if anyone complained about it they were told that a ms interruption once every few months was in their SLA. By the time they reconnected they went through another box so how were they to know it was any longer than that.

      --
      Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
    20. Re:Needed that bad? by scott_karana · · Score: 2, Insightful

      You could also virtualize over a network file system. Removes the need for 1:1 identical machines. :)

  2. Unless it fails. by Joe+Snipe · · Score: 2, Insightful

    honestly how much downtime are we talking here? 30 seconds?

    --
    Sometimes, life itself is sarcasm...
    1. Re:Unless it fails. by Anonymous Coward · · Score: 4, Funny

      honestly how much downtime are we talking here? 30 seconds? well, think about the fsck that happens after 180 days or 30+ mounts ?
    2. Re:Unless it fails. by UnknowingFool · · Score: 2, Informative

      For, your average computer and generic linux servers the downtime is small. But companies often have applications that they need to restart. That is the difference. Also linux is used on equipment other than generic servers: embedded systems, etc where loading isn't optimized cause the equipment should never go down.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    3. Re:Unless it fails. by Tychon · · Score: 4, Informative

      A company that I once had dealings with was quite proud of their five nines. The motivation? It cost them $18,000 per second they were down. 30 seconds isn't just 30 seconds sometimes.

    4. Re:Unless it fails. by m50d · · Score: 2, Insightful

      Uh, if you actually need that, then you needed it anyway. And if you don't need it but don't know how to disable it, you shouldn't be running a system.

      --
      I am trolling
    5. Re:Unless it fails. by Random+Destruction · · Score: 2, Interesting

      Well, care of google:
      100 - (((30 seconds) / (1 year)) * 100) = 99.9999049

      So if you're trying to keep up 6 9s for some super critical system, you've just used a years worth of downtime.

      Even for lower numbers of nines, you still don't get many minutes per year for patching, assuming no hardware failures ever.

      --
      :x
  3. Amazing by cromar · · Score: 4, Interesting

    That is truly amazing tech, right there. It would be interesting to know the security implications of being able to hot-patch the kernel, however.

    1. Re:Amazing by katz · · Score: 5, Funny

      Considering that you don't need to prepare the kernel in any way--just execute the program and bang, it's patched--means that someone with root access could slip a rootkit right under your nose (i.e., without the system administrator being aware of this).

      - Roey

    2. Re:Amazing by KeithJM · · Score: 5, Insightful

      someone with root access could slip a rootkit right under your nose Yeah, someone with root access can take control of your server. Oh, wait, they've got root access. They already have control of your server. At some point, you have to just accept that giving someone root access is a security risk.
    3. Re:Amazing by katz · · Score: 3, Insightful

      My bad, I meant to say,

          "A remote attacker who successfully executes a privilege escalation exploit and gains root access will have an easier time taking control of your server and hiding their tracks".

      Thanks for pointing that out

      - Roey

  4. Wrong way to solve the uptime problem by Anon+E.+Muss · · Score: 4, Insightful

    Trying to keep one server up 24/7/365 is a usually mistake. You'll never achieve 100% uptime. A much better idea is to use clustering and distributed computing so your overall system can survive the loss of individual servers.

    --
    The key sequence to access my Slashdot bookmark in Firefox is Alt-B-S. I don't believe this is a coincidence.
    1. Re:Wrong way to solve the uptime problem by Qzukk · · Score: 5, Funny

      Trust me, that was the first thing they thought of, then the CEO came in and said "Why are you ordering more equipment when we have half of our machines sitting there and doing nothing? We could be doing twice the work/traffic/whatever without paying more money!"

      --
      If I have been able to see further than others, it is because I bought a pair of binoculars.
    2. Re:Wrong way to solve the uptime problem by N1ck0 · · Score: 3, Informative

      Mainly why people in the telecom industry have been clamoring for it. Its very difficult to take over the termination of a circuit switched system without some interruption for the end user. And its also not aways easy to busy out all channels on a line as calls drop off so you can free up a machine for patching.

      Of course many of the reasons is a lot of commercial telecom apps are badly implemented and need better management controls.

    3. Re:Wrong way to solve the uptime problem by trybywrench · · Score: 4, Insightful

      Trying to keep one server up 24/7/365 is a usually mistake. You'll never achieve 100% uptime. A much better idea is to use clustering and distributed computing so your overall system can survive the loss of individual servers. People using Linux on BigIron(tm) bank on 24/7/365/25years uptime. When a single server costs hundreds of thousands or millions of dollars you can't afford a spare sitting idle. From day 1 the server needs to be making money and never ever stop. For smaller general purpose servers like you can buy at Dell.com then yeah having a fail-over makes sense.
      --
      I came to the datacenter drunk with a fake ID, don't you want to be just like me?
    4. Re:Wrong way to solve the uptime problem by Anonymous Coward · · Score: 4, Funny

      If he knew anything, he wouldn't be the CEO.

    5. Re:Wrong way to solve the uptime problem by Anon+E.+Muss · · Score: 2, Insightful

      People using Linux on BigIron(tm) bank on 24/7/365/25years uptime.

      I doubt there are many people running Linux on true Big Iron. I'm not saying it doesn't happen, I'm saying that most Big Iron runs something else. I know many financial institutions and telecom operators use HP NonStop systems. These can stay up 24/7/365/25years, and you pay millions of dollars for that. They have full redundant hardware inside the box, run a proprietary OS, and proprietary applications.

      --
      The key sequence to access my Slashdot bookmark in Firefox is Alt-B-S. I don't believe this is a coincidence.
    6. Re:Wrong way to solve the uptime problem by Anonymous Coward · · Score: 2, Insightful

      Now is not the time to claim banks know what they are doing.

    7. Re:Wrong way to solve the uptime problem by Ed+Avis · · Score: 2, Insightful

      Who cares about servers? I want my Linux desktop to stay up-to-date with security fixes without having to reboot it every few days.

      --
      -- Ed Avis ed@membled.com
    8. Re:Wrong way to solve the uptime problem by mr_mischief · · Score: 3, Informative

      Can we please kill the 24/7/265 phrasing? Where do you people live that there are 365 weeks in a year?

      Why not 24/7/52 or 24/7/4.3/12 or just 24/365 (or 24/365.242 for the pedants).

    9. Re:Wrong way to solve the uptime problem by afabbro · · Score: 2, Insightful

      I doubt there are many people running Linux on true Big Iron.

      And you would be wrong. Sure, most mainframes are running z/OS, but a goodly number of them are also running Linux images. I don't know the percentages but the IBM "run Linux on your mainframe" training classes are usually full.

      --
      Advice: on VPS providers
  5. Already been used by caluml · · Score: 4, Informative

    There was a kernel exploit recently where someone submitted a patch that modified the running kernel using this technology. It didn't work for me, so I had to resort to patching the .c that was affected - but a lot of people reported that it worked.

    --
    Get your own free personal location tracker
    1. Re:Already been used by ThisNukes4u · · Score: 2, Informative

      IIRC, that code was actually a modified version of the exploit where the payload was changed to fix the exploit instead of spawn a root shell. Pretty fucking ingenious if you ask me.

      --
      thisnukes4u.net
  6. Re:Maybe... by CogDissident · · Score: 5, Funny

    I thought their working slogan was:

    Windows 7, it's not awful like Vista!

  7. replace modules by hey · · Score: 2, Interesting

    Rather that a source code level system I'd prefer a way of replacing loadable kernel modules without a reboot. Then push more code into modules -- eg file system. (Hey sounds like a micro-kernel).

  8. The real test... by hal2814 · · Score: 4, Funny

    Can ksplice be installed without rebooting?

    1. Re:The real test... by LinuxDon · · Score: 2, Informative

      It's in the comment: "ksplice requires no kernel modifications"

      So yes, ksplice can be installed/used without rebooting.

  9. Impressive hack by EriktheGreen · · Score: 4, Informative
    For those that haven't read the paper, the technique used is straightforward in concept, but the devil is in the details.

    He basically compiles a patched and unpatched kernel with the same compiler, compares the ELF output, and uses that to generate a binary file that corresponds to the change. That gets wrapped in a generic module for use, another module installs it along with JMPs to bypass the old code and use the new, and he performs the checks needed to make sure he can safely install the redirects.

    He also has to differentiate real changes from incidental ones (the example given is changing the address of a function - all references to it will change, but they don't really need to be included in the binary diff).

    The only human work required is to check whether a patch makes semantic changes to a data structure... whether eg. an unsigned integer variable that was being used as a number is now a packed set of flags - the data declaration is the same, but it's being used differently.

    Interesting paper. Also a useful new set of capabilities for any Linux user who can't handle downtime for quarterly patching... worth its weight in gold in some businesses.

    Erik

    1. Re:Impressive hack by EriktheGreen · · Score: 3, Funny
      Well, let's see.

      A silver dollar, from which bits were commonly cut, weighs about .77 troy ounces.

      Today's gold price as of posting is about $889.95 US per troy ounce.

      A silver dollar was typically cut into 8 bits, which gives us a weight per bit of 0.096 ounces. That translates to about $85.66 per bit weight in gold. Remember, this is per system being patched.

      Since the patches being applied ranged from 1 line to 285 lines per the paper, and a reasonable estimate of compiled average bytes per line is something like 20, we get a value of $13,700 per line of patch in gold. Even for the smaller patches, this is significant. The largest patch would be worth nearly $4,000,000 USD in gold.

      Of course, for 64 bit systems vs. 32 bit, the value would be twice as much :)

      Erik

  10. If it's that critical, shouldn't you have two? by Paul+Carver · · Score: 4, Insightful

    I'd rather have at least two of anything important and have statefull failover between them.

    If you've got this system that's so critical you can't reboot it for a kernel upgrade, what do you do when the building catches fire or a tanker truck full of toxic waste hops the curb and plows through the wall of your datacenter?

    I'd rather have a full second set of anything that critical. It should be in a different state (or country) and have a well designed and frequently used method of seamlessly transferring the load between the two (or more) sites without dropping anything.

    If you can't transfer the workload to a location at least a couple hundred miles away without users noticing then you're not in the big league.

    And as long as the workload is in another datacenter, what's the big deal about rebooting for a kernel upgrade.

    1. Re:If it's that critical, shouldn't you have two? by EriktheGreen · · Score: 2
      In some engineered systems, it just isn't possible to have redundancy in the way you mean.

      Extreme example: Try to design a fail-over for the space shuttle's solid rocket boosters :)

      Interestingly, I've found that the skill needed (and the pay gathered) to deal with systems that can't be made redundant is much higher than that needed to work on "grid" or cluster systems where multiple cheap pieces of hardware are used.

      And they tend to be more reliable too.

  11. Over-engineered solution to a non-existent problem by hacker · · Score: 3, Insightful

    Once again, we have an over-engineered solution to a non-existent problem.

    Any enterprise-level customer is going to have a VERY lengthy Q&A process before deploying anything into production. This includes testing kernels, hardware, networks, interaction, application, data and so on. One pharmaceutical company I know of is federally mandated to do this twice a year, every year, for every single machine that reads, writes or generates data. Period.

    So you hot-patch a running Linux kernel. How do you Q&A that? How do you roll back if the patch fails? Where is your 'control'?

    The answer? A duplicate machine. But wait, if you have two identical machines... isn't that... a cluster?

    Exactly. And THIS is how you perform upgrades. You split the cluster, upgrade one half, verify that the upgrade worked, then roll the cluster over to that node, and upgrade the second portion of the cluster. If you have more machines in the cluster, you do 'round-robin' upgrades. You NEVER EVER touch a running, production system like that.

    Well, not if you want any sort of data integrity or control and want to pass any level of quality validation on that physical environment.

  12. And Microsoft claims to have invented it by davecb · · Score: 3, Informative

    Tomasz Chmielewski wrote on LKML: the idea seem to be patented by Microsoft, i.e. this patent from December 2002: http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching In essence, they patented kexec ;)

    Andi Kleen promptly provided prior art: The basic patching idea is old and has been used many times, long predating kexec. e.g. it's a common way to implement incremental linkers too.

    --
    davecb@spamcop.net
  13. Re:In Soviet Russia, by oodaloop · · Score: 2, Funny

    Let's get the rest of the usual jokes out of the way while we're at it.

    If there were no kernel, it would necessary to create our non-rebooting robot overlords are belong to Chuck Norris.

    --
    Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
  14. Not only the CEO by Moraelin · · Score: 4, Interesting

    Not only the CEO. I lived to see even a hardline IT guy (admittedly, one whose goal in life seems to be to be against whatever you want, and to avoid doing any extra work... actually, make that just: any work) argue along the lines of "nooo, you can't have the servers only 60% loaded! It's a waste of valuable hardware! Why, back in my day (of batch jobs on punched cards, presumably) we had the mainframe used at least an average of 95% before asking for an extra server!"

    It always irks me to see people just not understand concepts like "peak" vs "average", or "failing over".

    - A cluster of, say, 4 machines (small application, really) which are loaded to 90% of capacity, if one dies, the other 3 are now at 120% of capacity each. If you're lucky, it just crawls, if you're unlucky, Java clutches its chest and keels over with an "OutOfMemoryError" or such.

    - if you're at 90% most of the time, then fear Monday 9:00 AM, when every single business partner on that B2B application comes to work and opens his browser. Or fear the massive year-end batch jobs, when that machine/cluster sized barely enough to be ready with the normal midnight jobs by 9 AM, so those users can see their new offers and orders in their browsers, now has to do 20 times as much in a burst.

    Basically it amazes me how many people just don't seem to get that simple rule of thumb of clusters: you're either getting nearly 100% uptime and nearly guaranteed response times, _or_ you're getting that extra hardware fully used to support a bigger load. Not both. Or not until that cluster is so large that 1-2 servers failing add negligible load to the remaining machines.

    --
    A polar bear is a cartesian bear after a coordinate transform.
  15. No, No, No and No again. by Anonymous Coward · · Score: 5, Interesting

    As an admin for some -very- high availability systems, load balancers are not a silver bullet. This solution would most apply for running one-node clusters who are using a single machine as a perimeter network device. (ex. firewall) I see lots of these in the racks at our NOC provider.

    1. We connect to several load balanced systems and the complexity introduced by load balancers translates to inexplicable down time. No load balancers means a pretty steady diet of the latest and greatest server hardware, but no down time. The a few minutes of down time costs more than the server hardware.

    2. High availability translates more roughly into nodes that can fail (ex. power off) and not take the cluster down. This boils down to active-passive application architecture more than just using heartbeat.

    As an FYI, PostgreSQL clustering is a killer application for me. Erlang is also great in many ways, but requires application architecture with active-passive node awareness. Which isn't present in things like Yaws, or even my other favorite non-erlang app nginx. Heartbeat is the solution there, but I'd like to see yaws be cluster aware on its own. http://yaws.hyber.org/

    1. Re:No, No, No and No again. by hab136 · · Score: 4, Insightful

      As an admin for some -very- high availability systems, load balancers are not a silver bullet. This solution would most apply for running one-node clusters who are using a single machine as a perimeter network device. (ex. firewall) I see lots of these in the racks at our NOC provider.

      1. We connect to several load balanced systems and the complexity introduced by load balancers translates to inexplicable down time. No load balancers means a pretty steady diet of the latest and greatest server hardware, but no down time. The a few minutes of down time costs more than the server hardware.

      I spent a decade in perimeter networking at a Fortune 50 US bank. My group didn't do the internal network, just the perimiter, and we still had dozens of network sites and thousands of pieces of equipment. The bank itself has hundreds of thousands of employees, millions of users. Online banking and brokerage are about as high availability as you can get save utilities (power, water, telephony, etc) or military. Seconds of online brokerage downtime equated to millions of dollars lost.

      The idea that load balancing introduces inexplicable down time is completely unsupported by my experience.

      "One-node clusters" seems like marketing speak for "single point of failure". A cluster by definition is two or more nodes.

      Redundant routers, switches, firewalls, the works or you're not high-availability in my opinion. The fact that you're talking about Postgresql instead of Oracle or DB2 on mainframes makes me think that your idea of high availability is different than mine.
    2. Re:No, No, No and No again. by Anonymous Coward · · Score: 2, Funny

      one-node clusters

      Of all the industry oxymorons out there, this is one of the most annoying.

      One-goose gaggle.

      Single-wheel bicycle.

  16. Sorry... by PJ+The+Womble · · Score: 2, Funny

    This is old news down in the South.

    They don't bother splicing. Them good ol' boys been big on Kernel Sanders for years now.

  17. It's Not For 100% Uptime by Bob9113 · · Score: 2, Insightful

    Lots of people are saying, "100% uptime of a particular machine is neither necessary nor desirable, full failover is better. Full failover is the only way to handle catastrophic hardware failures." Or something to that extent.

    But this isn't about 100% uptime. It's about not having to reboot for a kernel upgrade. You should still have hot failover if you want HA, this just removes one more thing that requires a reboot.

    It's like people saying, "I don't mind rebooting after installing Office, I don't expect 100% uptime from my workstation." Of course you don't need to be able to do software installs without rebooting. But isn't it nice to have that option available?

    Same with this. When (and if) it gets stabilized and standardized, you'll use it. Not for 100% uptime, just because it's nice to not be required to reboot to enable a particular software install.

    --
    Stop-Prism.org: Opt Out of Surveillance
  18. You are Wrong by mpapet · · Score: 3, Insightful

    And THIS is how you perform upgrades. You split the cluster, upgrade one half, verify that the upgrade worked, then roll the cluster over to that node, and upgrade the second portion of the cluster. If you have more machines in the cluster, you do 'round-robin' upgrades

    Hmmm. I happen to live by your words in an environment where this is theoretically possible, but practically impossible. Why? Because when the cluster rolls to a passive node, the application times out on the existing connections. The time outs have business ($$$$) implications. I wish it were okay to have infinite retries, but it's viewed as a violation of the service agreement. Telephony is like this too.

    An academic ideal for sure, but please speak more humbly because it is no silver bullet.

    --
    http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
  19. Re:In Soviet Russia, by oodaloop · · Score: 5, Funny

    "But does it run linux?" That's a joke? I thought that was just one dedicated user who kept asking on every article.
    --
    Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
  20. Re:Gates Was Right by DragonTHC · · Score: 2, Funny

    I love anything that makes a billionaire whine.

    --
    They're using their grammar skills there.
  21. What!? Unpossible! by Linux_ho · · Score: 3, Funny

    This is GPL'd software. Bill Gates told me nobody could improve it. These Linux developers are truly renegades!

    --
    include $sig;
    1;
  22. Re:Over-engineered solution to a non-existent prob by mr_mischief · · Score: 2, Interesting

    You'd roll back much the same way, or even perhaps by rebooting into the previous kernel image from disk.

    Every production environment I've ever administered had a smaller version set aside for testing. We'd configure the machines identically and just make the cluster smaller. Then we'd test on the test machines any action that was to be made part of the admin process of the production machines. If it passes on the test machine and fails in production, then you didn't make the machines sufficiently similar.

    Round robin upgrades take ( ( (time_to_idle + time_to_upgrade + time_to_reboot) * machines ) / 2) on average to get a machine upgraded. If you have a "Critical" upgrade, that might be longer than you want.

    Not everyone has the exact same QA requirements you do, either. Some of us are happy with proving that it works, then proving that it worked on the production machine, then resuming our normally scheduled maintenance.

  23. Linux just gets better. by bannerman · · Score: 2, Insightful

    I would think that on top of the benefits of patching running high-uptime servers this would in the long run also result in yet another benefit to running Linux on your desktop instead of Windows. I don't see any reason RedHat, Ubuntu and everyone else wouldn't implement this type of kernel upgrade for convenience' sake.

    --
    I keep forgetting my place. Jesus is for losers. Why do I still play to the crowd?
  24. Re:And Microsoft claims to have invented it by johannesg · · Score: 2, Informative

    AmigaOS had its kernel in ROM, and could be patched on the fly. That was back in 1985, so even if it was patented, it isn't now.

    The patching function was not an accident either; there was an OS-function for this purpose. Originally it was intended to allow bug-fixed to be installed without having to change the ROM, but it was quickly coopted into a mechanism for enhancing the OS in various other ways as well.

  25. This was the smallest part of the interview... by tytso · · Score: 3, Informative

    Funny thing... this was the smallest part of my oh, hour and twenty minute interview with the reporter. The reason for the call was to hear about what was up with the 2.6.25 release; she probably spent more time talking with me about KVM and Xen; and I mentioned ksplice just as an aside, as an example of lots of really interesting and exciting work that doesn't necessarily happen as part of a mainline kernel release. I spent maybe 2-3 minutes tops talking to her about ksplice --- and that's what she ends up writing about and getting slashdotted!

  26. Microsoft has NOT patented this! by Anonymous+EPA · · Score: 2, Informative
    Tomasz Chmielewski is wrong. Microsoft applied for a patent and their application was rejected by the examiner, as was their appeal in the USPTO. Check out the file history of application US 2004-0107416.

    Their only resort is to appeal to court.

    There are no applications in other countries.

    A