Slashdot Mirror

← Back to Stories (view on slashdot.org)

How MIT and Caltech's Coding Breakthrough Could Accelerate Mobile Network Speeds

Posted by samzenpus on from the greased-lightning dept.

colinneagle (2544914) writes "What if you could transmit data without link layer flow control bogging down throughput with retransmission requests, and also optimize the size of the transmission for network efficiency and application latency constraints? In a Network World post, blogger Steve Patterson breaks down a recent breakthrough in stateless transmission using Random Linear Network Coding, or RLNC, which led to a joint venture between researchers at MIT, Caltech, and the University of Aalborg in Denmark called Code On Technologies.

The RLNC-encoded transmission improved video quality because packet loss in the RLNC case did not require the retransmission of lost packets. The RLNC-encoded video was downloaded five times faster than the native video stream time, and the RLNC-encoded video streamed fast enough to be rendered without interruption.

In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet."

96 of 129 comments (clear)

  1. shifts the reliance by Anonymous Coward · · Score: 1

    and what if the coefficients get lost or corrupted?

    1. Re: shifts the reliance by Anonymous Coward · · Score: 1

      It's either the entire packet comes or it's lost.

    2. Re:shifts the reliance by globaljustin · · Score: 1

      I wondered why a system would use it...if I understand it correctly...

      it's a new encoding that requires software on either end for TCP, that's in TFA...

      --
      Thank you Dave Raggett
    3. Re:shifts the reliance by Immerman · · Score: 1

      Acknowledgment is conceptually a separate thing from retransmission requests. If the normal transmission stream is such that dropped packets can be detected, then the only return communication necessary is the occasional "please repeat packet X" Of course you probably still want *some* acknowledgment to avoid the scenario where someone watches only the first 5 seconds of video before shutting off their computer but the server keeps streaming video for the next hour to a now non-existent client, but it need not be with every packet. An occasional "I'm still here, keep it coming" would be fine.

      --
      --- Most topics have many sides worth arguing, allow me to take one opposite you.
  2. Can we update the title please? by hamster_nz · · Score: 3, Interesting

    "A better coding for data error correction and redundancy than Reed-Solomon" - this is News for Nerds after all.

    And why the "oooh - flappy birds on my phone might be faster" slant? I want a faster SAN!.

    1. Re:Can we update the title please? by ajyand · · Score: 4, Insightful

      Reed-solmon is theoretically best. I hope this new encoding is practically better than Reed-Solmon.

    2. Re:Can we update the title please? by AK+Marc · · Score: 1

      How is R-S theoretically best, when it's been beaten by almost everything else in wireless FEC?

      I saw this and was looking for the tech details. It looks like they are putting in FEC at the application layer. It's been done before. They are just hiding it better to not have to explain why 1/8th of the data is padding (or whatever ratio they are using).

      --
      Learn to love Alaska
    3. Re:Can we update the title please? by Anonymous Coward · · Score: 1

      Probably from a misleading name. For example, Golay codes are perfect codes. Despite being perfect, there are better choices.

    4. Re:Can we update the title please? by AK+Marc · · Score: 1

      And it may be different applications. R-S may be "perfect" for the type of coding it does, but it isn't purely FEC, so it's both perfect, and beaten by lots of codes for wireless FEC. Or it's perfect for "random" drops/noise, but noise/drops aren't random. So many others can beat it because "perfect" is a mathematical truth, but doesn't work for the real world.

      --
      Learn to love Alaska
    5. Re:Can we update the title please? by Zironic · · Score: 3, Informative

      Because R-S assumes uniformly random error distribution which is usually not the case when it comes to wireless interference.

    6. Re:Can we update the title please? by TapeCutter · · Score: 1

      You win the internet - quote TFA - "tests show it could deliver dramatic potential gains in many use cases" (my emph.)

      --
      And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
    7. Re:Can we update the title please? by delt0r · · Score: 1

      That why we use turbo codes, or at least interleaves. The problem with RS codes is that to optimal decode them requires quite a bit of hardware/cpu cycles.

      --
      If information wants to be free, why does my internet connection cost so much?
    8. Re:Can we update the title please? by ShakaUVM · · Score: 1

      >"A better coding for data error correction and redundancy than Reed-Solomon" - this is News for Nerds after all.

      Well, yeah. There's lots of things better than Reed-Solomon for doing video encoding in a lossy environment. I worked on such a project at UCSD in 1998 or so with John Rogers and Paul Chu that could eat pretty impressive amounts of noise and still have a usable video signal without retransmissions. It's start getting pretty muddy looking when you really cranked up the losses, but unlike JPEG or something like that, a frame was still renderable even if you took out any particular chunk of the data.

  3. Nothing new under the sun, just new uses by preaction · · Score: 3, Insightful

    Sounds like Parchive from usenet, which is a really good idea in a lossy environment now that I think about it.

    1. Re:Nothing new under the sun, just new uses by NoNonAlphaCharsHere · · Score: 4, Informative

      Yup, sounds a lot like par2, except this system works inline, in a streaming context to repair mangled blocks/packets, where par2 uses out-of-band data (the par2 files) to do the repairs after all the data is transmitted.

    2. Re:Nothing new under the sun, just new uses by TubeSteak · · Score: 5, Informative

      And like par2, it's going to require a healthy amount of processing from your CPU

      The trends to higher-performance multicore processors and parallel operations everywhere in the network and on mobile devices lends itself to an encoding scheme utilizing linear algebra and matrix equations that might not have been possible in the past.

      Notice they talk about multicore processors and not some hardware decoding embedded in the networking chip.
      From their published paper

      Abstract-- Random Linear Network Coding (RLNC) provides
      a theoretically efficient method for coding. The drawbacks associ-
      ated with it are the complexity of the decoding and the overhead
      resulting from the encoding vector. Increasing the field size and
      generation size presents a fundamental trade-off between packet-
      based throughput and operational overhead      . On the one hand,
      decreasing the probability of transmitting redundant packets is
      beneficial for throughput and, consequently, reduces transmission
      energy. On the other hand, the decoding complexity and amount
      of header overhead increase with field size and generation
      length, leading to higher energy consumption. Therefore, the
      optimal trade-off is system and topology dependent, as it depends
      on the cost in energy of performing coding operations versus
      transmitting data. We show that moderate field sizes are the
      correct choice when trade-offs are considered. The results show
      that sparse binary codes perform the best, unless the generation
      size is very low.

      Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.

      --
      [Fuck Beta]
      o0t!
    3. Re:Nothing new under the sun, just new uses by AK+Marc · · Score: 1

      If you named all the files the same, then the repairs are done in-band. All that's "new" here is calling all the packets *.PAR, rather than some RAR and some PAR. You need one extra block per block lost (though you can have multiple blocks per packet and other tricks to hide the actual redundancy).

      --
      Learn to love Alaska
    4. Re: Nothing new under the sun, just new uses by TapeCutter · · Score: 1

      It probably will be if it turns out to be useful, but I doubt processing power will be a significant factor in the idea's commercial success or failure. The desktop video card is the supercomputer of a decade ago, albeit without the memory, disk space and air-conditioning bill. My $150 card maxes out at a teraflop. There are demos on the net of 3D modelling (with sound) running on a cell phone using Nvidia's APX2500 chip, android + java. Developers can develop stuff on their desktop video card and drop it straight into an android phone built around the chip.

      --
      And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
    5. Re: Nothing new under the sun, just new uses by garyebickford · · Score: 1

      Indeed. Back in the day my company used custom hardware - 8x8 binary convolver - to do pattern recognition on images. It ran at the unbelievable rate of 12 billion pixel operations per second. The chips were originally from the cruise missile program, so we had to get a DoD license for every one we shipped. By the mid-1990s a reasonably fast Pentium could do the same work in software. And I'm pretty sure that some ray tracing code that ran on the Cray X-MP would now run faster on my phone.

      --
      It's easier to be a result of the past, but more fun to be a cause of the future! http://www.spacefinancegroup.com/
    6. Re:Nothing new under the sun, just new uses by mcrbids · · Score: 1

      Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.

      I don't see this as a problem at all. This is the type of thing that screams for a dedicated processor, not a GP chip, and it's typical to see a 99% reduction in power consumption when you go this route.

      --
      I have no problem with your religion until you decide it's reason to deprive others of the truth.
  4. what the FEC... by bugs2squash · · Score: 3, Funny

    they've invented forward error correction, this could enable data communications and audio CDs.

    --
    Nullius in verba
    1. Re:what the FEC... by nyet · · Score: 2

      FEC generally does not seek to recover lost data, only the proper state of flipped bits.

    2. Re:what the FEC... by Guspaz · · Score: 1

      There's no difference between the two (what's the difference between a 1 flipped to a 0, or a 1 that was lost and so interpreted as a 0?), and FEC is frequently (typically?) used to recover lost data. Interleaving is often used such that losing an entire transmitted packet results only in losing small parts of multiple actual packets, which FEC can then compensate for.

    3. Re:what the FEC... by nyet · · Score: 1

      FEC algorithms do not treat (an unknown number) of lost bits as a stream of 0 bits, since it can't know how many bits are lost.

    4. Re:what the FEC... by timeOday · · Score: 1

      Do all of these ultimately result in a corrected transmission? How about a transmission that degrades gracefully instead, so you don't have to re-transmit OR send redundant information that isn't even used unless packets are lost. Each packet would contain only a little low-frequency information , since most of the bits are high-frequency information that will not be missed as much. Maybe your smartphone could receive 4k video broadcasts and just discard 80% of the packets before even decoding them. I'm sure there are encodings that use this technique, so what is it called?

    5. Re:what the FEC... by complete+loony · · Score: 1

      What if you could transmit data without link layer flow control bogging down throughput with retransmission requests

      TFS makes it look like network coding can magically send data without any form of ACK & Retransmit. Network coding still requires feedback from a flow control protocol. You need to tweak the percentage of extra packets to send based on the number of useful / useless packets arriving, and you still need to control the overall rate of transmitted packets to avoid congestion. The goal is to make sure that every packet that arrives is useful for decoding the stream, regardless of which packets are lost. So yes, it's a kind of automatically adapting error correction protocol for a stream service.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
    6. Re:what the FEC... by m00sh · · Score: 1

      they've invented forward error correction, this could enable data communications and audio CDs.

      I think they combined FEC with TCP.

      TCP sends packet and then waits for ACK. However they eliminated the ACK part and just added extra parity in the packets. So, even when packet is lost, it can be reconstructed.

      Of course there is a probability that multiple packets are lost so that it cannot be reconstructed. Of course, there is more redundant data in the stream instead of the small ACK packets. The packet loss characteristics of the channel probably makes this approach more efficient.

    7. Re:what the FEC... by YoopDaDum · · Score: 1

      FEC is commonly used in streaming over IP applications to support lost packets, see for example the "above IP" FEC used in LTE multicast (eMBMS) or DVB-H, Raptor. In those applications the L2 transport has its own error detection scheme, so IP packets will typically be either fully lost or ok (in other words, the probability of having a corrupt packet is very low).

    8. Re:what the FEC... by TapeCutter · · Score: 2

      Wow, were you aiming for humour? - Because it would be sad if you were serious

      What people are saying is they have implemented FEC at the packet level. I don't know much about wireless but I do remember the maths lectures I attended 25yrs ago on error correction techniques. At the time I could perform the FEC algorithm by hand, I consider myself "mathematically inclined" and have been awarded bits of paper attesting to that inclination but to this day I still have just enough understanding of the mathematical concepts of error correction to marvel at the geniuses who worked it all out in the first place.

      Packet FEC has been tried before and found to be wanting, but I would not be as quick as some to dismiss a strong claim by MIT just because others have failed in the past.

      --
      And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
    9. Re:what the FEC... by wagnerrp · · Score: 1

      The ACK is crucial. Even if you pipeline your packets so you do not have to wait for each ACK to return, they are still vital to measuring network capacity. Too many people trying to send too much data will eventually overwhelm the transmit buffer at some congestion point, and loss will shoot past the ability of your error correction to compensate for.

    10. Re:what the FEC... by Impy+the+Impiuos+Imp · · Score: 1

      So this is basically an efficient redundancy transmissiin scheme that out-performs (in spite of using extra data) retransmission.

      At a 3% artificially-induced error rate. What about real-world conditions? Wouldn't this clog a network more, not less?

      --
      (-1: Post disagrees with my already-settled worldview) is not a valid mod option.
    11. Re:what the FEC... by Guspaz · · Score: 1

      They don't need to. The underlying layer knows how many bits are missing based on the timing.

    12. Re:what the FEC... by hhawk · · Score: 1

      I would say FEC to is correct the current bit state, while this method RLNC is to correct a past or future packet.. I am totally suggesting is with fast enough processing power the protocol would assume every packet is missing and have recovered packets "ready to go" if one is missing...

      --
      http://www.hawknest.com/
    13. Re:what the FEC... by tolkienfan · · Score: 1

      Is there a paper I could read. This looks ineresting.

    14. Re:what the FEC... by complete+loony · · Score: 1

      I'm thinking of Network Coding Meets TCP. Though that doesn't give a great background. I've experimented with my own implementation, but had to shelve it due to lack of time. I'll try to quickly summarise the core idea;

      You have packets [p1, .. pn] in your stream's transmission window.

      Randomly pick coefficients [x1, ..., xn] and calculate a new packet = x1*p1 + x2*p2 + , .... (in a galois number field, so '+' is an XOR and '*' is a pre-computed lookup table). Sending the coefficients in the packet header.

      The receiver collects the packets, and attempts to combine pairs of packets to reduce the complexity of the coefficients. Basically like solving simultaneous equations. That sounds complicated, but the algorithm isn't too hard;

      - Keep the current set of packets, sorted by their coefficients. When you receive an incoming packet, you attempt to subtract each of your existing packets which have a smaller coefficient set (again, galois field math). If you're left with nothing, this packet didn't give you any new information, so throw it away.

      - Attempt to subtract this new packet from each of the existing packets in your set. Insert your new packet into the list.

      When you have a packet with a most significant coefficient of 1. You know you will be able to decode that packet eventually. And you know that you can eliminate that coefficient from any other incoming packet. So you can send an acknowledgement to the sender and they can advance their transmit window. Once you have eliminated all other coefficients you can deliver the packet to the application. Keep each packet in memory until the sender has advanced their window and stopped sending it.

      Each incoming packet may eliminate a coefficient from the packets in your list, while at the same time introducing a new one. If you don't send extra redundant packets you may never be able to decode anything. Consider the worst case where every packet is the XOR of two neighbouring packets in the stream, you can't decode anything until you receive a single packet by itself. Sending more redundant packets will reduce the latency to decode the stream while adding to the number of useless packets received.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
  5. in simplified terms, it's forward error correction by YesIAmAScript · · Score: 1

    And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?

    This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?

    --
    http://lkml.org/lkml/2005/8/20/95
  6. sounds like an ad for the future fast lane by CaptainStumpy · · Score: 5, Funny

    Xfinity video in your face 4650% faster! Xfinity introduces the RLNC fast lane data transmission! Its like an over caffeinated jaguar solving linear matrices while orbiting the earth in the space shuttle and doing coke. RAAAWRR! Don't like the jaguar? Tough floating jaguar shit, you don't have a choice! We own teh tubes! ©omcastic!

    --
    It will be better to purchase from an owner who is a good farmer and a good builder.
    1. Re:sounds like an ad for the future fast lane by blue+trane · · Score: 1

      Engineers want to try it because it's a cool idea. Biz, focused on profit, doesn't. This is why AT&T rejected Kleinrock and others who came to them with ideas of the internet, The market delayed implementation out of short-sighted, blinkered concentration on profit as opposed to the General Welfare.

    2. Re:sounds like an ad for the future fast lane by rasmusbr · · Score: 1

      With net neutrality, the incentive for trying such ideas will be nonexistent.

      As long as there is a significant bottleneck somewhere in the last mile or so (typically 4G/LTE or crowded WiFi) the incentive for efficient use of the available bandwidth will remain huge, regardless of politics.

      The thing that might hamper development is if one company manages to get a monopoly on the creative content so that everyone have to use their service regardless of how shitty it performs.

    3. Re:sounds like an ad for the future fast lane by AK+Marc · · Score: 1

      No, it's the other away around. You get "pay" priority performance in the "free" queue under FCC net brutality.

      --
      Learn to love Alaska
  7. GAAA! by Anonymous Coward · · Score: 1

    In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.

    If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.

    1. Re:GAAA! by AK+Marc · · Score: 1

      "Application level FEC" Is that easier? They try to make it sound cooler, or harder, but it's been done before.

      --
      Learn to love Alaska
    2. Re:GAAA! by skids · · Score: 1

      If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.

      If it is like most such articles, they use 35 pages of polynomial math over a GF(2) field as a way to mathematically formalize what could be better explained with a one-page circuit diagram with a few XOR and shift registers and one paragraph of commentary, so yeah, wait for the boiled down version.

      --
      Someone had to do it.
    3. Re:GAAA! by tibit · · Score: 1

      There's no single paper. There's a lot of papers that refine the technique and apply it to different problems. To even begin to understand it, you must have a good grasp of network coding as it's actually applied in real life. The butterfly network is but a starting point.

      --
      A successful API design takes a mixture of software design and pedagogy.
  8. Word by Dan+East · · Score: 5, Funny

    "the immediately earlier sequenced packet". There a word for that. It's called "previous". As in "the previous packet".

    --
    Better known as 318230.
    1. Re:Word by Anonymous Coward · · Score: 1

      Head east, Dan. Head! East!

      Nothing says the previous packet has the previous sequence number. Packets may arrive in any order, or not arrive at all (the point here). The stack makes it all work at the ends. This is all layman should need to understand, and as you have shown, all they can be expected to understand. It was so wored for a reason, and it it's lost on the layman, that's all right.

    2. Re:Word by Your.Master · · Score: 1

      Nothing says the previous packet has the previous sequence number

      The word "previous" says the previous packet has the previous sequence number. I that reasonable people skilled in the art would generally assume previous meant sequentially previous, not time-of-receipt previous.

      However I would say is that nothing says that the "immediately earlier" packet has the previous sequence number.

      We could get pedantic and talk about sequentially previous and chronologically previous, further clarifying that chronologically previous is from the point of view of the receiver (since in the sender's point of view, the sequence and chronology are likely identical since we don't usually have to re-transmit lost packets).

      As a result, I think "immediately earlier sequenced packet" is ambiguous. I could read "sequenced packet" as a compound noun, and thus it means the chronologically previous packet. Or I could read "immediately earlier sequenced" to mean previous, as in sequentially previous, as the GP did. From context, I think the latter is more likely, because what relevance could chronology possibly have? But that makes "immediately previous" an exceptionally poor choice of words, and thus I disagree that:

      it was so worded [sic] for a reason

    3. Re:Word by Wraithlyn · · Score: 2

      ...so does "earlier"

      --
      "Mind, as manifested by the capacity to make choices, is to some extent present in every electron." -Freeman Dyson
    4. Re:Word by Paradise+Pete · · Score: 1

      I that reasonable people skilled in the art would generally assume previous meant sequentially previous, not time-of-receipt previous.

      Sure, just like readers are able to reconstruct the missing second word in your post. But the word "previous" has some ambiguity. The original wording is more precise and specific.

    5. Re:Word by Paradise+Pete · · Score: 1

      ..so does "earlier"

      Thus the additional qualifiers "immediately" and "sequenced".

    6. Re:Word by AK+Marc · · Score: 1

      Maybe they are trying to cover the case where they come out of order, then there's a drop?

      But I agree in principle, the whole thing was poorly worded. Like it was written by a writer that doesn't understand it. That was dictated to by an engineer who was ordered to not be specific about it.

      --
      Learn to love Alaska
    7. Re:Word by c · · Score: 1

      There a word for that. It's called "previous". As in "the previous packet".

      When you're talking about packet protocols, where things get lost, duplicated, or reordered, "previous" is an incredibly imprecise word.

      --
      Log in or piss off.
  9. Network Coding by Boycott+BMG · · Score: 1

    I remember working on a project where network coding was proposed for micro satellite cluster communications. If I remember correctly, network coding requires that all the nodes in the network have complete knowledge of the state of the network at any given transmission window. This requires transmission of the network state which used something like 7% overhead. The routing of a message from one end of the cluster to the other was difficult. I believe it might have been an np-complete problem. Have they solved the routing issues?

  10. Re:in simplified terms, it's forward error correct by globaljustin · · Score: 1

    yeah TFA says it's link-layer flow control:

    What if you could transmit data without link layer flow control bogging down throughput with retransmission requests, and also optimize the size of the transmission for network efficiency and application latency constraints? Researchers from MIT, Caltech and the University of Aalborg claimed to have accomplished this...

    it can "piggyback" on TC-IP but you have to use their "software" on either end

    RLNC encoding can ride on top of the TCP-IP protocol, so implementation does not require the replacement of communications equipment. But it does require software incorporating RLNC-licensed technology to execute on both ends.

    i'm not sure where they want this to be used, "mobile" WiFi and LTE are mentioned but I dont see this as anything other than a demonstration of a capability, not an application

    i come from a CCNA background but I've never worked with or seen something like this...seems like they would use it in data centers?

    --
    Thank you Dave Raggett
  11. Good Improvement by Davidlogann2 · · Score: 1

    If the RLNC is five times faster than the native lines, It mean that the entire transmission network may go on a roll?

  12. More research needed by ketomax · · Score: 1

    How MIT and Caltech's Coding Breakthrough Could Accelerate Mobile Network Speeds

    I wish they also made some breakthrough to increase the data caps we are stuck with.

    1. Re:More research needed by citizenr · · Score: 1

      caps are Layer 8 problem

      --
      Who logs in to gdm? Not I, said the duck.
  13. Re:in simplified terms, it's forward error correct by tlambert · · Score: 1

    And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?

    This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?

    I believe the issue is that you can't sell it to the cable companies and the DSL providers that implement PPPOE in order to track your surfing, to make sure you are buying your television programming from them, rather than file sharing, and they can intentionally make things like Tor not work. Not that PMTUD works on those things unless the modem proxies the ICMP messages, which are usually blocked by the cable companies, unless you explicitly ifconfig down to 1492 yourself, or enable active probing for black hole (rfc4821).

  14. random? by superwiz · · Score: 1

    The random element seems to be there to avoid having to come up with an optimal distribution of information. Otherwise, it seems like a frame-level (or even finer) RAID.

    --
    Any guest worker system is indistinguishable from indentured servitude.
    1. Re:random? by AK+Marc · · Score: 1

      Go look at Silver Peak. They do a WAN optimization that uses variable FEC to increase FEC as packet loss increases, to minimize waste as conditions change.

      --
      Learn to love Alaska
  15. Packet loss models? by antifoidulus · · Score: 2

    TFA doesn't seem to state what their assumptions were on how packets get lost and how many packet losses the algorithm can deal with, and what their distribution is. There are a lot of ways you can drop k packets out of n packets sent.

    If you assume that every packet has a k/n chance of being lost, then being able to reconstruct a single missing packet could be incredibly useful. However cell phone packet losses tend to be incredibly bursty, i.e. they will have a very throughput for a while, but then all of a sudden(maybe you changed towers or went under a bridge etc) lose a whole ton of packets. Can this algorithm deal with bursty losses? I wish TFA was a bit more clear on that

    --
    Monstar L
    1. Re:Packet loss models? by complete+loony · · Score: 1

      (I haven't read this paper yet, but I've read other Network Coding data and experimented with the idea myself)

      With TCP, when a packet is lost you have to retransmit it. You could simply duplicate all packets, like using RAID1 for HDD redundancy. But this obviously wastes bandwidth.

      Network coding combines packets in a more intelligent way. More like RAID5 / RAID6 with continuous adaptation based on network conditions. Any packet that arrives may allow you to deduce information that you haven't seen before. Basically each packet is the result of an equation like; f(p1, p2, p3) = a*p1 + b*p2 + c*p3. When each packet arrives, you attempt to solve the set of simultaneous equations you have received. When you have reduced each expression to just a single packet, you send it up the protocol stack.

      You still need a TCP like acknowledgement scheme so that you can; rate limit the flow of packets based on measured congestion, tweak the percentage of redundant packets being sent due to measured packet loss, and advance the stream to include new data.

      If you get the network coding parameters wrong, the connection still might stall, or you might be sending too much redundant data. If everything is going well, a link with 10% packet loss just means that your stream is transferred 10% slower.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
  16. A better explanation by grahamsz · · Score: 1

    The story linked to seems to have an awful explanation of what's going on. This makes a lot more sense:

    http://www.codeontechnologies....

    Reminds me a little of a random project I started back in college where I'd transmit a file in a bunch of packets where each contained the original file modulo a specified prime number. That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them they could use the extended euclidean algorithm to reconstruct the original file.

    I was hoping we'd someday be able to multicast udp over the net to multiple random locations and this would be a fast way to send files.

    1. Re:A better explanation by Ingenium13 · · Score: 1

      It might be possible now with IPv6.

    2. Re:A better explanation by gnasher719 · · Score: 1

      That link shows their website, where they tell us that if the sender is about to send packet 83, and figures out it didn't get an acknowledgement for packet 22, it then has to resend packets 22 to 82. Which seems an entirely stupid thing to do, and an obvious improvement would be to resend only the packets that are actually lost.

      Guess what: According to the Wikipedia article about TCP, that's what the "selective acknowledgment" (SACK) option defined in RFC 2018 does. So _poof_ goes the benefit of this scheme, which was based on an incorrect representation of TCP/IP.

      On the other hand, my TV receives a purely digital signal with no way to ask for re-transmission of lost data and works just fine with it. There is a variant of h.264 specicially for streaming connections without retransmission, and that together with UDP instead of TCP/IP would solve the problem just fine. If it was a problem.

  17. Re:in simplified terms, it's forward error correct by AK+Marc · · Score: 2

    I think they are lying.

    Many of the details fail under closer examination. It doesn't "use" TCP-IP. It could use a propritary IP stack that is TCP/IP compatible. So it'll use IP addresses and port numbers like a TCP packet would so that switches and routers in the middle wouldn't know or care what it's doing. If they put it on an IPX/SPX core it'd fail to route across the Internet. But it isn't TCP. It doesn't use a TCP compliant stack. It just looks like one to the outside world. It will not re-transmit a lost packet via TCP mechanisms. But it'll work on the same hardware on both ends and software in the middle. You just have to replace the network stack on both ends, which isn't hard. Though they indicate that the only thing it needs is "software" on both ends. Maybe they'll be doing it over actual TCP/IP. That, and the way they keep saying TCP-IP, that includes UDP. They don't say TCP, or UDP. And I don't trust them when they keep saying TCP-IP, it should be a slash, not a dash. So is it running over TCP/IP (which could mean UDP)? Or does it run over TCP (which excludes UDP)?

    --
    Learn to love Alaska
  18. Right. by Animats · · Score: 4, Insightful

    This is yet another form of forward error correction. The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.

    This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.

    1. Re:Right. by MobyDisk · · Score: 2

      The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.

      Yeah, that would be impossible. But I don't think they meant to claim that there is no extra data. When they say "The combined packet length is no longer than either of the two packets from which it is composed." I took that to mean that each packet is a fixed length. Not that they didn't add extra data to get to that packet length.

      This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.

      Yeah. It might be good for for wireless networks, where even a "5 bar" wireless network connection has a fairly consistent 1% packet loss. The QUIC protocol is another attempt to handle this better by using packet pacing, but QUIC isn't worth it in general even if it addresses this one specific problem.

  19. Re:halve the packets? by citizenr · · Score: 1

    yes, by sending packet twice the size

    --
    Who logs in to gdm? Not I, said the duck.
  20. Re:Better than FEC by AK+Marc · · Score: 1

    There are 7/8 FEC (meaning 1/8 wasted on FEC), and if you code the FEC into the stream, then you'll hide the "loss". They look to be hiding the loss, so people don't complain about the "waste."

    --
    Learn to love Alaska
  21. Re:wtf? by AK+Marc · · Score: 1

    Why would you put that on top of a streaming protocol like TCP/IP that's got all the flow control and retry logic happening under the covers? Surely you'd run RLNC over UDP!

    UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.

    --
    Learn to love Alaska
  22. Excellent! by davidc · · Score: 1

    This means my mobile internet speed might soon be up to 10 bps instead of the 2 bps I seem to get at the moment!

    1. Re:Excellent! by Anonymous Coward · · Score: 1

      It means that your ISP can oversell their bandwidth even more.

  23. Re:wtf? by Anonymous Coward · · Score: 1

    UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.

    UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.

  24. I've seen this before by statemachine · · Score: 1

    Except it's called MPEG video. And it's used for TV.

    MPEG also has a mode to recover the errors, but it's expensive, and when you're streaming, who cares? If your link sucks, you don't blame the stream.

  25. Good press release / slashvertisement by Anonymous Coward · · Score: 1

    "Code On" has done well at generating some buzz.
    Unfortunately, the only details on their website are of the type "this is awesome", with no description of how the breakthrough works.

    http://www.codeontechnologies.com/technology/white-papers/

    I just wish they would post actual details on the encoding method. How does it compare to Fountain Codes such as RFC 5053?

  26. Really? by Viol8 · · Score: 1

    "That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them they could use the extended euclidean algorithm to reconstruct the original file."

    So if the receiver got 10K copies of the 1st packet and nothing else it could still reconstruct the file? Thats impressive. Which college was this, Hogworts?

    1. Re:Really? by skids · · Score: 1

      So if the receiver got 10K copies of the 1st packet and nothing else it could still reconstruct the file?

      Considering each one is only sent once, that would be some feirce level of broken compound load balancing.

      Note he said each packet contains the modulus of the entire original file with a different prime. The only thing that would cause duplicate packets would be running out of acceptably-sized primes.

      --
      Someone had to do it.
    2. Re:Really? by Viol8 · · Score: 1

      You need to re-read what he wrote:

      "That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them "

      How can you split something into 10K packets then get *any* 10K of them? He's either saying he sends more than 10K or the receiver has to receive all 10K packets that were sent to reconstruct the file which I doubt is what he meant.

    3. Re: Really? by grahamsz · · Score: 1

      Yeah that could have been more clear. I'm suggesting sending a constant stream of packets until the target asks you to stop. No need for windowing or flow control, but pretty computationally expensive

  27. That's the over-simplified version? by wonkey_monkey · · Score: 1

    In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.

    Uh... could you simplify it just a little more?

    How does a "later-sequenced packet [...] include earlier-sequenced packets"?

    --
    systemd is Roko's Basilisk.
    1. Re:That's the over-simplified version? by EmperorArthur · · Score: 1

      They say "randomly" generated coefficients, but I'll bet you can use a psudo random number generator and pass in the same seed value to both the sender and reciever. Bam, now both sides have the same set of semi random coefficients to use when doing the fancy linear algebra.

      --
      So lets pretend that we've just completed writing this code, as opposed to having just completed sabotaging it -Altera
    2. Re:That's the over-simplified version? by tibit · · Score: 1

      The linear algebra isn't even fancy. They need to do a lot of linear equation solving, that's about it.

      --
      A successful API design takes a mixture of software design and pedagogy.
  28. Time saver by PsyMan · · Score: 1

    Now I can free up 4 of the 5 minutes it used to take burning through my monthly bandwidth to do something constructive.

  29. Re:Neat by wagnerrp · · Score: 1

    Only in the most fundamental sense that they are both forms of error correction.

  30. parses like a teaspoon of sugar by epine · · Score: 2

    I've been parsing this kind of press release for a long, long time now. I can pretty much tell what we're dealing with by how hard it is to state the advantage of a new approach in narrow and precise language.

    That this blurb doesn't even disclose the error model class (error correction is undefined without doing so) suggests that the main advantage of this codec lies more in the targeting of a particular loss model than a general advance in mathematical power.

    Any error correction method looks good when fed data that exactly corresponds to the loss model over which it directly optimises.

    The innovators of this might have something valuable, but they are clearly trying to construe it as more than it is. This suggests that there are other, equally viable ways to skin this particular cat.

  31. Sounds like a Fountain Code to me by ka9dgx · · Score: 1

    This sounds exactly like a Fountain Code to me, which isn't exactly news.

  32. Re:in simplified terms, it's forward error correct by Mariner28 · · Score: 3, Informative

    Like Anonymous Coward, yours were my thoughts exactly. Why would one use TCP to stream video? It's one of the tenants of networking that losing packets is preferable to increasing jitter in the video or audio feed. And off the top of my head, I'd say that there hasn't been a widely used connection-based layer 2 protocol since X.25. Hell, that's why Frame Relay and later ATM were invented - to let the transport layer handle error detection (and retransmission if required). Even Ethernet uses just a CRC for forward error correction - if the receiver can't fix errors, the frame is dropped. It's up to the upper layers do actually do anything about it. And let's not get started about a 3% random error distribution in a wireless link - everyone knows that fading causes a whole stream of consecutive packets to be lost, not just an even statistical distribution of them. Stephen Max Patterson at Network World just proved he isn't qualified to write for Network World... And just a nit for you, AK Marc - if someone says UDP is "running over TCP/IP", tell them to put down the router and step away from the rack. They just aren't qualified.

    --
    "A little misunderstanding? Galileo and the Pope had a little misunderstanding."
  33. Re:in simplified terms, it's forward error correct by RatherBeAnonymous · · Score: 1

    I think the writer is confused. This sounds like a non-routed layer 2 error-correction protocol for error prone networks, like cell phone data and Wi-Fi. The only relation this has to the Internet is that it can carry TCP/IP traffic, just like Ethernet, Frame Relay, ATM, and any number of other layer 2 protocols can carry TCP/IP.

  34. Re:only send the last packet? by skids · · Score: 1

    Usually in such schemes you can recover a packet if you have the surrounding packets before and after it.

    --
    Someone had to do it.
  35. Re:wtf? by AK+Marc · · Score: 1

    UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.

    http://en.wikipedia.org/wiki/I... TCP/IP is shorthand for The Internet Protocol Suite. UDP is a protocol in The Internet Protocol Suite. Therefore, you are 100% wrong, in your wording, UDP is a subset of TCP. You just chose to leave off the "/IP" because you know you are wrong.

    --
    Learn to love Alaska
  36. Re:in simplified terms, it's forward error correct by AK+Marc · · Score: 1

    I think they are using UDP, but didn't want to get into specifics, so the writer screwed it up. UDP is a member of the TCP/IP suite of protocols.

    --
    Learn to love Alaska
  37. Re:in simplified terms, it's forward error correct by AK+Marc · · Score: 1

    And just a nit for you, AK Marc - if someone says UDP is "running over TCP/IP", tell them to put down the router and step away from the rack. They just aren't qualified.

    How about "UDP is a member of the TCP/IP suite of protocols"? Though most leave off the "suite of protocols" at the end, it's implied. I know lots of people smarter than you that would say "UDP is a subset of TCP/IP" or some other wording to that effect. Even Wikipedia agrees with me. TCP/IP is shorthand for The Internet Protocol Suite, which includes UDP. So UDP "runs over" TCP/IP

    --
    Learn to love Alaska
  38. Re:in simplified terms, it's forward error correct by RatherBeAnonymous · · Score: 1

    Yeah, they probably are using UDP in some video tests, but I think that is irrelevant. If this is a competing technology to Reed-Solomon encoding, then it is almost certainly a data-link layer protocol and is agnostic to network or session protocols.

    From Wikipedia:

    Reed–Solomon codes have since found important applications from deep-space communication to consumer electronics. They are prominently used in consumer electronics such as CDs, DVDs, Blu-ray Discs, in data transmission technologies such as DSL and WiMAX, in broadcast systems such as DVB and ATSC, and in computer applications such as RAID 6 systems.

    I think the writer's mistake was in seeing everything through Internet-colored glasses. The article specifically says "link layer", not "session layer". TCP is a session layer technology. My best guess from the article is that this is not an end-to-end error correction protocol to reduce IP packet retransmission. It is an error correction protocol to reduce OSI layer 2 packet (or more properly, "frame") retransmission.

  39. Re:in simplified terms, it's forward error correct by AK+Marc · · Score: 1

    It is an error correction protocol to reduce OSI layer 2 packet (or more properly, "frame") retransmission.

    But the "frame" doesn't exist end-to-end, but is re-created for each network segment. So their statements that it runs on the endpoints and doesn't require any changes in the middle indicates it is a higher layer protocol. Some of the layer 2 already include FEC. So why have double FEC? (it's bad). It's not double FEC at the same layer, but an application-layer FEC (based on their description). You can have them on different layers because they are correcting different things. L-2 FEC is correcting bit errors from noise or other physical issues. L-7 FEC is correcting lost packets.

    But the one thing we do know for sure is that the article is crap. It's detailed when it shouldn't be, but leaves out all useful details. That takes some skill. But skill in what, we aren't sure.

    --
    Learn to love Alaska
  40. Re:Could someone explain why... by MidSpeck · · Score: 1

    I have the same question. From the article, "This is because RLNC can recreate any packet lost on the receiving side from a later sequenced packet." If this is truly the case, just send me the very last packet and I can recreate all the packets that were supposed to come before by working backwards.