How MIT and Caltech's Coding Breakthrough Could Accelerate Mobile Network Speeds

colinneagle (2544914) writes "What if you could transmit data without link layer flow control bogging down throughput with retransmission requests, and also optimize the size of the transmission for network efficiency and application latency constraints? In a Network World post, blogger Steve Patterson breaks down a recent breakthrough in stateless transmission using Random Linear Network Coding, or RLNC, which led to a joint venture between researchers at MIT, Caltech, and the University of Aalborg in Denmark called Code On Technologies.

The RLNC-encoded transmission improved video quality because packet loss in the RLNC case did not require the retransmission of lost packets. The RLNC-encoded video was downloaded five times faster than the native video stream time, and the RLNC-encoded video streamed fast enough to be rendered without interruption.

In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet."

Comments

shifts the reliance

[Anonymous Coward]

and what if the coefficients get lost or corrupted?

Re: shifts the reliance

[Anonymous Coward]

It's either the entire packet comes or it's lost.

Re:shifts the reliance

[Anonymous Coward]

and what if the coefficients get lost or corrupted?

How? The packet before or after will have the same coefficients and presumably some portion of the previous and next packet. Or, did your head explode trying to read the last paragraph of the summary like mine almost did?

Re:shifts the reliance

[Anonymous Coward]

" The packet before or after will have the same coefficients"
All the summary (and the article for that matter) say is that it's encoded using randomly generated generated coefficients. There's nothing about these staying the same from one packet to the next. There's nothing about how or when these coefficients are transmitted either. Are these coefficients sent with the packet?
Reading through again (for about the 5th time) it looks like it uses the coefficients from the later packet (which makes more sense), until of course you lose two packets in a row, cannot reconstruct anything, your transmission ceases, and since there is no acknowledgement needed (and I'm guessing none sent), you'll start losing packets completely, rather than just having to wait for them to be resent.

Re:shifts the reliance

[globaljustin]

I wondered why a system would use it...if I understand it correctly...

it's a new encoding that requires software on either end for TCP, that's in TFA...

Re:shifts the reliance

[Immerman]

Acknowledgment is conceptually a separate thing from retransmission requests. If the normal transmission stream is such that dropped packets can be detected, then the only return communication necessary is the occasional "please repeat packet X" Of course you probably still want *some* acknowledgment to avoid the scenario where someone watches only the first 5 seconds of video before shutting off their computer but the server keeps streaming video for the next hour to a now non-existent client, but it need not be with every packet. An occasional "I'm still here, keep it coming" would be fine.

Can we update the title please?

[hamster_nz]

"A better coding for data error correction and redundancy than Reed-Solomon" - this is News for Nerds after all.

And why the "oooh - flappy birds on my phone might be faster" slant? I want a faster SAN!.

Re:Can we update the title please?

[ajyand]

Reed-solmon is theoretically best. I hope this new encoding is practically better than Reed-Solmon.

Re:Can we update the title please?

[AK Marc]

How is R-S theoretically best, when it's been beaten by almost everything else in wireless FEC?

I saw this and was looking for the tech details. It looks like they are putting in FEC at the application layer. It's been done before. They are just hiding it better to not have to explain why 1/8th of the data is padding (or whatever ratio they are using).

Re:Can we update the title please?

[Anonymous Coward]

Probably from a misleading name. For example, Golay codes are perfect codes. Despite being perfect, there are better choices.

Re:Can we update the title please?

[AK Marc]

And it may be different applications. R-S may be "perfect" for the type of coding it does, but it isn't purely FEC, so it's both perfect, and beaten by lots of codes for wireless FEC. Or it's perfect for "random" drops/noise, but noise/drops aren't random. So many others can beat it because "perfect" is a mathematical truth, but doesn't work for the real world.

Re:Can we update the title please?

[Zironic]

Because R-S assumes uniformly random error distribution which is usually not the case when it comes to wireless interference.

Re:Can we update the title please?

[Anonymous Coward]

Reed-solmon is theoretically best. I hope this new encoding is practically better than Reed-Solmon.

Reed-Solomon is good for a stream. If packet-loss is more likely than bit-errors or bursts then it is less ideal.

Re:Can we update the title please?

[TapeCutter]

You win the internet - quote TFA - "tests show it could deliver dramatic potential gains in many use cases" (my emph.)

Re:Can we update the title please?

[Anonymous Coward]

You win the internet - quote TFA - "tests show it could deliver dramatic potential gains in many use cases" (my emph.)

now that he's won the internet, maybe he can ask the nsa to get off of it.

Re:Can we update the title please?

[delt0r]

That why we use turbo codes, or at least interleaves. The problem with RS codes is that to optimal decode them requires quite a bit of hardware/cpu cycles.

Re:Can we update the title please?

[Anonymous Coward]

Well, no.

The best error-correction code (ECC) depends on your needs, and channel type and conditions.
But clearly, modern ECC like LDPC, Turbo and even Polar codes are very good as well.

e.g.
- DVB-S2: LDPC + RS
- LTE: Turbo
- WiMAX: Turbo or LDPC
- 10G Ethernet: LDPC
- Magnetic storage: mostly RS atm, companies are looking at LDPC and Polar
- Quantum Key Distribution: Polar

Re:Can we update the title please?

[ShakaUVM]

>"A better coding for data error correction and redundancy than Reed-Solomon" - this is News for Nerds after all.

Well, yeah. There's lots of things better than Reed-Solomon for doing video encoding in a lossy environment. I worked on such a project at UCSD in 1998 or so with John Rogers and Paul Chu that could eat pretty impressive amounts of noise and still have a usable video signal without retransmissions. It's start getting pretty muddy looking when you really cranked up the losses, but unlike JPEG or something like that, a frame was still renderable even if you took out any particular chunk of the data.

Re:Can we update the title please?

[Anonymous Coward]

Whatever happened to Trellis encoding?

I had a bad experience with an extremely noisy/unreliable packet radio system 25 years ago. FEC is great and I would not walk away from the benefits it imparts. However my experience was that when the transmission medium was really poor, nothing will save you. Even the operator interfaces tend to break down. If you're lucky a timeout will intervene and basically warn you, "this isn't going to work. Stop now before your blood pressure kills you."

How about instead

[Anonymous Coward]

Code on 45

Ay? Ay?

Nothing new under the sun, just new uses

[preaction]

Sounds like Parchive from usenet, which is a really good idea in a lossy environment now that I think about it.

Re:Nothing new under the sun, just new uses

[NoNonAlphaCharsHere]

Yup, sounds a lot like par2, except this system works inline, in a streaming context to repair mangled blocks/packets, where par2 uses out-of-band data (the par2 files) to do the repairs after all the data is transmitted.

Re:Nothing new under the sun, just new uses

[TubeSteak]

And like par2, it's going to require a healthy amount of processing from your CPU

The trends to higher-performance multicore processors and parallel operations everywhere in the network and on mobile devices lends itself to an encoding scheme utilizing linear algebra and matrix equations that might not have been possible in the past.

Notice they talk about multicore processors and not some hardware decoding embedded in the networking chip.
From their published paper

Abstract-- Random Linear Network Coding (RLNC) provides
a theoretically efficient method for coding. The drawbacks associ-
ated with it are the complexity of the decoding and the overhead
resulting from the encoding vector. Increasing the field size and
generation size presents a fundamental trade-off between packet-
based throughput and operational overhead. On the one hand,
decreasing the probability of transmitting redundant packets is
beneficial for throughput and, consequently, reduces transmission
energy. On the other hand, the decoding complexity and amount
of header overhead increase with field size and generation
length, leading to higher energy consumption. Therefore, the
optimal trade-off is system and topology dependent, as it depends
on the cost in energy of performing coding operations versus
transmitting data. We show that moderate field sizes are the
correct choice when trade-offs are considered. The results show
that sparse binary codes perform the best, unless the generation
size is very low.

Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.

Re:Nothing new under the sun, just new uses

[AK Marc]

If you named all the files the same, then the repairs are done in-band. All that's "new" here is calling all the packets *.PAR, rather than some RAR and some PAR. You need one extra block per block lost (though you can have multiple blocks per packet and other tricks to hide the actual redundancy).

Re: Nothing new under the sun, just new uses

[Anonymous Coward]

If this is successfull it will be implemented in hardware.

Re: Nothing new under the sun, just new uses

[TapeCutter]

It probably will be if it turns out to be useful, but I doubt processing power will be a significant factor in the idea's commercial success or failure. The desktop video card is the supercomputer of a decade ago, albeit without the memory, disk space and air-conditioning bill. My $150 card maxes out at a teraflop. There are demos on the net of 3D modelling (with sound) running on a cell phone using Nvidia's APX2500 chip, android + java. Developers can develop stuff on their desktop video card and drop it straight into an android phone built around the chip.

Re: Nothing new under the sun, just new uses

[garyebickford]

Indeed. Back in the day my company used custom hardware - 8x8 binary convolver - to do pattern recognition on images. It ran at the unbelievable rate of 12 billion pixel operations per second. The chips were originally from the cruise missile program, so we had to get a DoD license for every one we shipped. By the mid-1990s a reasonably fast Pentium could do the same work in software. And I'm pretty sure that some ray tracing code that ran on the Cray X-MP would now run faster on my phone.

Re:Nothing new under the sun, just new uses

[mcrbids]

Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.

I don't see this as a problem at all. This is the type of thing that screams for a dedicated processor, not a GP chip, and it's typical to see a 99% reduction in power consumption when you go this route.

what the FEC...

[bugs2squash]

they've invented forward error correction, this could enable data communications and audio CDs.

Re:what the FEC...

[nyet]

FEC generally does not seek to recover lost data, only the proper state of flipped bits.

Re:what the FEC...

[Guspaz]

There's no difference between the two (what's the difference between a 1 flipped to a 0, or a 1 that was lost and so interpreted as a 0?), and FEC is frequently (typically?) used to recover lost data. Interleaving is often used such that losing an entire transmitted packet results only in losing small parts of multiple actual packets, which FEC can then compensate for.

Re:what the FEC...

[nyet]

FEC algorithms do not treat (an unknown number) of lost bits as a stream of 0 bits, since it can't know how many bits are lost.

Re:what the FEC...

[timeOday]

Do all of these ultimately result in a corrected transmission? How about a transmission that degrades gracefully instead, so you don't have to re-transmit OR send redundant information that isn't even used unless packets are lost. Each packet would contain only a little low-frequency information , since most of the bits are high-frequency information that will not be missed as much. Maybe your smartphone could receive 4k video broadcasts and just discard 80% of the packets before even decoding them. I'm sure there are encodings that use this technique, so what is it called?

Re:what the FEC...

[complete loony]

What if you could transmit data without link layer flow control bogging down throughput with retransmission requests

TFS makes it look like network coding can magically send data without any form of ACK & Retransmit. Network coding still requires feedback from a flow control protocol. You need to tweak the percentage of extra packets to send based on the number of useful / useless packets arriving, and you still need to control the overall rate of transmitted packets to avoid congestion. The goal is to make sure that every packet that arrives is useful for decoding the stream, regardless of which packets are lost. So yes, it's a kind of automatically adapting error correction protocol for a stream service.

Re:what the FEC...

[m00sh]

they've invented forward error correction, this could enable data communications and audio CDs.

I think they combined FEC with TCP.

TCP sends packet and then waits for ACK. However they eliminated the ACK part and just added extra parity in the packets. So, even when packet is lost, it can be reconstructed.

Of course there is a probability that multiple packets are lost so that it cannot be reconstructed. Of course, there is more redundant data in the stream instead of the small ACK packets. The packet loss characteristics of the channel probably makes this approach more efficient.

Re:what the FEC...

[YoopDaDum]

FEC is commonly used in streaming over IP applications to support lost packets, see for example the "above IP" FEC used in LTE multicast (eMBMS) or DVB-H, Raptor. In those applications the L2 transport has its own error detection scheme, so IP packets will typically be either fully lost or ok (in other words, the probability of having a corrupt packet is very low).

Re:what the FEC...

[TapeCutter]

Wow, were you aiming for humour? - Because it would be sad if you were serious

What people are saying is they have implemented FEC at the packet level. I don't know much about wireless but I do remember the maths lectures I attended 25yrs ago on error correction techniques. At the time I could perform the FEC algorithm by hand, I consider myself "mathematically inclined" and have been awarded bits of paper attesting to that inclination but to this day I still have just enough understanding of the mathematical concepts of error correction to marvel at the geniuses who worked it all out in the first place.

Packet FEC has been tried before and found to be wanting, but I would not be as quick as some to dismiss a strong claim by MIT just because others have failed in the past.

Re:what the FEC...

[wagnerrp]

The ACK is crucial. Even if you pipeline your packets so you do not have to wait for each ACK to return, they are still vital to measuring network capacity. Too many people trying to send too much data will eventually overwhelm the transmit buffer at some congestion point, and loss will shoot past the ability of your error correction to compensate for.

Re:what the FEC...

[Impy the Impiuos Imp]

So this is basically an efficient redundancy transmissiin scheme that out-performs (in spite of using extra data) retransmission.

At a 3% artificially-induced error rate. What about real-world conditions? Wouldn't this clog a network more, not less?

Re:what the FEC...

[Guspaz]

They don't need to. The underlying layer knows how many bits are missing based on the timing.

Re:what the FEC...

[hhawk]

I would say FEC to is correct the current bit state, while this method RLNC is to correct a past or future packet.. I am totally suggesting is with fast enough processing power the protocol would assume every packet is missing and have recovered packets "ready to go" if one is missing...

Re:what the FEC...

[tolkienfan]

Is there a paper I could read. This looks ineresting.

Re:what the FEC...

[complete loony]

I'm thinking of Network Coding Meets TCP. Though that doesn't give a great background. I've experimented with my own implementation, but had to shelve it due to lack of time. I'll try to quickly summarise the core idea;

You have packets [p1, .. pn] in your stream's transmission window.

Randomly pick coefficients [x1, ..., xn] and calculate a new packet = x1*p1 + x2*p2 + , .... (in a galois number field, so '+' is an XOR and '*' is a pre-computed lookup table). Sending the coefficients in the packet header.

The receiver collects the packets, and attempts to combine pairs of packets to reduce the complexity of the coefficients. Basically like solving simultaneous equations. That sounds complicated, but the algorithm isn't too hard;

- Keep the current set of packets, sorted by their coefficients. When you receive an incoming packet, you attempt to subtract each of your existing packets which have a smaller coefficient set (again, galois field math). If you're left with nothing, this packet didn't give you any new information, so throw it away.

- Attempt to subtract this new packet from each of the existing packets in your set. Insert your new packet into the list.

When you have a packet with a most significant coefficient of 1. You know you will be able to decode that packet eventually. And you know that you can eliminate that coefficient from any other incoming packet. So you can send an acknowledgement to the sender and they can advance their transmit window. Once you have eliminated all other coefficients you can deliver the packet to the application. Keep each packet in memory until the sender has advanced their window and stopped sending it.

Each incoming packet may eliminate a coefficient from the packets in your list, while at the same time introducing a new one. If you don't send extra redundant packets you may never be able to decode anything. Consider the worst case where every packet is the XOR of two neighbouring packets in the stream, you can't decode anything until you receive a single packet by itself. Sending more redundant packets will reduce the latency to decode the stream while adding to the number of useless packets received.

in simplified terms, it's forward error correction

[YesIAmAScript]

And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?

This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?

sounds like an ad for the future fast lane

[CaptainStumpy]

Xfinity video in your face 4650% faster! Xfinity introduces the RLNC fast lane data transmission! Its like an over caffeinated jaguar solving linear matrices while orbiting the earth in the space shuttle and doing coke. RAAAWRR! Don't like the jaguar? Tough floating jaguar shit, you don't have a choice! We own teh tubes! ©omcastic!

Re:sounds like an ad for the future fast lane

[blue trane]

Engineers want to try it because it's a cool idea. Biz, focused on profit, doesn't. This is why AT&T rejected Kleinrock and others who came to them with ideas of the internet, The market delayed implementation out of short-sighted, blinkered concentration on profit as opposed to the General Welfare.

Re:sounds like an ad for the future fast lane

[rasmusbr]

With net neutrality, the incentive for trying such ideas will be nonexistent.

As long as there is a significant bottleneck somewhere in the last mile or so (typically 4G/LTE or crowded WiFi) the incentive for efficient use of the available bandwidth will remain huge, regardless of politics.

The thing that might hamper development is if one company manages to get a monopoly on the creative content so that everyone have to use their service regardless of how shitty it performs.

Re:sounds like an ad for the future fast lane

[AK Marc]

No, it's the other away around. You get "pay" priority performance in the "free" queue under FCC net brutality.

GAAA!

[Anonymous Coward]

In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.

If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.

Re:GAAA!

[AK Marc]

"Application level FEC" Is that easier? They try to make it sound cooler, or harder, but it's been done before.

Re:GAAA!

[skids]

If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.

If it is like most such articles, they use 35 pages of polynomial math over a GF(2) field as a way to mathematically formalize what could be better explained with a one-page circuit diagram with a few XOR and shift registers and one paragraph of commentary, so yeah, wait for the boiled down version.

Re:GAAA!

[tibit]

There's no single paper. There's a lot of papers that refine the technique and apply it to different problems. To even begin to understand it, you must have a good grasp of network coding as it's actually applied in real life. The butterfly network is but a starting point.

Re:GAAA!

[Anonymous Coward]

If only all papers were formulated this way. The problem with mathematicians is that they never express things in terms that an engineer or programmer can use.

Word

[Dan East]

"the immediately earlier sequenced packet". There a word for that. It's called "previous". As in "the previous packet".

Re:Word

[Anonymous Coward]

Head east, Dan. Head! East!

Nothing says the previous packet has the previous sequence number. Packets may arrive in any order, or not arrive at all (the point here). The stack makes it all work at the ends. This is all layman should need to understand, and as you have shown, all they can be expected to understand. It was so wored for a reason, and it it's lost on the layman, that's all right.

Re:Word

[Your.Master]

Nothing says the previous packet has the previous sequence number

The word "previous" says the previous packet has the previous sequence number. I that reasonable people skilled in the art would generally assume previous meant sequentially previous, not time-of-receipt previous.

However I would say is that nothing says that the "immediately earlier" packet has the previous sequence number.

We could get pedantic and talk about sequentially previous and chronologically previous, further clarifying that chronologically previous is from the point of view of the receiver (since in the sender's point of view, the sequence and chronology are likely identical since we don't usually have to re-transmit lost packets).

As a result, I think "immediately earlier sequenced packet" is ambiguous. I could read "sequenced packet" as a compound noun, and thus it means the chronologically previous packet. Or I could read "immediately earlier sequenced" to mean previous, as in sequentially previous, as the GP did. From context, I think the latter is more likely, because what relevance could chronology possibly have? But that makes "immediately previous" an exceptionally poor choice of words, and thus I disagree that:

it was so worded [sic] for a reason

Re:Word

[Anonymous Coward]

'Previous' implies temporal sequencing

Re:Word

[Wraithlyn]

...so does "earlier"

Re:Word

[Paradise Pete]

I that reasonable people skilled in the art would generally assume previous meant sequentially previous, not time-of-receipt previous.

Sure, just like readers are able to reconstruct the missing second word in your post. But the word "previous" has some ambiguity. The original wording is more precise and specific.

Re:Word

[Paradise Pete]

..so does "earlier"

Thus the additional qualifiers "immediately" and "sequenced".

Re:Word

[AK Marc]

Maybe they are trying to cover the case where they come out of order, then there's a drop?

But I agree in principle, the whole thing was poorly worded. Like it was written by a writer that doesn't understand it. That was dictated to by an engineer who was ordered to not be specific about it.

Re:Word

[c]

There a word for that. It's called "previous". As in "the previous packet".

When you're talking about packet protocols, where things get lost, duplicated, or reordered, "previous" is an incredibly imprecise word.

Neat

[Anonymous Coward]

Sounds like a principle similar to http://en.wikipedia.org/wiki/Parchive

I like.

Re:Neat

[Anonymous Coward]

Sounds like some people are too stupid, fat or lazy to type Parchive huh?

Re:Neat

[wagnerrp]

Only in the most fundamental sense that they are both forms of error correction.

Network Coding

[Boycott BMG]

I remember working on a project where network coding was proposed for micro satellite cluster communications. If I remember correctly, network coding requires that all the nodes in the network have complete knowledge of the state of the network at any given transmission window. This requires transmission of the network state which used something like 7% overhead. The routing of a message from one end of the cluster to the other was difficult. I believe it might have been an np-complete problem. Have they solved the routing issues?

Zmodem

[Anonymous Coward]

Resume.

Re:in simplified terms, it's forward error correct

[globaljustin]

yeah TFA says it's link-layer flow control:

What if you could transmit data without link layer flow control bogging down throughput with retransmission requests, and also optimize the size of the transmission for network efficiency and application latency constraints? Researchers from MIT, Caltech and the University of Aalborg claimed to have accomplished this...

it can "piggyback" on TC-IP but you have to use their "software" on either end

RLNC encoding can ride on top of the TCP-IP protocol, so implementation does not require the replacement of communications equipment. But it does require software incorporating RLNC-licensed technology to execute on both ends.

i'm not sure where they want this to be used, "mobile" WiFi and LTE are mentioned but I dont see this as anything other than a demonstration of a capability, not an application

i come from a CCNA background but I've never worked with or seen something like this...seems like they would use it in data centers?

Better than FEC

[Anonymous Coward]

It seems the coding sequence does some compression. Forward error correction uses *extra* bits to allow bit errors to be corrected. The redundant data means that the TV station I'm watching on the other monitor is streaming data into my TV tuner card at 32 Mb/s, and the effective data rate is 19.39 Mb/s (so 12.61 Mb/s is the redundant data). They mentioned (at least twice) that the packet length for this scheme is exactly the same. So if its like FEC, its damned efficient.

Re:Better than FEC

[AK Marc]

There are 7/8 FEC (meaning 1/8 wasted on FEC), and if you code the FEC into the stream, then you'll hide the "loss". They look to be hiding the loss, so people don't complain about the "waste."

Good Improvement

[Davidlogann2]

If the RLNC is five times faster than the native lines, It mean that the entire transmission network may go on a roll?

More research needed

[ketomax]

How MIT and Caltech's Coding Breakthrough Could Accelerate Mobile Network Speeds

I wish they also made some breakthrough to increase the data caps we are stuck with.

Re:More research needed

[citizenr]

caps are Layer 8 problem

Re:in simplified terms, it's forward error correct

[tlambert]

And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?

This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?

I believe the issue is that you can't sell it to the cable companies and the DSL providers that implement PPPOE in order to track your surfing, to make sure you are buying your television programming from them, rather than file sharing, and they can intentionally make things like Tor not work. Not that PMTUD works on those things unless the modem proxies the ICMP messages, which are usually blocked by the cable companies, unless you explicitly ifconfig down to 1492 yourself, or enable active probing for black hole (rfc4821).

random?

[superwiz]

The random element seems to be there to avoid having to come up with an optimal distribution of information. Otherwise, it seems like a frame-level (or even finer) RAID.

Re:random?

[AK Marc]

Go look at Silver Peak. They do a WAN optimization that uses variable FEC to increase FEC as packet loss increases, to minimize waste as conditions change.

Packet loss models?

[antifoidulus]

TFA doesn't seem to state what their assumptions were on how packets get lost and how many packet losses the algorithm can deal with, and what their distribution is. There are a lot of ways you can drop k packets out of n packets sent.

If you assume that every packet has a k/n chance of being lost, then being able to reconstruct a single missing packet could be incredibly useful. However cell phone packet losses tend to be incredibly bursty, i.e. they will have a very throughput for a while, but then all of a sudden(maybe you changed towers or went under a bridge etc) lose a whole ton of packets. Can this algorithm deal with bursty losses? I wish TFA was a bit more clear on that

Re:Packet loss models?

[complete loony]

(I haven't read this paper yet, but I've read other Network Coding data and experimented with the idea myself)

With TCP, when a packet is lost you have to retransmit it. You could simply duplicate all packets, like using RAID1 for HDD redundancy. But this obviously wastes bandwidth.

Network coding combines packets in a more intelligent way. More like RAID5 / RAID6 with continuous adaptation based on network conditions. Any packet that arrives may allow you to deduce information that you haven't seen before. Basically each packet is the result of an equation like; f(p1, p2, p3) = a*p1 + b*p2 + c*p3. When each packet arrives, you attempt to solve the set of simultaneous equations you have received. When you have reduced each expression to just a single packet, you send it up the protocol stack.

You still need a TCP like acknowledgement scheme so that you can; rate limit the flow of packets based on measured congestion, tweak the percentage of redundant packets being sent due to measured packet loss, and advance the stream to include new data.

If you get the network coding parameters wrong, the connection still might stall, or you might be sending too much redundant data. If everything is going well, a link with 10% packet loss just means that your stream is transferred 10% slower.

Re:Packet loss models?

[Anonymous Coward]

This sounds like RAID for network data streams, with some fancy encoding to, I assume, compress the data (otherwise why not just send everything twice?) So what if you /can't/ reconstruct a lost packet because a subsequent packet is also lost? When you use something like UDP, you build recovery mechanisms into your app (if you care). Will anyone who uses RNLC do that?

Fountain/Raptor codes?

[Anonymous Coward]

Isn't this just raptor encoding? This idea has been around for a long time, I'd like to hear how this specific instance differs from what came before.

http://en.wikipedia.org/wiki/Raptor_code

Re: Fountain/Raptor codes?

[Anonymous Coward]

It is like raptor codes, bit with higher decoding complexity. Instead of carefully choosing the coefficents they draw them at random. Good thing is that they are cover ed by differenti patents.

A better explanation

[grahamsz]

The story linked to seems to have an awful explanation of what's going on. This makes a lot more sense:

http://www.codeontechnologies....

Reminds me a little of a random project I started back in college where I'd transmit a file in a bunch of packets where each contained the original file modulo a specified prime number. That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them they could use the extended euclidean algorithm to reconstruct the original file.

I was hoping we'd someday be able to multicast udp over the net to multiple random locations and this would be a fast way to send files.

Re:A better explanation

[Ingenium13]

It might be possible now with IPv6.

Re:A better explanation

[gnasher719]

That link shows their website, where they tell us that if the sender is about to send packet 83, and figures out it didn't get an acknowledgement for packet 22, it then has to resend packets 22 to 82. Which seems an entirely stupid thing to do, and an obvious improvement would be to resend only the packets that are actually lost.

Guess what: According to the Wikipedia article about TCP, that's what the "selective acknowledgment" (SACK) option defined in RFC 2018 does. So _poof_ goes the benefit of this scheme, which was based on an incorrect representation of TCP/IP.

On the other hand, my TV receives a purely digital signal with no way to ask for re-transmission of lost data and works just fine with it. There is a variant of h.264 specicially for streaming connections without retransmission, and that together with UDP instead of TCP/IP would solve the problem just fine. If it was a problem.

Re:A better explanation

[Anonymous Coward]

Most video streaming protocols (like RTP/RTCP) are UDP based due to the faster data transfer. It's up to the application to decide what to do if any packets are dropped or arrive out of order. ie, if the packets is too old, ignore it and let the video "skip".

wtf?

[Anonymous Coward]

More importantly, RLNC encoding can ride on top of the TCP-IP protocol, so implementation does not require the replacement of communications equipment.

What? Why would you do that? The whole point of RLNC from the article is that it's to avoid retransmissions due to packet loss. Why would you put that on top of a streaming protocol like TCP/IP that's got all the flow control and retry logic happening under the covers? Surely you'd run RLNC over UDP!

The RLNC-encoded video was downloaded five times faster than the native video stream time, and the RLNC-encoded video streamed fast enough to be rendered without interruption.

That's probably because you ran that test with RLNC over UDP where there was no flow control and retransmissions. UDP streaming without RLNC is probably just as fast.

Re:wtf?

[AK Marc]

Why would you put that on top of a streaming protocol like TCP/IP that's got all the flow control and retry logic happening under the covers? Surely you'd run RLNC over UDP!

UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.

Re:wtf?

[Anonymous Coward]

UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.

UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.

Re:wtf?

[AK Marc]

UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.

http://en.wikipedia.org/wiki/I... TCP/IP is shorthand for The Internet Protocol Suite. UDP is a protocol in The Internet Protocol Suite. Therefore, you are 100% wrong, in your wording, UDP is a subset of TCP. You just chose to leave off the "/IP" because you know you are wrong.

halve the packets?

[Anonymous Coward]

does this effectively negate the need for every other packet in a loss-less environment?

Re:halve the packets?

[citizenr]

yes, by sending packet twice the size

Re:in simplified terms, it's forward error correct

[AK Marc]

I think they are lying.

Many of the details fail under closer examination. It doesn't "use" TCP-IP. It could use a propritary IP stack that is TCP/IP compatible. So it'll use IP addresses and port numbers like a TCP packet would so that switches and routers in the middle wouldn't know or care what it's doing. If they put it on an IPX/SPX core it'd fail to route across the Internet. But it isn't TCP. It doesn't use a TCP compliant stack. It just looks like one to the outside world. It will not re-transmit a lost packet via TCP mechanisms. But it'll work on the same hardware on both ends and software in the middle. You just have to replace the network stack on both ends, which isn't hard. Though they indicate that the only thing it needs is "software" on both ends. Maybe they'll be doing it over actual TCP/IP. That, and the way they keep saying TCP-IP, that includes UDP. They don't say TCP, or UDP. And I don't trust them when they keep saying TCP-IP, it should be a slash, not a dash. So is it running over TCP/IP (which could mean UDP)? Or does it run over TCP (which excludes UDP)?

Right.

[Animats]

This is yet another form of forward error correction. The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.

This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.

Re:Right.

[MobyDisk]

The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.

Yeah, that would be impossible. But I don't think they meant to claim that there is no extra data. When they say "The combined packet length is no longer than either of the two packets from which it is composed." I took that to mean that each packet is a fixed length. Not that they didn't add extra data to get to that packet length.

This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.

Yeah. It might be good for for wireless networks, where even a "5 bar" wireless network connection has a fairly consistent 1% packet loss. The QUIC protocol is another attempt to handle this better by using packet pacing, but QUIC isn't worth it in general even if it addresses this one specific problem.

PAR2 for packets?

[Anonymous Coward]

Seems plausible. I hope the patent reviewers are knowledgable on prior art on this one.

Excellent!

[davidc]

This means my mobile internet speed might soon be up to 10 bps instead of the 2 bps I seem to get at the moment!

Re:Excellent!

[Anonymous Coward]

It means that your ISP can oversell their bandwidth even more.

I've seen this before

[statemachine]

Except it's called MPEG video. And it's used for TV.

MPEG also has a mode to recover the errors, but it's expensive, and when you're streaming, who cares? If your link sucks, you don't blame the stream.

Turbo codes

[Anonymous Coward]

And I barely understand those and they go back to 1960...

Good press release / slashvertisement

[Anonymous Coward]

"Code On" has done well at generating some buzz.
Unfortunately, the only details on their website are of the type "this is awesome", with no description of how the breakthrough works.

http://www.codeontechnologies.com/technology/white-papers/

I just wish they would post actual details on the encoding method. How does it compare to Fountain Codes such as RFC 5053?

only send the last packet?

[Anonymous Coward]

so... if all the packets include the last packet and the current one... then all you need to send is the last packet and the rest can be inferred? i.e.

packet 256 contains packet 255 which contains 254 which contains 253?

Re:only send the last packet?

[skids]

Usually in such schemes you can recover a packet if you have the surrounding packets before and after it.

been there, done that... (Sqore:1,000,000)

[Anonymous Coward]

It's called U.D.P. (The periods help you read it slowly).

Geez! Really, this is an improvement?!

Really?

[Viol8]

"That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them they could use the extended euclidean algorithm to reconstruct the original file."

So if the receiver got 10K copies of the 1st packet and nothing else it could still reconstruct the file? Thats impressive. Which college was this, Hogworts?

Re:Really?

[skids]

So if the receiver got 10K copies of the 1st packet and nothing else it could still reconstruct the file?

Considering each one is only sent once, that would be some feirce level of broken compound load balancing.

Note he said each packet contains the modulus of the entire original file with a different prime. The only thing that would cause duplicate packets would be running out of acceptably-sized primes.

Re:Really?

[Viol8]

You need to re-read what he wrote:

"That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them "

How can you split something into 10K packets then get *any* 10K of them? He's either saying he sends more than 10K or the receiver has to receive all 10K packets that were sent to reconstruct the file which I doubt is what he meant.

Re: Really?

[grahamsz]

Yeah that could have been more clear. I'm suggesting sending a constant stream of packets until the target asks you to stop. No need for windowing or flow control, but pretty computationally expensive

Nucleus

[zawarski]

The greatness of human accomplishment has always been measured by size. The bigger, the better. Until now. Nanotech. Smart cars. Small is the new big. In the coming months, Hooli will deliver Nucleus, the most sophisticated compression software platform the world has ever seen. Because if we can make your audio and video files smaller, we can make cancer smaller. And hunger. And AIDS.

That's the over-simplified version?

[wonkey_monkey]

In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.

Uh... could you simplify it just a little more?

How does a "later-sequenced packet [...] include earlier-sequenced packets"?

Re:That's the over-simplified version?

[EmperorArthur]

They say "randomly" generated coefficients, but I'll bet you can use a psudo random number generator and pass in the same seed value to both the sender and reciever. Bam, now both sides have the same set of semi random coefficients to use when doing the fancy linear algebra.

Re:That's the over-simplified version?

[tibit]

The linear algebra isn't even fancy. They need to do a lot of linear equation solving, that's about it.

huffman error correction was in fax machines

[Anonymous Coward]

so this really doesn't seem like a 'breakthrough'. It's just a new application of existing technologies. http://www.fileformat.info/mir...

Time saver

[PsyMan]

Now I can free up 4 of the 5 minutes it used to take burning through my monthly bandwidth to do something constructive.

IANANE

[Anonymous Coward]

What if you could actually understand the first sentence of the summary?

Or even the last sentence?

parses like a teaspoon of sugar

[epine]

I've been parsing this kind of press release for a long, long time now. I can pretty much tell what we're dealing with by how hard it is to state the advantage of a new approach in narrow and precise language.

That this blurb doesn't even disclose the error model class (error correction is undefined without doing so) suggests that the main advantage of this codec lies more in the targeting of a particular loss model than a general advance in mathematical power.

Any error correction method looks good when fed data that exactly corresponds to the loss model over which it directly optimises.

The innovators of this might have something valuable, but they are clearly trying to construe it as more than it is. This suggests that there are other, equally viable ways to skin this particular cat.

Re: in simplified terms, it's forward error correc

[Anonymous Coward]

My thoughts exactly. To test this with 3% loss, and no retransmits on existing cell phones, the test must have been over UDP.

The issue I'm having is they're comparing this to http streaming of a movie; implying TCP. Perhaps if they wrote their own TCP layer in the stack to still send retransmits if there is too much packet loss, such that the application can still open a TCP socket (expecting in-order lossless transmission) then it would work; so long as no network gear in between inspects layer 4.

Sounds like a Fountain Code to me

[ka9dgx]

This sounds exactly like a Fountain Code to me, which isn't exactly news.

Re:in simplified terms, it's forward error correct

[Mariner28]

Like Anonymous Coward, yours were my thoughts exactly. Why would one use TCP to stream video? It's one of the tenants of networking that losing packets is preferable to increasing jitter in the video or audio feed. And off the top of my head, I'd say that there hasn't been a widely used connection-based layer 2 protocol since X.25. Hell, that's why Frame Relay and later ATM were invented - to let the transport layer handle error detection (and retransmission if required). Even Ethernet uses just a CRC for forward error correction - if the receiver can't fix errors, the frame is dropped. It's up to the upper layers do actually do anything about it. And let's not get started about a 3% random error distribution in a wireless link - everyone knows that fading causes a whole stream of consecutive packets to be lost, not just an even statistical distribution of them. Stephen Max Patterson at Network World just proved he isn't qualified to write for Network World... And just a nit for you, AK Marc - if someone says UDP is "running over TCP/IP", tell them to put down the router and step away from the rack. They just aren't qualified.

Re:in simplified terms, it's forward error correct

[RatherBeAnonymous]

I think the writer is confused. This sounds like a non-routed layer 2 error-correction protocol for error prone networks, like cell phone data and Wi-Fi. The only relation this has to the Internet is that it can carry TCP/IP traffic, just like Ethernet, Frame Relay, ATM, and any number of other layer 2 protocols can carry TCP/IP.

Re:in simplified terms, it's forward error correct

[AK Marc]

I think they are using UDP, but didn't want to get into specifics, so the writer screwed it up. UDP is a member of the TCP/IP suite of protocols.

Re:in simplified terms, it's forward error correct

[AK Marc]

And just a nit for you, AK Marc - if someone says UDP is "running over TCP/IP", tell them to put down the router and step away from the rack. They just aren't qualified.

How about "UDP is a member of the TCP/IP suite of protocols"? Though most leave off the "suite of protocols" at the end, it's implied. I know lots of people smarter than you that would say "UDP is a subset of TCP/IP" or some other wording to that effect. Even Wikipedia agrees with me. TCP/IP is shorthand for The Internet Protocol Suite, which includes UDP. So UDP "runs over" TCP/IP

Re:in simplified terms, it's forward error correct

[RatherBeAnonymous]

Yeah, they probably are using UDP in some video tests, but I think that is irrelevant. If this is a competing technology to Reed-Solomon encoding, then it is almost certainly a data-link layer protocol and is agnostic to network or session protocols.

From Wikipedia:

Reed–Solomon codes have since found important applications from deep-space communication to consumer electronics. They are prominently used in consumer electronics such as CDs, DVDs, Blu-ray Discs, in data transmission technologies such as DSL and WiMAX, in broadcast systems such as DVB and ATSC, and in computer applications such as RAID 6 systems.

I think the writer's mistake was in seeing everything through Internet-colored glasses. The article specifically says "link layer", not "session layer". TCP is a session layer technology. My best guess from the article is that this is not an end-to-end error correction protocol to reduce IP packet retransmission. It is an error correction protocol to reduce OSI layer 2 packet (or more properly, "frame") retransmission.

Could someone explain why...

[Anonymous Coward]

If any particular packet can be "recreated" from information in a different packet, I can't figure out how that doesn't halve the amount of data going through the pipe. If it doesn't, can someone tell me how significantly differs from just some great compression applied to the packet to just compress two packets into the size of one?

Or, was that pretty much the point?

Re:Could someone explain why...

[MidSpeck]

I have the same question. From the article, "This is because RLNC can recreate any packet lost on the receiving side from a later sequenced packet." If this is truly the case, just send me the very last packet and I can recreate all the packets that were supposed to come before by working backwards.

Re:in simplified terms, it's forward error correct

[AK Marc]

It is an error correction protocol to reduce OSI layer 2 packet (or more properly, "frame") retransmission.

But the "frame" doesn't exist end-to-end, but is re-created for each network segment. So their statements that it runs on the endpoints and doesn't require any changes in the middle indicates it is a higher layer protocol. Some of the layer 2 already include FEC. So why have double FEC? (it's bad). It's not double FEC at the same layer, but an application-layer FEC (based on their description). You can have them on different layers because they are correcting different things. L-2 FEC is correcting bit errors from noise or other physical issues. L-7 FEC is correcting lost packets.

But the one thing we do know for sure is that the article is crap. It's detailed when it shouldn't be, but leaves out all useful details. That takes some skill. But skill in what, we aren't sure.