The only solution is to regularly read everything:
The chance of avoiding double errors in the form of unreadable sectors during rebuild about doubles each time you halve the time between full reads of all sectors on a drive. (True to about weekly full reads.)
This is because a full read will allow each drive in the array to discover sectors that are becoming iffy (soft/recoverable read errors) and then remap them.
See lwn.net for a discussion and links to some good papers.
It seems like the US is both blessed and cursed by the same short-term profit based thinking:
It is relatively easy to get competing providers for internet connectivity, but each of those providers would much rather squeeze the maximum amount of short-term profits out of their costumers rather than trying to deliver a great product with payback times measured in years.
Here in Norway we put up a cabin in the central mountains a couple of years ago (i.e. a _very_ sparsely populated area), but as soon as we ordered an electric power hookup, the local power company by default also pulled a fiber along.
At home in Oslo I have had ADSL for ~5 years, currently getting 7/1 Mbit/s down/up at a cost of around $60 (NOK is very strong vs $ these days), but now I've had two different fiber providers offering FTTH so in a week or two I'll get symmetrical 30 Mbit/s for the same cost as ADSL.
The last two times I've been to a job interview, it has been at least as important for me to meet the people I would be working with, as for them to meet me.
In fact, on my last job interview I came in twice, the second time was so that I could hold a highly technical presentation in front of all the people I would be working with. This was in order for me to meet them all and see if they understood the stuff I really care about.
Terje
PS. I didn't take the job, but that was for unrelated reasons, the interview and the people were great.
A few decades ago, Norwegian arctic researchers wanted to determine if polar bears really hibernate (like the brown bear does), or just take a lot of long naps.
To check this the father of a guy I knew used to dig/push his way into polar bear burrows, with a revolver and a rectal thermometer in front of him.
He never had to fire the gun, but the temp readings he got showed that the polar bears were only sleeping, not hibernating.
This is a perfect sport for a thinking person, where you start individually, run/jog/walk around in the forest for some time trying to visit a number of control points (marked on the map you got at the start), before ending up at the finish line.
The main point is that you have no spectators at all, you move around at your own speed, and while running you have to concentrate a lot on finding the best possible route to the next control. (In very steep uphills I often find that I need to walk a lot "to check the map".:-)
Since a lot of orienteers are geeks, we have software available to organize events, custom RFID tags used to document each control point as you visit it, and lots of software to do post-competition analysis of your run.
I live in Oslo, Norway where I can take part in 'O' events 3-4 times each week, but in the US (if this is where you live) there are active clubs in all the main geek areas (Bay Area, Seattle, Portland etc).
When I visited Seattle in April I contacted the local club and got electronic copies of three maps in the area that have permanent courses on them.
This has been tested many times, it seems like 10 years, and/or 10K hours of training, is required to gain mastery at any difficult activity, physical or mental.
I.e. this holds for chess as well as programming, karate, running, rock climbing or orienteering (which is about half & half physical & mental).
You need a special camera version which contains firmware (hopefully tamperproof) which uses public key crypto to digitally sign each photo as it taken, making it possible to prove that the photo file hasn't been modified at all.
One example is the Fujifilm IS Pro which can be delivered in this form:
Any bomb that fits easily into a standard freight container is already a horrible nightmare:
These containers travel worldwide, are rarely inspected if the paperwork seems to be OK, and they can easily stay in a harbor area of a major city for many months.
The only trigger you need is a cell phone, so you can preplace them wherever you like and blow up any coastal city in the world, whenever you want to.
Stopping this scenario is probably (or should be) the real nightmare for most of the three-letter agencies in the world.
The theory is still good, it depends on all QC channels having a non-zero error rate, even when nobody is trying to eavesdrop, so the protocol used must be able to deal with those errors, right?
If the real error rate is well below the communication error ceiling (where it stops working), then Charlie sitting in the middle can extract a few bits out of each packet.
OTOH, assuming this channel is used to exchange the 256-bit AES key to be used for the bulk communication, then the parties can simply set the acceptable error rate very close to the real limit, and then discard any packet (and the corresponding key!) which goes above this limit, even if the contents are recoverable using ECC coding.
Doing this would make it possible to guarantee that maximum X bits (approximately equal to the channel error rate) can ever be eavesdropped out of the 256-bit key, leaving more than enough unknown bits as to make the communication link totally secure.
Okay, let's start with my instructions to you: no matter how enticing I may make this film sound, do NOT spend money on it. Don't see it in the theater, and don't rent it. Buying the DVD would be a crime against humanity. For that matter, don't bother seeing it for FREE, either. Spending your TIME on this film is a crime against your employer, your family, and the Baby New Year. You would be better off using an hour and thirty-four minutes eating junk food and watching Weather Channel repeats you've accidentally TIVO'd.
They key to hiding data in executables is to realize that there are many instructions with multiple possible encodings.
You can also reverse the order of many comparison operations as long as you also modify the following branch/set instructions.
If you want to jam such a channel you would have to do the same job, first identifying all the possible locations for such transformations, then randomly flip half of them.
(Un?)fortunately neither the encoding nor the jamming process can be totally secure, because you can check (or know up front) which compiler had generated the original executable, then decompile/recompile and check which encodings the compiler tend to use.
Thanks for making my day!:-)
You're Terje Mathisen.
For most of us the brilliant flash of insight that allows us to fully integrate a problem is an epiphany - that rare and fleeting moment where our brilliance truly shines. It's neither frequent nor persistent enough to build a plan on which we must rely.
For you it's more like the rising and setting of the sun. It most certainly isn't anything like that, great code is almost always a result of Edison-like effort:
"10% inspiration, 90% perspiration."
Great code, at least for me, depends on being willing to "try & try again", something that's only possible when I have the luxury of not working to a deadline.
Last fall I wrote the worlds fastest Ogg Vorbis decoder, starting from Sean T Barret's public domain code:
To get the last 30% of speed (to beat the best existing decoder) I had to throw away at least 75-80% of all my optimization attempts, since they turned out to slow it down instead of speeding it up like my insight told me it should do.
I.e. even when working on what I really love, I can be wrong a lot more often than I'm right.:-(
There are two ideas of thought about corner cases (and the GP pointed out one). Thought #1) (GP) There's no such things as a corner. It is a requirement - it may be that fewer people/fewer processes use it; but, it is still a section of the total solution that must be designed to overcome some problematic section. Otherwise, why is the code being written?
Thought #2) Corner cases only effect a small number of your user-base; therefore, code to satisfy 95%-99% of your customers. The underlying principle here is that the manager will wait for another release. This approach is usually taken when the project manager failed to account for something and says (and I quote), "We'll just re-design it after the first release." I have taken part in a few optimization competitions, and each time #1 has been a crucial part of the solution:
The usual approach is to optimize the 90-95% case, then bail on the remainder, but this will almost always be beaten by code which manages to turn everything into the "normal" case, with no if/else handling, no testing, no branching.
When I was beaten by David Stafford in Dr.Dobbs Game of Life challenge, I had lots of specialcase code to handle all the border cases, while David had managed to embed that information into his lookup tables and data structures. (He had also managed to make the working set so much smaller that it would mostly fit in the L1 cache.:-)
When my Pentomino solver won another challenge, being twice as fast as #2, the crucial idea was to make the solver core as tiny as possible, with very little data movement and the minimum possible number of tests.
As the MIT report states, the key was to make the chip operate at 0.3 V instead of ca 1.0V
Since power usage is (roughly!) proportional to voltage squared, getting the chip to run at less than one third the usual voltage will indeed give an order of magnitude reduction in power usage.
From the report:
One of the biggest problems the team had to overcome was the variability that occurs in typical chip manufacturing. At lower voltage levels, variations and imperfections in the silicon chip become more problematic. "Designing the chip to minimize its vulnerability to such variations is a big part of our strategy," Chandrakasan says. I.e. current state of the art transistors does not work reliably at such voltage levels, I'm guessing that they have to give up significant parts of the theoretical power reduction in order to make it work at all.
It has been quite obvious to several people in the usenet news:comp.arch newsgroup that the future should give us chips that contain multiple cores with different capabilites:
As long as all these cores share the same basic architecture (i.e. x86, Power, ARM), it would be possible to allow all general-purpose code to run on any core, while some tasks would be able to ask for a core with special capabilites, or the OS could simply detect (by trapping) that a given task was using a non-uniform resource like vector fp, mark it for the scheduler, and restart it on a core with the required resource.
An OS interrupt handler could run better on a short pipeline in-order core, a graphics driver could use something like Larrabee, while SPECfp (or anything else that needs maximum performance from a single thread would run best on an Out-of-Order core like the current Core 2.
The first requirement is that Intel/AMD must develop the capability to test & verify multiple different cores on the same chip, the second that Microsoft must improve their OS scheduler to the point where it actually understands NUMA principles not just for memory but also cpu cores. (I have no doubt at all that Linux and *BSD will have such a scheduler available well before the time your & I can buy a computer with such a cpu in it!)
So why do I believe that such cpus are inevitable?
Power efficiency!
A relatively simple in-order core like the one that Intel just announced as Atom delivers maybe an order of magnitude better performance/watt than a high-end Core 2 Duo. With 16 or 80 or 256 cores on a single chip, this will become really crucial.
Terje
PS As other posters have noted, keeping tomorrow's multi-core chips fed will require a lot of bandwith, this is neither free nor low-power.:-(
I'm Norwegian, and I've watched most of these programs when they were originally broadcast:
The discussion page on http://nrkbeta.no/last-ned-lars-monsens-nordkalotten-365-gratis-og-i-full-kvalitet/ (Norwegian only) contains a lot of comments from the NRK people where they answer questions about all kinds of technical details (camera, sw, scaling/de-interlacing from 1080i to 576p, redoing the first episode to improve the quality etc.)
They also explain that the main/only reason they cannot do this with most of their productions is due to licensing issues, particularly for music for the soundtrack.
For this series they planned around this from the beginning, including commissioning a custom musical soundtrack.
It really sounds like they are trying to make BT distribution the default some time in the future, even though this will cut into the income they currently get from DVD sales.
BluRay disks needs to be higher capacity, since they (as I noted in my first post) also allow higher encoding bitrates: 40 vs 30 Mbit/s.
However, even with 1080p content, HD-DVD's 30 Mbit/s seems to be enough to encode even very dynamic content with no visual errors.
I have watched many times a number of 720p and 1080p test videos, of stuff like shuttle takeoffs and fighter aircraft exhibitions, encoded with various bitrates from 15 to 40 Mbit, and the difference (at least to my 50-year old eyes) is actually _very_ small, with the step from 30 to 40 seemingly giving no actual improvement.
The 40 Mbit cap might become important when even higher resolutions, like 1440p or 2160p become available at some point in the (far off?) future.
Graphics Processing Units (GPUs) are very good at doing repetitive fp and fixed-point operations, and much less good a bit-twiddling. I.e. the motion compensation stage of video decoding, where you copy (possibly sub-pixel-located) source blocks into the target frame has been handled by graphics hw since the very first sw DVD players, like Zoran's SoftDVD which was the first.
(In fact, SoftDVD was capable of handling 30 fps even without hw assists, running on a PentiumMMX 300 Mhz cpu, and without dropping any frames, but having the motion comp hw made it much easier to avoid drops. BTW, I did a very small bit of asm optimization work on that sw player.)
High bitrate HD/BR video is encoded using the CABAC (Content-Adaptive Binary Arithmetic Coding) algorithm, which provides slightly better compression rates, but which is also particularly unsuited to a GPU: Each decoding step requires multiple if/then/else stages, just to decode a single bit of information. It is also completely serial, in that you normally cannot determine the context to use for the next decoded bit until you've finished the current bit and possibly even branched on the result!
When you need to do this more than 50 million times/second, CABAC becomes the real bottleneck.
I have spent a couple of months optimizing code for HD decoding, and mostly the format doesn't really matter:
Both use the same codecs, they support the same resolutions, and the maximum bitrate is more or less the same (30 vs 40 Mbit/s for HD vs BR).
The one important difference is that a "full HD" 1080x1920 BR frame will always be encoded as four quadrants, each at 540x960.
This does lead to marginally lower compression rates, since you get more borders, but the great benefit is that you can have multiple CPU cores (up to 4) work in parallel on each of the parts!
You can of course do the same with a multi-core decoder for HD-DVD, but only by starting each cpu/thread at a different key frame, and since each 1080p picture requires 2 Mpixels, it is far too easy to trash both the TLB tables and the L2 caches when doing the motion compensation step which normally requires multiple source frames to be available to generate each target frame.
The Confluence Project http://confluence.org/ is an international effort to perform a systematic sampling of the Earth's surface, i.e. all those locations where both longitude and latitude has integer values.
So far more than 10,000 visitors have documented more than 5,000 of these points.
http://www.agu.org/pubs/crossref/2006/2006GL027768.shtml is a link to a paper by a Japanese researcher (Koki Iwao) and his associates: They have used the DCP information to check/verify the quality of the various land cover databases:
Which parts of the Earth is mountains/lakes/forests/rice fields/grassland/etc.?
What they found is that the best of these databases have a hit ratio of just 60% or less.
I have recently written what I believe is the world's fastest Ogg Vorbis decoder, it takes about 600 ms to decode my longest song sample (4:05 minutes encoded with 192 Kbit/s for a final filesize of 5.7 MB).
IMHO there are just a few problems with Vorbis, cpu load is not one of them:
a) It is not at all suitable for contineous streaming, with multiple receivers connecting/disconnecting on the fly, since you have to start by decoding the 4-8 KB header before you can make any sense of the sound frames.
b) To get decent decoding performance, you have to unpack & cache all the codebook information in the header packets, this requires from about 50 to 300 KB, which can be significant in a small device.
c) Even though Vorbis is in theory independent of the Ogg container format, most existing source code expects to find Ogg frames surrounding all Vorbis packets. This is an implementation and not a specification problem.
d) Vorbis really prefers to have fast fp support available, but Theora is an open-source fixed-point implementation which has been used as the starting point for quite low-resource embedded implementations.
I do kite sailing in the winter here in Norway, and the kite shown in the article is almost identical, except for size of course, with the kite I use. (I have also windsurfed since around 1990.)
My kite is a Peter Lynn Venom II http://www.peterlynnkiteboarding.com/, this is a twinskin kite which keeps its airfoil shape due to internal air pressure: A set of small mesh openings in the leading edge allows air into the opening between the front and back side.
This form of kite is an airfoil, not a spinnaker, the difference is huge:
A spinnaker is effectively a large bag to catch the wind, while a kite works best by having air moving faster on one side than the other. Among other things, this means that a kite allows you to sail much faster at an angle to the wind instead of straight downwind.
Another nice trick you can do with a kite, unlike a windsurfing rig, it to let the kite loop around in little figure-of-eights: This makes the airfoil move even faster through the air, increasing the lift particularly during a lull in the wind.
I do get significant amounts of spam, I'm probably left with about 30-100 per day that manage to pass through both the corporate filters and SeaMonkey's adaptive rules.
Getting rid of the rest takes me a minute or less, which I consider time well spent when the alternative would be to give in to the spammers and switch to a secret address.
I've gotten _many_ tips/requests/leads via email over the years, some of them led to very interesting projects.
Slashdot Mirror
The only solution is to regularly read everything:
The chance of avoiding double errors in the form of unreadable sectors during rebuild about doubles each time you halve the time between full reads of all sectors on a drive. (True to about weekly full reads.)
This is because a full read will allow each drive in the array to discover sectors that are becoming iffy (soft/recoverable read errors) and then remap them.
See lwn.net for a discussion and links to some good papers.
Terje
It seems like the US is both blessed and cursed by the same short-term profit based thinking:
It is relatively easy to get competing providers for internet connectivity, but each of those providers would much rather squeeze the maximum amount of short-term profits out of their costumers rather than trying to deliver a great product with payback times measured in years.
Here in Norway we put up a cabin in the central mountains a couple of years ago (i.e. a _very_ sparsely populated area), but as soon as we ordered an electric power hookup, the local power company by default also pulled a fiber along.
At home in Oslo I have had ADSL for ~5 years, currently getting 7/1 Mbit/s down/up at a cost of around $60 (NOK is very strong vs $ these days), but now I've had two different fiber providers offering FTTH so in a week or two I'll get symmetrical 30 Mbit/s for the same cost as ADSL.
Terje
The last two times I've been to a job interview, it has been at least as important for me to meet the people I would be working with, as for them to meet me.
In fact, on my last job interview I came in twice, the second time was so that I could hold a highly technical presentation in front of all the people I would be working with. This was in order for me to meet them all and see if they understood the stuff I really care about.
Terje
PS. I didn't take the job, but that was for unrelated reasons, the interview and the people were great.
True story:
A few decades ago, Norwegian arctic researchers wanted to determine if polar bears really hibernate (like the brown bear does), or just take a lot of long naps.
To check this the father of a guy I knew used to dig/push his way into polar bear burrows, with a revolver and a rectal thermometer in front of him.
He never had to fire the gun, but the temp readings he got showed that the polar bears were only sleeping, not hibernating.
See the polar bear FAQ.
Terje
Please take a close look at orienteering.
This is a perfect sport for a thinking person, where you start individually, run/jog/walk around in the forest for some time trying to visit a number of control points (marked on the map you got at the start), before ending up at the finish line.
The main point is that you have no spectators at all, you move around at your own speed, and while running you have to concentrate a lot on finding the best possible route to the next control. (In very steep uphills I often find that I need to walk a lot "to check the map". :-)
Since a lot of orienteers are geeks, we have software available to organize events, custom RFID tags used to document each control point as you visit it, and lots of software to do post-competition analysis of your run.
I live in Oslo, Norway where I can take part in 'O' events 3-4 times each week, but in the US (if this is where you live) there are active clubs in all the main geek areas (Bay Area, Seattle, Portland etc).
When I visited Seattle in April I contacted the local club and got electronic copies of three maps in the area that have permanent courses on them.
Good luck!
Terje
This has been tested many times, it seems like 10 years, and/or 10K hours of training, is required to gain mastery at any difficult activity, physical or mental.
I.e. this holds for chess as well as programming, karate, running, rock climbing or orienteering (which is about half & half physical & mental).
Terje
The requirements are pretty though though:
You need a special camera version which contains firmware (hopefully tamperproof) which uses public key crypto to digitally sign each photo as it taken, making it possible to prove that the photo file hasn't been modified at all.
One example is the Fujifilm IS Pro which can be delivered in this form:
dpreview Fuji IS Pro review
Terje
The way I would have done this would have been to use a particular SMS text message, and not just a voice call.
:-(
With 140 bytes (160 7-bit chars), you can make the detonation key arbitrarily un-guessable.
For extra credit, add a dead-man switch: An encrypted message which must be received every day (hour/week/whatever) to delay detonation.
At this point you really wouldn't want to experience a longterm cell phone outage.
Terje
Any bomb that fits easily into a standard freight container is already a horrible nightmare:
These containers travel worldwide, are rarely inspected if the paperwork seems to be OK, and they can easily stay in a harbor area of a major city for many months.
The only trigger you need is a cell phone, so you can preplace them wherever you like and blow up any coastal city in the world, whenever you want to.
Stopping this scenario is probably (or should be) the real nightmare for most of the three-letter agencies in the world.
Terje
The theory is still good, it depends on all QC channels having a non-zero error rate, even when nobody is trying to eavesdrop, so the protocol used must be able to deal with those errors, right?
If the real error rate is well below the communication error ceiling (where it stops working), then Charlie sitting in the middle can extract a few bits out of each packet.
OTOH, assuming this channel is used to exchange the 256-bit AES key to be used for the bulk communication, then the parties can simply set the acceptable error rate very close to the real limit, and then discard any packet (and the corresponding key!) which goes above this limit, even if the contents are recoverable using ECC coding.
Doing this would make it possible to guarantee that maximum X bits (approximately equal to the channel error rate) can ever be eavesdropped out of the 256-bit key, leaving more than enough unknown bits as to make the communication link totally secure.
Terje
This is definitely the time to direct everyone to the rant Howard Tayler of Schlock Mercenary fame wrote about BloodRayne:
http://www.schlockmercenary.com/blog/index.php/2006/01/06/movie-review-bloodrayne/
Okay, let's start with my instructions to you: no matter how enticing I may make this film sound, do NOT spend money on it. Don't see it in the theater, and don't rent it. Buying the DVD would be a crime against humanity. For that matter, don't bother seeing it for FREE, either. Spending your TIME on this film is a crime against your employer, your family, and the Baby New Year. You would be better off using an hour and thirty-four minutes eating junk food and watching Weather Channel repeats you've accidentally TIVO'd.
Terje
They key to hiding data in executables is to realize that there are many instructions with multiple possible encodings.
You can also reverse the order of many comparison operations as long as you also modify the following branch/set instructions.
If you want to jam such a channel you would have to do the same job, first identifying all the possible locations for such transformations, then randomly flip half of them.
(Un?)fortunately neither the encoding nor the jamming process can be totally secure, because you can check (or know up front) which compiler had generated the original executable, then decompile/recompile and check which encodings the compiler tend to use.
Terje
For most of us the brilliant flash of insight that allows us to fully integrate a problem is an epiphany - that rare and fleeting moment where our brilliance truly shines. It's neither frequent nor persistent enough to build a plan on which we must rely.
For you it's more like the rising and setting of the sun. It most certainly isn't anything like that, great code is almost always a result of Edison-like effort:
"10% inspiration, 90% perspiration."
Great code, at least for me, depends on being willing to "try & try again", something that's only possible when I have the luxury of not working to a deadline.
Last fall I wrote the worlds fastest Ogg Vorbis decoder, starting from Sean T Barret's public domain code:
To get the last 30% of speed (to beat the best existing decoder) I had to throw away at least 75-80% of all my optimization attempts, since they turned out to slow it down instead of speeding it up like my insight told me it should do.
I.e. even when working on what I really love, I can be wrong a lot more often than I'm right.
Terje
Thought #1) (GP) There's no such things as a corner. It is a requirement - it may be that fewer people/fewer processes use it; but, it is still a section of the total solution that must be designed to overcome some problematic section. Otherwise, why is the code being written?
Thought #2) Corner cases only effect a small number of your user-base; therefore, code to satisfy 95%-99% of your customers. The underlying principle here is that the manager will wait for another release. This approach is usually taken when the project manager failed to account for something and says (and I quote), "We'll just re-design it after the first release." I have taken part in a few optimization competitions, and each time #1 has been a crucial part of the solution:
The usual approach is to optimize the 90-95% case, then bail on the remainder, but this will almost always be beaten by code which manages to turn everything into the "normal" case, with no if/else handling, no testing, no branching.
When I was beaten by David Stafford in Dr.Dobbs Game of Life challenge, I had lots of specialcase code to handle all the border cases, while David had managed to embed that information into his lookup tables and data structures. (He had also managed to make the working set so much smaller that it would mostly fit in the L1 cache.
When my Pentomino solver won another challenge, being twice as fast as #2, the crucial idea was to make the solver core as tiny as possible, with very little data movement and the minimum possible number of tests.
Terje
Since power usage is (roughly!) proportional to voltage squared, getting the chip to run at less than one third the usual voltage will indeed give an order of magnitude reduction in power usage.
From the report: One of the biggest problems the team had to overcome was the variability that occurs in typical chip manufacturing. At lower voltage levels, variations and imperfections in the silicon chip become more problematic. "Designing the chip to minimize its vulnerability to such variations is a big part of our strategy," Chandrakasan says. I.e. current state of the art transistors does not work reliably at such voltage levels, I'm guessing that they have to give up significant parts of the theoretical power reduction in order to make it work at all.
Terje
It has been quite obvious to several people in the usenet news:comp.arch newsgroup that the future should give us chips that contain multiple cores with different capabilites:
:-(
As long as all these cores share the same basic architecture (i.e. x86, Power, ARM), it would be possible to allow all general-purpose code to run on any core, while some tasks would be able to ask for a core with special capabilites, or the OS could simply detect (by trapping) that a given task was using a non-uniform resource like vector fp, mark it for the scheduler, and restart it on a core with the required resource.
An OS interrupt handler could run better on a short pipeline in-order core, a graphics driver could use something like Larrabee, while SPECfp (or anything else that needs maximum performance from a single thread would run best on an Out-of-Order core like the current Core 2.
The first requirement is that Intel/AMD must develop the capability to test & verify multiple different cores on the same chip, the second that Microsoft must improve their OS scheduler to the point where it actually understands NUMA principles not just for memory but also cpu cores. (I have no doubt at all that Linux and *BSD will have such a scheduler available well before the time your & I can buy a computer with such a cpu in it!)
So why do I believe that such cpus are inevitable?
Power efficiency!
A relatively simple in-order core like the one that Intel just announced as Atom delivers maybe an order of magnitude better performance/watt than a high-end Core 2 Duo. With 16 or 80 or 256 cores on a single chip, this will become really crucial.
Terje
PS As other posters have noted, keeping tomorrow's multi-core chips fed will require a lot of bandwith, this is neither free nor low-power.
I'm Norwegian, and I've watched most of these programs when they were originally broadcast:
The discussion page on http://nrkbeta.no/last-ned-lars-monsens-nordkalotten-365-gratis-og-i-full-kvalitet/ (Norwegian only) contains a lot of comments from the NRK people where they answer questions about all kinds of technical details (camera, sw, scaling/de-interlacing from 1080i to 576p, redoing the first episode to improve the quality etc.)
They also explain that the main/only reason they cannot do this with most of their productions is due to licensing issues, particularly for music for the soundtrack.
For this series they planned around this from the beginning, including commissioning a custom musical soundtrack.
It really sounds like they are trying to make BT distribution the default some time in the future, even though this will cut into the income they currently get from DVD sales.
Terje
BluRay disks needs to be higher capacity, since they (as I noted in my first post) also allow higher encoding bitrates: 40 vs 30 Mbit/s.
However, even with 1080p content, HD-DVD's 30 Mbit/s seems to be enough to encode even very dynamic content with no visual errors.
I have watched many times a number of 720p and 1080p test videos, of stuff like shuttle takeoffs and fighter aircraft exhibitions, encoded with various bitrates from 15 to 40 Mbit, and the difference (at least to my 50-year old eyes) is actually _very_ small, with the step from 30 to 40 seemingly giving no actual improvement.
The 40 Mbit cap might become important when even higher resolutions, like 1440p or 2160p become available at some point in the (far off?) future.
Terje
Graphics Processing Units (GPUs) are very good at doing repetitive fp and fixed-point operations, and much less good a bit-twiddling. I.e. the motion compensation stage of video decoding, where you copy (possibly sub-pixel-located) source blocks into the target frame has been handled by graphics hw since the very first sw DVD players, like Zoran's SoftDVD which was the first.
(In fact, SoftDVD was capable of handling 30 fps even without hw assists, running on a PentiumMMX 300 Mhz cpu, and without dropping any frames, but having the motion comp hw made it much easier to avoid drops. BTW, I did a very small bit of asm optimization work on that sw player.)
High bitrate HD/BR video is encoded using the CABAC (Content-Adaptive Binary Arithmetic Coding) algorithm, which provides slightly better compression rates, but which is also particularly unsuited to a GPU: Each decoding step requires multiple if/then/else stages, just to decode a single bit of information. It is also completely serial, in that you normally cannot determine the context to use for the next decoded bit until you've finished the current bit and possibly even branched on the result!
When you need to do this more than 50 million times/second, CABAC becomes the real bottleneck.
OK?
Terje
I have spent a couple of months optimizing code for HD decoding, and mostly the format doesn't really matter:
Both use the same codecs, they support the same resolutions, and the maximum bitrate is more or less the same (30 vs 40 Mbit/s for HD vs BR).
The one important difference is that a "full HD" 1080x1920 BR frame will always be encoded as four quadrants, each at 540x960.
This does lead to marginally lower compression rates, since you get more borders, but the great benefit is that you can have multiple CPU cores (up to 4) work in parallel on each of the parts!
You can of course do the same with a multi-core decoder for HD-DVD, but only by starting each cpu/thread at a different key frame, and since each 1080p picture requires 2 Mpixels, it is far too easy to trash both the TLB tables and the L2 caches when doing the motion compensation step which normally requires multiple source frames to be available to generate each target frame.
Terje
The Confluence Project http://confluence.org/ is an international effort to perform a systematic sampling of the Earth's surface, i.e. all those locations where both longitude and latitude has integer values.
So far more than 10,000 visitors have documented more than 5,000 of these points.
http://www.agu.org/pubs/crossref/2006/2006GL027768.shtml is a link to a paper by a Japanese researcher (Koki Iwao) and his associates: They have used the DCP information to check/verify the quality of the various land cover databases:
Which parts of the Earth is mountains/lakes/forests/rice fields/grassland/etc.?
What they found is that the best of these databases have a hit ratio of just 60% or less.
Terje
(Scandinavian DCP coordinator)
I have recently written what I believe is the world's fastest Ogg Vorbis decoder, it takes about 600 ms to decode my longest song sample (4:05 minutes encoded with 192 Kbit/s for a final filesize of 5.7 MB).
IMHO there are just a few problems with Vorbis, cpu load is not one of them:
a) It is not at all suitable for contineous streaming, with multiple receivers connecting/disconnecting on the fly, since you have to start by decoding the 4-8 KB header before you can make any sense of the sound frames.
b) To get decent decoding performance, you have to unpack & cache all the codebook information in the header packets, this requires from about 50 to 300 KB, which can be significant in a small device.
c) Even though Vorbis is in theory independent of the Ogg container format, most existing source code expects to find Ogg frames surrounding all Vorbis packets. This is an implementation and not a specification problem.
d) Vorbis really prefers to have fast fp support available, but Theora is an open-source fixed-point implementation which has been used as the starting point for quite low-resource embedded implementations.
Terje
I tried to reply by just writing out '.' and '-' characters for the Morse code of the previous post, but that triggered one of SlashDot's filters:
:-)
Lameness filter encountered. Post aborted!
Reason: Please use fewer 'junk' characters.
So I'll try this instead:
dah dah dih, dah dah dah, dah dah dah, dah dih dih
dih dah dah dah, dah dah dah, dah dih dih dih
73, de la8nw
License since 1978
I do kite sailing in the winter here in Norway, and the kite shown in the article is almost identical, except for size of course, with the kite I use. (I have also windsurfed since around 1990.)
My kite is a Peter Lynn Venom II http://www.peterlynnkiteboarding.com/, this is a twinskin kite which keeps its airfoil shape due to internal air pressure: A set of small mesh openings in the leading edge allows air into the opening between the front and back side.
This form of kite is an airfoil, not a spinnaker, the difference is huge:
A spinnaker is effectively a large bag to catch the wind, while a kite works best by having air moving faster on one side than the other. Among other things, this means that a kite allows you to sail much faster at an angle to the wind instead of straight downwind.
Another nice trick you can do with a kite, unlike a windsurfing rig, it to let the kite loop around in little figure-of-eights: This makes the airfoil move even faster through the air, increasing the lift particularly during a lull in the wind.
Terje
I do get significant amounts of spam, I'm probably left with about 30-100 per day that manage to pass through both the corporate filters and SeaMonkey's adaptive rules.
:-)
Getting rid of the rest takes me a minute or less, which I consider time well spent when the alternative would be to give in to the spammers and switch to a secret address.
I've gotten _many_ tips/requests/leads via email over the years, some of them led to very interesting projects.
I've even been paid for some of them!
Terje