I'd think it would be far more efficiant to use arrays of the largest (or possiblly the second biggest if your language or runtime environment doesn't let you get at the carry flag and/or lacks unsigned integer support) of your available integer types.
spaces have the issue that they may cause misintepretation as two seperate numbers, especially if full sized spaces are used. And i'm not convinced fonts are widespread enough for using special thin spaces for it to be a good idea to do so on the web.
Another approach is never to use three digits after the decimal seperator, that way it is obvious that it is not a thousand seperator.
It's a very easy thing to screw up, especially with development tool vendors doing stupid things. For example in delphi floattostr is locale dependent but this is far from made clear in the docs....
Now, that is a bloody travesty! It turns a neat, easy to read (scalable fonts!) document with nicely searchable contents into a bitmap... It ought to be possible to properly black-out information, that does not want to be free, without turning the rest of the text into a picture. It is possible, the question is what is the risk of someone screwing the procedure up and is that risk of screwup worth the improved quality of the copy.
With modern modulation methods and the better performance of vinyl compared to cassette it should be possible to get quite a bit more data per second than the old microcomputers stored on cassettes.
Look at the development of communication over copper phone lines. In the bad old days you got 300/300 or 1200/75. With modern dialup technology you get just about 50K down and 30K up limited mainly by the digitisation at the phone exchange. With DSL we get speeds in the megabits per second.
I have watched loads of top gear and while they do sometimes do some car smashing (or bus smashing or campervan smashing or....) they also seem to be adult enough to understand what they are and aren't allowed to smash up. Expensive cars received on loan definately fall into the not allowed to be smashed up category.
Do you have a source for that figure on the cost of keeping a book in a library? It seems rather on the high side to me and i'd like to know what the assumptions are.
1: in GTA you have auto-targetting 2: in GTA you have a radar showing the cops and (in the case of GTA4) you just have to get out of their search area for a few seconds to be safe. 3: in GTA all the negotiations happen in cutscenes 4: in GTA getting busted and "wasted" (the closet thing the game has to getting killed) are relatively minor things, you just reload and/or take the relatively minor hit. 5: the enemies in GTA are generally far less effective combatants than you are, going alone against huge hordes of gangsters like GTA forces you to in the real world would be suicide.
IIRC the Xbox enforces DVD region codes. Afaict every console that can play DVDs enforces DVD region codes for them. This does not imply that they use the DVD region coding system for games.
I've just done some googling and it seems the xbox and xbox 360 use the same PAL/NTSC/NTSC-J region system that most other consoles use.
BTW if your debit card is a visa one then afaict you can use it anywhere that asks for a visa credit card. The foreign transaction fees can be annoying though.
for console gamers you still have to contend with the region locks Afaict the PS2, gamecube and wii only have one PAL region so you can import from any PAL country (e.g. the uk).
The PS3 seems to have australia in a seperate region BUT afaict most if not all PS3 games aren't actually region locked.
Most media currently rely on pretty well documented formats (JPEG, MPEG, MP3, etc.). Nonetheless there are formats, both on the consumer end (MS-Word) and on the professional end (RAW pictures format) which are poorly documented or kept secret. And as such nothing guarantees that 20years down the line these will still be opennable. IMO the best approach here is to archive both the original and one or more conversions of it to formats that are likely to be more stable. A good document archival system therefore needs to be able to store multiple versions of the same document and to store metadata such as which one is the original and what conversion methods were used to produce each.
For the word 97 document I would probablly suggest creating both a pdf (preserves exact formatting and widely supported but not really editable) and an odf (editable but less likely to preserve exact formatting and not so widely supported).
For the raw image I would suggest processing the image into one or more tiffs or pngs (supported everywhere but some information can be lost in the conversion from raw to rgb) and possiblly also converting it to dng (which afaict will preserve all the raw data but is less widely supported).
Except this is no longer true in a full-duplex world, you can approach 99% utilization on Ethernet at full-duplex. At the time token-ring was competitive, full-duplex Ethernet was just emerging. While IBM's marketing and some of the complexity of token-ring hurt it, what really killed it was the widespread emergence of full-duplex ethernet switches which basically eliminated the under-utilization problem while not having the complexity of dealing with a token-ring network. Agreed, lots of people seem to think that ethernet implies CSMA/CD and make claims about modern ethernet networks based on CSMA/CD ethernet networks.
One problem is by default many network devices/OSes do bandwidth distribution on a per _connection_ basis not on a per IP basis. Standard IP networks do bandwith distribution on the basis of the clients backing down (or if nobody backs down on the basis of who can get packets in when the queue isn't full). If the systems all use equally agressive implementations of TCP then each TCP connection will get a roughly equal bandwidth. OTOH a custom UDP based protocol can easilly take far more or far less than it's fair share depending on how agressive it is.
A much fairer way would be to share bandwidth amongst users on a per IP basis. That means if two users are active they get about 50% each, even if one user has 100 P2P connections and the other user has only one measly http connection.
It's a nice idea but there are a couple of issues 1: it takes more computing resources to implement a source based priority queue than to implement a simple fifo queue. 2: to be of any use such prioritisation needs to happen at the pinch point (that is at the place there is actually a queue built up), unless you deliberately pinch off bandwidth or you overbuild large parts of your network predicting the pinch point location may not be easy.
Unless my thinking is completely screwed up, It is, lots of people get caught out by very big numbers.
I'd think the chance of finding a collision across say 5-6 completely different hash algorithms would be quite slim. In mathematics proving that something exists and actually finding an example of it are not the same thing.
one could easily store, say, a blu-ray movie by calculating and storing a few different hashes (MD5, SHA1, SHA2, etc.) lets consider you have 16 hash algorithms each producing a 32 byte hash. When combined the hashes give 512 bytes of output so there are 8^512 possible outputs.
Lets consider that we know the file is exactly 20GB since you communicated that information seperately.
It's difficult to put an exact figure on how many valid 20GB HD videos there are but if we assume that less than half the information in the file is fixed overhead (a resonable assumption IMO) then there are 8^10000000000 of them.
So for every output you have an average of 8^9999999744 possible inputs!
You may have to do one conversion to get the files into a widely supported format with a portable opensource implementation at intial import time but after that there should be little need for further conversion of the archive.
In general if a file format has been widely supported for over a decade and has an opensource implementation I don't see it dissapearing any time soon. Indeed with the software patent mess older formats will have the advantage that any valid patents pertaining to the original methods of encoding and decoding them will have expired.
I notice that there are tools out there that claim to be able to recover corrupt jpegs and some of them even have free trials, may be worth giving them a go.
The PNG image format divides the image data into "chunks", typically 8kbytes each Do you have a source for this claim, I was under the impression that unless doing streaming generation it was more normal to use just one big chunk for all the image data.
Even if you didn't want to do that, ISO 9660, the filesystem used by default on data CDs, contains its own error correction scheme (288 bytes of redundancy for every 2048 byte block). I thought that scheme was below the fileystem layer.
but anyway the problem with schemes like that (there are similar ones built into hard disks) is that they work on very small blocks and all the data from an error correction block is stored close together. This means that it's easy to lose a whole block at once to scratches etc.
Actually all you need to do in a delta-compressed case is have some sort of full-frame resync at reasonable intervals (say 1 second of video) - this gives nearly the full benefits of the compression and means that no glitch can cascade more than 1 second past the end of the glitch. Mpeg does this, you have iframes (complete frames), pframes (frames based on the previous iframe or pframe) and bframes (frames based on both the previous iframe or pframe and the next one) . The frequency of each frame type is up to the encoder.
Digital TV puts iframes fairly often to reduce the impact of corruption and to make channel switch times tolerable but still still losing a second or so is a lot more annoying than losing a single frame.
It also has the effect that relatively static areas of the screen start off as very poor quality when an iframe comes in and then gradually improve in quality before returning to bad again on the next iframe. Some may find this affect irritating.
You're right about CD's always having contained this type of ECC mechanism. For that matter you will see this type of ECC in Radio based communications infrastructure and data that gets written to Hard disks too! In other words - all modern data storage devices (except maybe Flash..) contain ECC mechanisms that allow burst error detection and correction. There are a few issues
1: the mechanisms aren't end to end. The data is protected by one system when on disk, another when in memory (and that is assuming you bought ECC ram, builders of lower end machines generally don't bother) another when on the network and I don't think it's protected at all when flowing accross the computers busses. 2: the mechanisms are often pretty weak and the user generally has no control over thier strength. 3: the machanisms often work on pretty small blocks, if something obliterates a whole block they probablly won't help you.
An end to end system where the ammount of redundancy is controlled by the archivist not by a penny pinching hardware manufacturer (who will be trying to put in just enough ecc that thier results don't come out as appalling) sounds far more attractive.
Though i'm not sure complex ecc algorithms are the best soloution, I think a better way is probablly to store copies of the file along with strong checksums of blocks of the file in multiple seperate locations. When a checksum check fails the block can be retrived from other copies.
The good thing about this system is that you get corruption protection for very little extra cost over the redundancy you need for disaster protection anyway.
In the case of an image, it could shift the whole thing one bit to the left which results in a distorted but still-viewable photo. Unless it's a one bit black and white image then shifting one bit will have massive effects as what was the least significant bit of one value becomes the most significant bit of another. The same issue applies with audio.
Slashdot Mirror
I'd think it would be far more efficiant to use arrays of the largest (or possiblly the second biggest if your language or runtime environment doesn't let you get at the carry flag and/or lacks unsigned integer support) of your available integer types.
spaces have the issue that they may cause misintepretation as two seperate numbers, especially if full sized spaces are used. And i'm not convinced fonts are widespread enough for using special thin spaces for it to be a good idea to do so on the web.
Another approach is never to use three digits after the decimal seperator, that way it is obvious that it is not a thousand seperator.
It's a very easy thing to screw up, especially with development tool vendors doing stupid things. For example in delphi floattostr is locale dependent but this is far from made clear in the docs....
and pulling it out of a pdf that is "secured" but with a blank user password is only slightly less trivial.
pdf user restictions only prevent the most casual of extraction attempts.
Now, that is a bloody travesty! It turns a neat, easy to read (scalable fonts!) document with nicely searchable contents into a bitmap... It ought to be possible to properly black-out information, that does not want to be free, without turning the rest of the text into a picture.
It is possible, the question is what is the risk of someone screwing the procedure up and is that risk of screwup worth the improved quality of the copy.
With modern modulation methods and the better performance of vinyl compared to cassette it should be possible to get quite a bit more data per second than the old microcomputers stored on cassettes.
Look at the development of communication over copper phone lines. In the bad old days you got 300/300 or 1200/75. With modern dialup technology you get just about 50K down and 30K up limited mainly by the digitisation at the phone exchange. With DSL we get speeds in the megabits per second.
I have watched loads of top gear and while they do sometimes do some car smashing (or bus smashing or campervan smashing or....) they also seem to be adult enough to understand what they are and aren't allowed to smash up. Expensive cars received on loan definately fall into the not allowed to be smashed up category.
Do you have a source for that figure on the cost of keeping a book in a library? It seems rather on the high side to me and i'd like to know what the assumptions are.
My guess is that an increasing number of journals will move from the pay to read model to the pay to publish (open access) model.
Of course this model has it's problems too but at least it allows some level of price competition between journals.
I very much doubt it for a few reasons
1: in GTA you have auto-targetting
2: in GTA you have a radar showing the cops and (in the case of GTA4) you just have to get out of their search area for a few seconds to be safe.
3: in GTA all the negotiations happen in cutscenes
4: in GTA getting busted and "wasted" (the closet thing the game has to getting killed) are relatively minor things, you just reload and/or take the relatively minor hit.
5: the enemies in GTA are generally far less effective combatants than you are, going alone against huge hordes of gangsters like GTA forces you to in the real world would be suicide.
IIRC the Xbox enforces DVD region codes.
Afaict every console that can play DVDs enforces DVD region codes for them. This does not imply that they use the DVD region coding system for games.
I've just done some googling and it seems the xbox and xbox 360 use the same PAL/NTSC/NTSC-J region system that most other consoles use.
BTW if your debit card is a visa one then afaict you can use it anywhere that asks for a visa credit card. The foreign transaction fees can be annoying though.
for console gamers you still have to contend with the region locks
Afaict the PS2, gamecube and wii only have one PAL region so you can import from any PAL country (e.g. the uk).
The PS3 seems to have australia in a seperate region BUT afaict most if not all PS3 games aren't actually region locked.
I dunno what the situation is with MS consoles.
Most media currently rely on pretty well documented formats (JPEG, MPEG, MP3, etc.). Nonetheless there are formats, both on the consumer end (MS-Word) and on the professional end (RAW pictures format) which are poorly documented or kept secret. And as such nothing guarantees that 20years down the line these will still be opennable.
IMO the best approach here is to archive both the original and one or more conversions of it to formats that are likely to be more stable. A good document archival system therefore needs to be able to store multiple versions of the same document and to store metadata such as which one is the original and what conversion methods were used to produce each.
For the word 97 document I would probablly suggest creating both a pdf (preserves exact formatting and widely supported but not really editable) and an odf (editable but less likely to preserve exact formatting and not so widely supported).
For the raw image I would suggest processing the image into one or more tiffs or pngs (supported everywhere but some information can be lost in the conversion from raw to rgb) and possiblly also converting it to dng (which afaict will preserve all the raw data but is less widely supported).
Except this is no longer true in a full-duplex world, you can approach 99% utilization on Ethernet at full-duplex. At the time token-ring was competitive, full-duplex Ethernet was just emerging. While IBM's marketing and some of the complexity of token-ring hurt it, what really killed it was the widespread emergence of full-duplex ethernet switches which basically eliminated the under-utilization problem while not having the complexity of dealing with a token-ring network.
Agreed, lots of people seem to think that ethernet implies CSMA/CD and make claims about modern ethernet networks based on CSMA/CD ethernet networks.
One problem is by default many network devices/OSes do bandwidth distribution on a per _connection_ basis not on a per IP basis.
Standard IP networks do bandwith distribution on the basis of the clients backing down (or if nobody backs down on the basis of who can get packets in when the queue isn't full). If the systems all use equally agressive implementations of TCP then each TCP connection will get a roughly equal bandwidth. OTOH a custom UDP based protocol can easilly take far more or far less than it's fair share depending on how agressive it is.
A much fairer way would be to share bandwidth amongst users on a per IP basis. That means if two users are active they get about 50% each, even if one user has 100 P2P connections and the other user has only one measly http connection.
It's a nice idea but there are a couple of issues
1: it takes more computing resources to implement a source based priority queue than to implement a simple fifo queue.
2: to be of any use such prioritisation needs to happen at the pinch point (that is at the place there is actually a queue built up), unless you deliberately pinch off bandwidth or you overbuild large parts of your network predicting the pinch point location may not be easy.
And Modern Warfare 2 hasn't caused a skyrocket in enlistment ;).
Indeed, I wonder if the more realistic modern warfare 3 will change that
Unless my thinking is completely screwed up,
It is, lots of people get caught out by very big numbers.
I'd think the chance of finding a collision across say 5-6 completely different hash algorithms would be quite slim.
In mathematics proving that something exists and actually finding an example of it are not the same thing.
one could easily store, say, a blu-ray movie by calculating and storing a few different hashes (MD5, SHA1, SHA2, etc.)
lets consider you have 16 hash algorithms each producing a 32 byte hash. When combined the hashes give 512 bytes of output so there are 8^512 possible outputs.
Lets consider that we know the file is exactly 20GB since you communicated that information seperately.
It's difficult to put an exact figure on how many valid 20GB HD videos there are but if we assume that less than half the information in the file is fixed overhead (a resonable assumption IMO) then there are 8^10000000000 of them.
So for every output you have an average of 8^9999999744 possible inputs!
You may have to do one conversion to get the files into a widely supported format with a portable opensource implementation at intial import time but after that there should be little need for further conversion of the archive.
In general if a file format has been widely supported for over a decade and has an opensource implementation I don't see it dissapearing any time soon. Indeed with the software patent mess older formats will have the advantage that any valid patents pertaining to the original methods of encoding and decoding them will have expired.
I notice that there are tools out there that claim to be able to recover corrupt jpegs and some of them even have free trials, may be worth giving them a go.
The PNG image format divides the image data into "chunks", typically 8kbytes each
Do you have a source for this claim, I was under the impression that unless doing streaming generation it was more normal to use just one big chunk for all the image data.
Even if you didn't want to do that, ISO 9660, the filesystem used by default on data CDs, contains its own error correction scheme (288 bytes of redundancy for every 2048 byte block).
I thought that scheme was below the fileystem layer.
but anyway the problem with schemes like that (there are similar ones built into hard disks) is that they work on very small blocks and all the data from an error correction block is stored close together. This means that it's easy to lose a whole block at once to scratches etc.
Actually all you need to do in a delta-compressed case is have some sort of full-frame resync at reasonable intervals (say 1 second of video) - this gives nearly the full benefits of the compression and means that no glitch can cascade more than 1 second past the end of the glitch.
Mpeg does this, you have iframes (complete frames), pframes (frames based on the previous iframe or pframe) and bframes (frames based on both the previous iframe or pframe and the next one) . The frequency of each frame type is up to the encoder.
Digital TV puts iframes fairly often to reduce the impact of corruption and to make channel switch times tolerable but still still losing a second or so is a lot more annoying than losing a single frame.
It also has the effect that relatively static areas of the screen start off as very poor quality when an iframe comes in and then gradually improve in quality before returning to bad again on the next iframe. Some may find this affect irritating.
You're right about CD's always having contained this type of ECC mechanism. For that matter you will see this type of ECC in Radio based communications infrastructure and data that gets written to Hard disks too! In other words - all modern data storage devices (except maybe Flash..) contain ECC mechanisms that allow burst error detection and correction.
There are a few issues
1: the mechanisms aren't end to end. The data is protected by one system when on disk, another when in memory (and that is assuming you bought ECC ram, builders of lower end machines generally don't bother) another when on the network and I don't think it's protected at all when flowing accross the computers busses.
2: the mechanisms are often pretty weak and the user generally has no control over thier strength.
3: the machanisms often work on pretty small blocks, if something obliterates a whole block they probablly won't help you.
An end to end system where the ammount of redundancy is controlled by the archivist not by a penny pinching hardware manufacturer (who will be trying to put in just enough ecc that thier results don't come out as appalling) sounds far more attractive.
Though i'm not sure complex ecc algorithms are the best soloution, I think a better way is probablly to store copies of the file along with strong checksums of blocks of the file in multiple seperate locations. When a checksum check fails the block can be retrived from other copies.
The good thing about this system is that you get corruption protection for very little extra cost over the redundancy you need for disaster protection anyway.
In the case of an image, it could shift the whole thing one bit to the left which results in a distorted but still-viewable photo.
Unless it's a one bit black and white image then shifting one bit will have massive effects as what was the least significant bit of one value becomes the most significant bit of another. The same issue applies with audio.