Slashdot Mirror

It's probably all down to the user-befuddling issue of charsets (also known as character sets, character codes, encoding etc.). In this case it's probably the fault of Microsoft and understandably ignorant Microsoft users.

Microsoft (ab)use invalid (non-ASCII, non-ISO-8859, non-Unicode) codes to represent typographical sugar like prettier hyphens and backwards quotes in their own charsets. And if a system doesn't know what charset source material is in- or is told the wrong charset- then it can't correctly translate the material into some other charset.

In this case, it may have been a non-standard quote (92 hex in Windows-1252), replaced by a simple question mark (34 hex in ASCII and supersets) because it wasn't valid in ISO-8859-1 (Windows-1252 and Unicode are divergent supersets of ISO-8859-1). I doubt the author could even tell the next person what charset they used.

Hopefully divergent incompatible charsets (even the more modern ISO-8859-15) will die out as the world standardises on ASCII, ISO-8859-1 and Unicode according to their needs. I'd like to see Windows charsets die as well but it's another Microsoft lock-in method. I'd also like to see web sites etc. reject any submitted text containing non-ASCII codes (e.g. £) and insist that the user type charset-neutral entities instead (e.g. £). Or should I write those as £ and £ to make them look right?

Having said all this, I'm constantly learning new quirks of charsets. There's an excellent resource at http://czyborra.com/charsets/iso8859.html , complete with images of glyphs for various charsets.

Imagine you want to write out Jean-Baptist Moliere's name correctly--and in all caps, to boot. Now, that first e should carry a grave accent. So do you just find a keyboard with a capital e+grave on it? Let's say that your system interprets a keypress there to mean character number 0xC8. In the ISO 8859-1 (Latin1 for Western European languages) eight-bit encoding, this number is indeed a LATIN CAPITAL LETTER E WITH GRAVE.

So you might appear to be all taken care of. But you aren't. Tomorrow, you decide you'd like to write "correctly" the famous name of the inventor of robots, Karel Capek (aka Karla Capka). That C there should carry a caron, because it's not pronounced "Kapka", but "Chapka". So you go find yourself a Czech keyboard, and lo and behold, it has the proper character!

Are you set? Not at all; to the contrary, now you're I in trouble. Because you might well find that the character generated by that key, as recognized by your computer, is also number 0xC8. In the ISO 8859-2 (Latin2 for Eastern European languages) eight-bit encoding, that same 0xC8 is now taken to mean a LATIN CAPITAL LETTER C WITH CARON.

See the problem? If you look at Karel's name in your trusty Latin1 locale, it will be screwed up, and if you look at Jean-Baptist's under a Latin2 locale, then it will be screwed up. You can't win.

Now, as for the Euro symbol, you're going to have even more (none-)fun, because you aren't going to find a suitable ISO eight-bit encoding that includes it. The 8859's just aren't going to do it for you.

Of course, were this but in ISO 10646 (that is, in Unicode), these particular problems do go away. There, the LATIN CAPITAL LETTER E WITH GRAVE is at U+C8 (yes, really; the same as in Latin1), but the LATIN CAPITAL LETTER C WITH CARON is at U+10C, a completely distinct numeric code point. This is as it should be, since those really are different glyphs, so they shouldn't share the same numeric representation. On the matter of the Euro for your keyboard, under Unicode, you've even got EURO SIGN sitting there at U+20AC for you.

Even if you tried to go this route, I suspect that you're probably just exchanging one set of problems with another. After all, how well is your system truly set up for you to use Unicode? Can it map keyboard events into appropriate code points? And what about the tools you're using? What are you going to do with it once you have it? Consider the multiplicity of external encodings for the same code points, such as for disk storage, network transfers, etc, that you find in UTF-8, UTF16-LE, etc.

So, I don't think there are answers to the submitter's query that are at all so simple as others have presented the matter here. For the curious, here's a good reference on the mess we're in now, called appropriately enough, ISO Alphabet Soups.

--tom

Two points:

1. Being able to read the bits doesn't matter if you can't understand how they're laid out, because you still have to understand the file system (will systems still understand ISO 9660, with all of its permutations and extensions 50 years from now? Hard to say) and you still have to decode the bits once you find the right ones. Decoding isn't just a problem for new things like audio and video, it affects "flat text" files as well, or at least it has in the past. I'll grant that new encodings for latin text are unlikely; ISO 8859-1 and the Unicode transformation formats (which, hopefully, will settle out to a small number of encodings which will not change any more) both take care to be supersets of US ASCII, so ASCII text is probably safe -- assuming the documents you want to archive are in some European language.

2. You still haven't explained what forces are going to ensure that the new hardware always remains backward compatible with the old formats.

Two points:

1. Being able to read the bits doesn't matter if you can't understand how they're laid out, because you still have to understand the file system (will systems still understand ISO 9660, with all of its permutations and extensions 50 years from now? Hard to say) and you still have to decode the bits once you find the right ones. Decoding isn't just a problem for new things like audio and video, it affects "flat text" files as well, or at least it has in the past. I'll grant that new encodings for latin text are unlikely; ISO 8859-1 and the Unicode transformation formats (which, hopefully, will settle out to a small number of encodings which will not change any more) both take care to be supersets of US ASCII, so ASCII text is probably safe -- assuming the documents you want to archive are in some European language.

2. You still haven't explained what forces are going to ensure that the new hardware always remains backward compatible with the old formats.

Two points:

1. Being able to read the bits doesn't matter if you can't understand how they're laid out, because you still have to understand the file system (will systems still understand ISO 9660, with all of its permutations and extensions 50 years from now? Hard to say) and you still have to decode the bits once you find the right ones. Decoding isn't just a problem for new things like audio and video, it affects "flat text" files as well, or at least it has in the past. I'll grant that new encodings for latin text are unlikely; ISO 8859-1 and the Unicode transformation formats (which, hopefully, will settle out to a small number of encodings which will not change any more) both take care to be supersets of US ASCII, so ASCII text is probably safe -- assuming the documents you want to archive are in some European language.

2. You still haven't explained what forces are going to ensure that the new hardware always remains backward compatible with the old formats.

... that plain old ASCII is one constant that hasn't needed changing.

I think you're a little unclear as to what ASCII is. As the "A" in "ASCII" indicates, it's oriented towards American applications. And it consists of a mere 127 characters, which includes 32 control characters that you don't use in text.

In point of fact, Project Gutenberg has long outgrown the 96 graphic characters in ASCII, though I think they themselves are ignorant of the fact. The seem to have experimented with characters until they found a set that displays the same on "normal" Windows, Macs and Unix/Linux. The result is something they call "extended ASCII" but that's actually subset of both ISO's Latin1 character set and Microsoft's Latin1 code page.

When is this an issue? Well, I'm a DP volunteer, and I'm concentrating on the Britannica 11th edition. Lots of geographic entries, all of which contain degree symbols. This symbol is not in ASCII! If you follow the DP instructions, you end up entering byte 186 (decimal). If you're using the ISO or Microsoft Latin1 set (and if your computer is localized for the U.S., Canada, or Western Europe, you probably are) then 186 does in fact display as a degree symbol. But if your system is localized for Eastern Europe, you're probably using Latin2, and this byte stands for an S with a cedilla accent!

In short, "ASCII" is actually less universal than well-formed HTML. In which you represent the degree symbol with a character entity (°) that's the same everywhere.

Indeed, you can open up the original Declaration of Independence document with your standard web browser, and you can still read it just fine.

Hardly a representative example. The Declaration of Independence was hand-written, and thus doesn't include a lot of fancy fonts or formatting. A better example is a contemporary novel, such as 1984.

As it happens I just finished re-reading this one. I read a Plucker file that somebody had transformed from an HTML version, which in turn came from the Project Gutenberg "ASCII" version. Readable enough. But all the typographic nicities -- italics, boldface, etc. -- were reduced to ALL CAPS in the text version, and that was retained in the HTML version. Pretty distracting -- made me feel like somebody was shouting at me. Double Plus Ungood! Thoughtcrime!

...once the data is put into ASCII text format, projects like this [XML] can and are being done.

You make it sound easy. A lot of information is lost when your primary version is "ASCII". It all has to be put back by hand. There's no avoiding this for the large body of existing Gutenberg texts. And of course as recently as 5 years ago, there wasn't a real choice anyway. Even HTML had issues, and serious XML tools didn't exist.

But now XML technology is pretty mature. It makes sense to store new Gutenberg texts in XML. If people still want "ASCII" copies, the XML is easily transformed into that. Though I a lot more people will want the HTML version -- a format which is actually accessible to more people than "ASCII".

There are two reasons this won't happen soon.

The first is that somebody will have to design and implement the necessary XML apps for inputing and proofreading the texts. (Which would alsio elminate a lot of the errors proofreaders make, like entering [Greek: Tau] when they mean [Greek: T].) A huge project. As it stands, the people who maintain the DP web site have their work cut out just to keep the existing software working. That's a vali

The ae and oe to which you refer are indeed ASCII...

If you use Linux, try man ascii. That will show you all the characters in ascii. See Roman Czyborra's character set page for more detail than you could ever want. As a point, ae and oe are not in ASCII; oe isn't even in Latin-1.

Mathematics has it's own language which I doubt Unicode can rival.

Huh? What's that supposed to mean?


As far as culture, you can keep your native language... Just keep one language standard on the internet!

You can keep your language, as long as we don't see it, then? Use our language where we can see it. That's not arrogant? What if you were asked to learn Mandarian Chinese (the language of more people than any other) before you could use the Internet?

One of the amazing things about the Internet are when grandparents can communicate with grandchildren; forcing old dogs to learn new tricks and young kids to pick up a second language before they've fully learned their first isn't going to make the Internet more accessable to everyone.

I said:
St. Cyrill developed the Glagolic alphabet

RelliK said:
Uhhm, no. Glagolic is the alphabet that was used before Cyrill came along. It looks nothing at all like Cyrillic.

I am sorry, but you are wrong. The Glagolic did not look anything like the Cyrillic, but it was the alphabet created by St. Cyrill and his brother Methodius. The Glagolic looked somewhat like the ancient Armenian, Georgian, and Ethiopian alphabets, also developed by Byzanthine missionaries for the native languages in these areas.

The Cyrillic was created by St. Clement, a student of St. Cyrill. Thew alphabet was much simpler - for all the common sounds, it used the Greek letters, and only made up new letters for the sounds specific to the Slavs.

Here is a link, although not everything there is historically accurate (if there is such a thing):
http://www.volgawriter.com/VW Cyrillic.htm

Another, with a definitive set of Cyrillic encodings is here: http://czyborra.com/charsets/cyrillic.html.

I found out about Clement after more digging. For example according to this link, Clement took the glagolitic, invented by Cyrill and Methodius, rewrote it into cyrillic, and spread in Bulgaria. Later it spread further east and north.

Sounds like a browser issue. I was able to type it in fine, and Teoma did in fact come back with the ó printed correctly, however the results were garbage. My browser, Macintosh MSIE/5.0, submitted ó as %F3 which is its ISO-8859-1 codepoint (ISO-8859-1 is a superset of ASCII with 128 more characters, including most Western Euro accented vowels). How did yours submit it? If it represented the character as three %XX entities (dont know them off the top of my head), that means youre browser is using UTF-8, which is much less-supported.

Shuttle? Are you smoking something? My entire comment was two sentences, and made no mention of anything specific, let alone a shuttle. What I was saying is that you seemed to imply that since something is maliciously-planned it cannot be foreseen. I proceeded to say that you obviously dont (or shouldnt) be working in any security-related field, as most things one has to foresee are malicious in nature. And thats all I said.

And why dont you check the HTML source before implying that because Im typing something correctly I must be using Windows. First off, Windows does use typographically-correct quotes (so-called curlies) but uses the wrong codes: it uses Windows Code Page 1252 codes: 146, 147, 148, 149 (decimal). Codes from 128 to 159 are undefined in both ISO-8859 (an 8-bit superset including ASCII that nearly parallels CP1252) and ISO-10646 (Unicode, UTF-8, etc.), and thus dont work on any systems unless the page charset is specified as windows-1252 or the browser assumes what they mean (the Macintosh MSIE does this, naturally).

However: I type punctuation using the correct Unicode values which should work on any standards-compliant browser. These are U+2018, U+2019, U+201C, U+201D (hexadecimal). You can type codes like this by typing &#xNNNN; for hex codes, or just &#NNNN; for decimal. You can also use the ISO-8879 names for these characters lsquo, ldquo, rsquo, rdquo but a lot more browsers have problems with those.

"I don't think you can create a password that is easily memorizable that is 20 characters long," Kirovski said.

1. Since five die rolls can have 7776 possible combinations (6^5), each "word" has an entropy of just over 12.924 bits. (2^12.924 ~ 7776, so that many bits are necessary to represent each combination five die rolls can create).
   
2. Now, one "character", if we take it to mean an integer with values 0 through 255 inclusive, has entropy of 8 bits.
   
3. Therefore, every two diceware words correspond to three completely random bytes.
   

1. "56k modems are worse junk than what Napoleon had at Elba -- a bleating piece of lard is faster down an incline if you've given it a push, for chrissakes!!" together with the picture of a goat bleating in terror as it rides a chunk of lard down a hill. Also picture the goat in a Napoleon posture (one hoof inside vest) so you remember elba.[5]
   
2. "Wacko tries being a minister: comes up with wacky sermon about how we need to annex canada. I for one think it should be swept under the rug. (the idea advanced by the sermon or canada? :) )"
   Picture: arm stretching borders of alaska over canda.
   

• Note that the 7776 words diceware uses are all short. There are far more than that many common English words, but by including obscure shorter words and semi-words (like numbers), which are less common but equally memorable once you've thought about it / looked it up, the total typing is reduced. However, this leads to:
  
• Be very sure to accept any words you're given. If you need to look up a word to know what it means, do so. By avoiding words you don't know (rolling again), you reduce entropy.
  
• Don't change words. If I change "56" to "56k" above, and make that the word in my passphrase, it's not enough that I make sure 56k isn't already one on the list: I need to make sure that none of the other 7776 words are ones I might change to 56k if I roll them. In other words, just don't change words.
  

...hex 0xA4 or 164 decimal. And it's not really ASCII but it's part of the ISO 8859-15 character set. That's also what you've got to tell all your X11 applications. And you must have ISO 8859-15 aware fonts installed, which is the case with recent Linux distribution (e.g. SuSE 7.3 is fully Euro enabled).

In the web site outlining the patent the references section lists: "iDNS--Internationalized Domain Name System, by Center for Internet Research(http://www.apng.org/ids/, Jan. 1998.". Actually the web address is incorrect and should be http://www.apng.org/idns/ so I wonder if that is just a typo on the web page OR in the actual patent. If it's in the actual patent then ... doesn't the patent office at least check the references ?.

Anyway can someone explain to me how this referenced web site doesn't constitute prior art ?

I only quickly read the patent and the above reference but they seem to be presenting the same idea.

i.e. The http://www.apng.org/idns/ site suggests UTF-5 whereas the WALID patent suggests Unicode. See this site for a description of the two.

Well I just can't see how taking an existing idea which is clearly still in draft and changing one small aspect of it (ie. the encoding UTF-8 instead of UTF-5) can constitute a new idea. What the hell is the US Patent system doing to free trade if stuff like this is going on ?

Actually I've just been re-reading about Unicode aka UTF-8 & Co and it seems that UTF-5 is actually oneof the unicode formats. In this case how is specifying a patent that specifies "Unicode" all that different from specifying the same idea with one of the particular implementations of Unicode (ie. UTF-5) ?

Anyway given the large number of comments already and the fact that I only get posted at Score:1 I don't expect too many people to read this but I'd be interested in a comment from someone a bit more clued in to this than I am.

Thanks.

In the web site outlining the patent the references section lists: "iDNS--Internationalized Domain Name System, by Center for Internet Research(http://www.apng.org/ids/, Jan. 1998.". Actually the web address is incorrect and should be http://www.apng.org/idns/ so I wonder if that is just a typo on the web page OR in the actual patent. If it's in the actual patent then ... doesn't the patent office at least check the references ?.

Anyway can someone explain to me how this referenced web site doesn't constitute prior art ?

I only quickly read the patent and the above reference but they seem to be presenting the same idea.

i.e. The http://www.apng.org/idns/ site suggests UTF-5 whereas the WALID patent suggests Unicode. See this site for a description of the two.

Well I just can't see how taking an existing idea which is clearly still in draft and changing one small aspect of it (ie. the encoding UTF-8 instead of UTF-5) can constitute a new idea. What the hell is the US Patent system doing to free trade if stuff like this is going on ?

Actually I've just been re-reading about Unicode aka UTF-8 & Co and it seems that UTF-5 is actually oneof the unicode formats. In this case how is specifying a patent that specifies "Unicode" all that different from specifying the same idea with one of the particular implementations of Unicode (ie. UTF-5) ?

Anyway given the large number of comments already and the fact that I only get posted at Score:1 I don't expect too many people to read this but I'd be interested in a comment from someone a bit more clued in to this than I am.

Thanks.

The only languages that can comfortably be written with the repertoire of US-ASCII happen to be Latin, Swahili, Hawaiian and American English without most typographic frills. It is rumoured that there are more languages in the world.

(Roman Czyborra on his page about ASCII)

I love that quote!

The only languages that can comfortably be written with the repertoire of US-ASCII happen to be Latin, Swahili, Hawaiian and American English without most typographic frills. It is rumoured that there are more languages in the world.

(Roman Czyborra on his page about ASCII)

I love that quote!

Actually, I'm pretty much amazed. I expected all hell to break lose. In fact, not only my browser correctly displayed the Chinese characters in the post, but when I clicked on the link it opened a nice little window saying "www..com could not be found. Please check the name and try again." With the Chinese characters correctly displayed, even in the error message!

And it passes the Unicode test better than I thought it did: in fact, only the Sanskrit bit was incorrect (and only slightly so: the ligatures weren't made). This is because I have the GNU Unifont installed, but Mozilla definitely rocks.