Slashdot Mirror
Unicode Encoding Flaw Widespread
LordNikon writes "According to this CERT advisory: 'Full-width and half-width encoding is a technique for encoding Unicode characters. Various HTTP content scanning systems fail to properly scan full-width/half-width Unicode encoded HTTP traffic. By sending specially-crafted HTTP traffic to a vulnerable content scanning system, an attacker may be able to bypass that content scanning system.' A proof of concept affecting IIS is already being posted to security mailing lists. Cisco IPS and other IDS products are also affected." The CERT advisory lists 93 systems, with 6 reported as vulnerable (including 3com, Cisco, and Snort), 5 known not vulnerable (including Apple and HP), and the rest unknown.
This appears to be limited to content scanning, and isn't really a vulnerability in itself. Relying on content scanning to prevent an exploit to reach an exploitable system is a pretty bad idea, much better to fix the system than the extra layer of defense on the outside.
Content scanning is mostly useful against filtering known exploits, and is hardly meant to be your primary defense. Being able to bypass this scanning won't buy you much. If the content scanner is aware of an exploit it scans for, chances are so are the systems being targeted and are patched to protect against it.
I.O.U One Sig.
Oh, you said public.. hehe, forget I said anything.
How we know is more important than what we know.
I work incident response in a large web company (hence anonymous posting, natch) and currently we're treating this as "interesting, but case not proven". We test our web apps filter all input so I'm adding double-width unicode to our security regression test cases; however I'm happy to let the FD posters lab it out between them in the short term. These alleged IIS exploits don't work for us - which is not to say that we don't have some system, somewhere, for which this is an issue. At the end of the day it's just a clear restatement of something that's obvious to anyone - you need to filter input carefully, and you need to be aware of issues around alternative encodings. But it's not a "BRB" (big-red-button, ie emergency stop and all hands to the pumps to fix a vulnerability) issue for us - yet. The last time we had one of those, it was the Microsoft DNS server remote root... because most of our internal domain controllers were also running DNS servers.
According to the advisory, Apple products do not provide HTTP content filtering and are therefore not vulnerable. This will do nothing to help someone build a functioning protection system.
I'm wondering if the great firewalls (Cisco product?) are also vulnerable to this. At least it'll force them to do longer string matching.
To think that even English fits in 7-bit ASCII is naïve.
Would some of the things that led to computers - morse code, telegraphy etc have been feasible using, say, Chinese in its normal written form? Are computers biased towards English (and other languages using the same or similar alphabets) because they were largely invented by English speakers, or is the language fundamentally more amenable to small, simple encoding?
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
You'd prefer securing against vulnerabilities in dozens, if not hundreds of different encodings? The only people who are against Unicode are those that have never had to work with more than one written language in the same project. Yes, it's a lot easier to secure stuff when you only accept ASCII or ISO8859-1/Windows CP-1252, but then you're limiting your software to about a third of the world (if that). Crappy engineers are going to write crappy code no matter what the encoding. No sense compromising for the sake of poorly written software.
Ooh, poor old BSD sounds really sick there. I hope that it doesn't die!
To think that English doesn't fit in 7-bit ASCII is na\"ive.
Wrong, the flaw in Cisco's "security" software and IIS is due to them converting things to 8-bit charsets, not due to Unicode. In fact, the whole idea of "code pages" is fundamentally broken, as it assumes all data ever moves to another places only in the same region.
The idea of double-width characters is broken too, yeah, and they are there only to appease the users of some broken Chinese/Japanese software -- but there's nothing wrong with having strange characters in file names. They don't match any file they are not supposed to unless you try to shoehorn them into a limited character set.
So, it's a flaw in the software, not Unicode by itself.
The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
If you want to represent a language on a computer (and not just numbers) then you need a way to enter and store all the characters that language uses. Obviously the less characters the better. The latin alphabet with all its variations, the cyrillic , hebrew, arabic & korean all lend themselves to this quite easily since they all have a manageable number of letters. Languages such as Chinese and Japanese don't , they don't even use alpabets , they use characters for each object/concept which as you can imagine is a bugger when you need a keyboard of a manageable size not to mention the memory to store each character bitmap (not an issue now , but 40 years ago it would have been a nightmare).
This is only a guess on my part but I suspect computers would have developed completely differently if they'd been developed by a culture that used symbols rather than alphabets.
Down below this post, there's a troll writing something like 'lol if u cant just use ASCII u shud let ur language die u foreign creeps lol k thx'.
And a whole bunch of people then jump on the troll and criticize him for his US-centrism, and so on, and the troll is at -1.
Yet the post I'm replying to, which is at +4, really comes to the same thing as this troll; it's simply UNIX 8-bit centric rather than USA ASCII centric.
The fact is, computers are used for text, and much if not most text is non-ASCII. How would you rather represent that text:
--With Unicode
--With KOI-8, KOI-8R, KOI-8RU, EBCDIC, EUC-KR, EUC-JP, shift-JIS, Shift-JIS-the-Jphone-version, ISCII, VISCII, ISO-2022-*, and the many many other encodings that have evolved in different times and environments.
Seriously, which is going to be easier to secure (and otherwise manage) -- one encoding (which is HEAVILY documented and discussed) or a large number of encodings (the actual number being ever-changing and impossible to really know) many of which are not well documented and have forgotten ramifications and assumptions?
Right -- so now you know why people use Unicode so much.
But the interesting question is, why is one error ("All teh world is teh USA lol! Shouldn't you learn to speak English?") rightly jumped on and pounded flat, whereas another form that's actually more problematic ("All teh world is C on UNIX lolz!! Shouldn't you stop wanting dangerous extra features?") isn't?
Actually, I see in another window that some people have indeed been pounding the parent poster flat, so perhaps my question isn't valid after all.
Whence? Hence. Whither? Thither.
4) You are an idiot
5) You are an asshole
It is a vulnerability, in the strict sense.
...
It is a self-inflicted misbehaviour as in common sense.
It is like those silly Cisco content inspectors on port 25, that try to avoid attacks on flimsy MTAs.
It is like someone dying from a jab against measles: the jab protected that person from contracting measles, actually.
It is like those stupid anti-virus programs that are more vulnerable than the daemons they profess to protect.
When the attacker uses a codepage different from the one that you think she ought to use, she can circumvent your content filter. Which ought not be an attack vector, in any case.
As I said: nothing to see, move along
The notable difference between Chinese and English (or most other written languages) is that several English characters combine to form syllables, which combine to form words (i.e., we use an alphabet). In Chinese, each character corresponds directly with a word (each character is a logogram). If you're interested you can look up Alphabet on Wikipedia as a starting point, although I must admit I find the article hard to follow even though I know what it should be saying.
The practical result of this is that English is normally encoded as a long sequence of 0-25 values (a-z), whereas Chinese would be encoded as a shorter sequence of 0-~100,000 values (Wikipedia reports Chinese dictionaries with 85,000 characters). Naturally, there would be fewer Chinese characters required for a message as each character corresponds to an entire word.
I guess that since morse code is rather like binary and English letters can be encoded using 5 bits, Chinese morse codes would need to be... about 20 bits long? It's late at night, brain not work so good. It seems to me that morse codes using 20 dots/dashes would be extremely difficult to learn; but on the other hand it shouldn't be any more difficult than learning Chinese characters in the first place.
I wouldn't be surprised if English morse codes were more robust against poor data, siny Englxsh is stvll reahible even if sew2eral cheracter; are wrong.
Disclaimer: I don't know anything about the subject, I'm talking out of my elbow for the sake of discussion.
.evom ton seod gis eht
1) unicode is better than having a hundred other encodes to debug
2)there's is nearly two billion chinese and Indians, who can't use your encoding.
3)I get just as much spam from US companies as I do foreign ones
i thought once I was found, but it was only a dream.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Unicode is of course not the problem at all.
The problem is using character sets that can represent huge amounts of different characters, and among them characters that have similar looking glyphs. That is at the same time a feature that people really really want.
So spam filters will have a problem. They filter out "Viagra" but they don't filter out sequences of letters that look the same. Well, tough. If you follow the rule not to follow any links in emails but type them in yourself, that gets you mostly around it.
The other "problem" is filtering to prevent SQL injection and all that crap. There I'd have to say two things: 1. It is just common sense if you accept Unicode to translate it into a canonical form first, either precomposed canonical or predecomposed canonical (by the way, predecomposed canonical UTF8 is what the MacOS X file system uses). Once that is done, nothing unexpected should slip through. 2. Why would you need to filter out anything at all? This is a completely brain-damaged approach in the first place, using user input to form commands that could potentially be dangerous and filtering out user input that would produce dangerous commands. Instead, there shouldn't be any commands that could be dangerous in the first place.
The answer would seem to be -- sort of ... maybe. See http://www.njstar.com/tools/telecode/jim-reeds-ctc .htm.
Summary: For telegraphy, Chinese characters are assigned numeiic codes in radical-stroke count order. That's the way that Japanese, and -- I assume -- Chinese, dictionaries, are arranged.
It may seem inefficient to use 20 bits (sort of) to encode a character, but remember that each character is a word, not a letter, and that composite words like "Beijing" or "paleontology" are only two words. That means that most "words" will be either 2.5 or 5 eight "bit" characters. Conventional telegraphy is really a trinary rather than a binary code -- pause, short, and long, and the 'digits' differ in length -- so bit count isn't really all that accurate an analogy.
So, no, the Chinese language probably wouldn't have made the development of computers by the Chinese all that much more difficult than European languages did. And the classic Chinese numeric notation is not as convenient as 'arabic' notation. But it's much less unwieldy than say Roman numerals, so I don't think it would have been an insumountable hurdle either.
You can't see ANYTHING from a car, You've got to get out of the goddamned contraption and walk...Edward Abbey
Apparently, Vista's networking stack has been rewritten from scratch -- which does make you wonder how much of the reason for that was technical, and how much was MS wanting to be seen to get rid of all the BSD/*nix code in Windows in preparation for their patent offensive...
Why should using BSD code come in the way of their patent offensive?
Using BSD code isn't infringing on BSD's or someone else's patent.
They are there for compatability with some Japanese and Chinese character sets, which contained most of the ascii characters in both "half" and "full width" forms. The full-width ones were twice as wide to match the square characters, which was useful for lining up columns.
This is all pointless now with proportionally-spaced fonts (and multiple fonts, you could easily select the "wide" font to print those characters instead). However Unicode had as a design requirement that translating from any common encoding to unicode and back again would be the identity transform. Thus if any character set existed with two ways of representing the same character, then there had to be two ways to represent it in Unicode. Therefore the full-width characters. This is also why Unicode has hundreds of random accented characters even though the combining characters would allow all of them to be represented easily with only a few dozen characters.
The notable difference between Chinese and English (or most other written languages) is that several English characters combine to form syllables, which combine to form words (i.e., we use an alphabet). In Chinese, each character corresponds directly with a word (each character is a logogram).
/. filters, consider the word zi4ran2. The zi4 syllable is a word, and means "from" or "since" (and is also used like "-ly" to form adverbs). The ran2 syllable is also a word, and basically means "correct" or "yes". The zi4ran2 combination means "nature" or "naturally". Like "insight", you might be able to kludge some sort of connection here, but in reality you just have to learn zi4ran2 as a separate word unrelated to its two syllables. It may have been a two-word idiom several thousand years ago; it's a two-syllable word now.
;-)
Actually, this is pretty much a myth that originated from people with very little knowledge of Chinese language and writing. In all the Chinese languages ("dialects";-), most of the vocabulary is two-syllable words, as in English. Three-syllable words aren't uncommon. The writing system is actually a sort of syllabary, and the meaning of most two-character words can't be inferred from knowing what the syllables mean as standalone words.
It's similar to how lots of English words, e.g. "insight", can be parsed as two words ("in"+"sight"), but this doesn't really help you understand what the word actually means. Or, an example that shows how such things evolve is the English word "upstairs". If I say I'm going upstairs and take the elevator, did I lie to you? Of course not, because "upstairs" doesn't mean going up stairs. It did a few centuries ago, but hasn't meant that during the lifetime of anyone alive now. Similarly, proto-Chinese of N thousand years ago may have been mostly single-syllable words, but this hasn't been true for at least the few thousand years that we have readable examples of the writing system.
For a Mandarin example, which I'll write in pinyin (or pin1yin1;-) to get past the
For an entertaining debunking of both this myth and a very common trope among Western pseudo-intellectuals and pop psychologists, read this article at languagelog. After chuckling at that particular bit of silliness about Chinese writing, you can find other articles there that go into the general problem in more detail. A number of experts in East-Asian linguistics regularly contribute to that blog, and they've been pushing for a campaign to debunk the nonsense that Westerners insist on saying about these languages.
Oh, well; I haven't yet heard any claim that Chinese doesn't have a word for "freedom". But I wouldn't be surprised. (Hint: the word starts with the same character as the above "zi4ran2", but has a different second character.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
"No one likes working in a hamster wheel, and your shop smells of cedar shavings from here." - TaleSpinner
1) You escape a Unicode string that contains fullwidth characters. The fullwidth characters have no special properties, so they aren't escaped.
2) You translate the escaped Unicode string into ASCII. Fullwidth characters are translated into halfwidth characters. Some of those halfwidth characters, like quotes, have special properties.
The fullwidth quote was perfectly safe, because it wasn't treated like a quote. It was treated the same as an "A" or "b". But when it was translated to a "normal" quote, it went from being a plain old character to being a quote character, with a completely different meaning.
The lesson here is that you should never translate fullwidth characters into halfwidth characters unless you know whether they should be escaped or not, and you should escape them during translation if they need to be. Also, it's not a good idea to translate an escaped string between character sets.
[T]he classic Chinese numeric notation is not as convenient as 'arabic' notation. But it's much less unwieldy than say Roman numerals, so I don't think it would have been an insumountable hurdle either.
Actually, classical Chinese numbers are only slightly worse than Arabic notation (which apparently developed in India but was spread by Arab traders who knew a good accounting system when they saw it). The Chinese notation was far better than any of the Western number notations that the Arabic notation supplanted, such as the Greek or Hebrew notation. Roman was probably the worst notation ever invented, and nobody ever really used it for accounting.
The basis of the Chinese system was symbols for 1 to 9, and symbols for powers of 10. To illustrate with ascii characters, the symbol for 10 looks like a large '+' sign, so we can use + for 10, H for hundred, T for thousand. We'd write the number 5347 as 5T3H4+7. Unused powers of 10 are omitted, so 2007 would be 2T7. 1024 would be T2+4 or 1T2+4. And so on. There are symbols for a few more powers of 10, and they can be chained to get higher powers of 10, so HT could be used for 100,000.
Nowadays, most numeric work in East Asia is done using the Western version of Arabic notation. But you also see a hybrid form that uses the Chinese 1-9 characters plus the Western 0. Converting between this notation and the traditional Chinese notation is essentially trivial and can be done as fast as you can write the numbers. But for arithmetic on paper, the Arabic form (or Arabic with Chinese digits) is a bit simpler than the traditional Chinese notation, since using 0 as a place holder results in correct alignment in columns of numbers, and the digits 1-9 are a bit faster to write than the Chinese digits.
An interesting aspect of the Chinese system is that the basic symbols have alternate "fancy" forms with a lot more strokes. These characters have the property that you can't add strokes to convert them to a different character. So they're an anti-tampering, fraud-proof way of writing numbers. I don't know of another numeric notation with this feature. Asian financial documents have historically used these fancy forms of numbers.
Actually, the Chinese and Arabic notations are the 3rd and 2nd easiest numeric notation that various societies have invented. A few years ago, Scientific American had an interesting article explaining the Mayan number system, and included an explanation of why it was a lot easier to use than the Arabic system. For example, instead of the big multiplication table that we memorized in school, the Mayan system really only needs one rule: 5x5=15. (This makes sense if you understand that they used base 20.) The rest of the rules for adding, subtracting and multiplying consist of the techniques for "carrying" and "borrowing", and are essentially similar to what you do with an abacus.
But I suppose we're stuck with the Arabic system. It's good enough, really, for the remaining uses where we don't bother with a computer.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.