Slashdot Mirror
Unicode and the Unix Console?
Phactorial asks: "At it's current state, most UNIX consoles (not graphical terminal emulators, mlterm is out for this) I have dealt with do not handle unicode properly. This is essential when it comes to dealing with languages that require characters that are not in the current ASCII set. I was wondering if anyone out there is developing a solution for non-Linux platforms. I know the Arabeyes project is currently working on a project called 'Akka' which provides UTF-8 (kinda) support and even shaping and bidirectional code (essential for many languages in the East, the program works fine and I am working on getting a FreeBSD port out). However, I was pondering, how are other UNIX consoles doing? Do any of them fully support unicode, even bidirectional characters? shaping? (a great many of today's UNIX applications lack many if not all of these ;(). If you know of such applications or are working on support for a platform, could you give feedback as to your experiences and thoughts on the current state of the UNIX console?"
From text of question:
(not graphical terminal emulators, mlterm is out for this)
I was wondering if anyone out there is developing a solution for non-Linux platforms.
The answer "Sure, there's this graphical terminal emulator in a recent linux distro!" seems somewhat inappropriate to the question.
(This will be considered flamebait, but someone has to say it.)
The way I see it, we shouldn't be cluttering a clear, simple and sane interface like the unix console with complexity like unicode. Unix is inherently byte based, and unix terminals are byte based. If it's not a byte, don't put it in a unix terminal.
This isn't to say that we shouldn't have other mechanisms for supporting foreign languages - but this particular path has been travelled before and it's not pretty. Look at the AS/400 - tables stored in the DB/FS are marked as being in a particular character set, and the OS tries real hard to fix up and convert from set to set as needed. This causes countless problems in the infrequent cases where there is no possible mapping between sets.
Another way to look at it - why don't we have unicode support for grep? Why aren't all files tagged with an appropriate character set, so we know what they're really supposed to look like? When you 'tail -n 20' a file, how does tail know that those line feeds and carriage returns aren't part of some unicode char?
In short, unix is byte based. All the unix tools are byte based. If you want to use unicode, build a unicode layer on top of the bytes, but dont screw with the existing stuff that already works perfectly well.
Alter Aeon Multiclass MUD - http://www.alteraeon.com
If you start handling Unicode in files, then you need unicode in file names, because users will try to name them that way. If you allow unicode in file names, then you need to have make understand unicode, because someone will name all their .c files with Cyrillic characters. The shells then need it for completion. Soon you realize that you need the C compiler to understand unicode as well, so that you can have unicode variable names, etc.
So I think it would be best to bite the bullet and go all the way. It will require some planning -- the gcc folks would have to decide gcc will read unicode in 4.0, and Linus would have to decide linux 3.0 will be in unicode. Then the various distributions will have to come out with "unstables" or "rawhides" or whatever they call them, and slowly beat thousands of little apps each with their own presumptions on the size of a text character into submission.
plan9 is unicode inside and out. I'm not advocating it over simply improving good old linux, but it can be examined for lessons and ideas.
Not flamebait, just nonsense.
.Net are 100% Unicode. Windows and Macintosh are now all Unicode based.
Unix isn't byte based, it's text based. Of course one layer deeper, it's byte based, but so is every other OS, and below that it's transistor based, etc.
What distinguishes Unix from other OSes is an emphasis on working in text with text utilities, often thru windows (telnet clients) on other machines -- windows whose only supported datatype is text.
In Unix, as in XML, text is sort of considered the ultimate data type. Bytes are just the medium used to represent the text under the surface. If the bytes were what mattered, people would usually work in a hex editor and do hex I/O, but they don't. They work at the text level of abstraction most of the time. It's the text that matters, not the bytes used to digitize it.
For text to reach its full potential, though, you have to say goodbye to grampa's ASCII and move on to a rich, universal form of text: Unicode. It's ludicrous for someone to say that speakers of non-Western languages should never have the ability to use the full range of Unix the way Westerners can. People who make comments like that are usually unaware of the problems that even English speakers have with single-byte encodings. (The second most powerful currency on earth is the Euro. Where is the Euro sign in Latin-1? Where are the curly quotes used by almost all English-language press? What happens when a press release destined for Time Magazine gets piped thru a series of single-byte Unix utilities? Undefined!)
XML, another system that considers text the universal data type, is Unicode based. They understand the concept of "universal". Same for HTML now. More and more Web pages are going to UTF-8, even for English, to avoid weird problems with Macs vs. PCs, Euro signs, curly quotes, embedded non-English text, etc. Are such pages really supposed to be out of reach of standard Unix utilities?
Java and
IETF and W3C have made it clear that no non-Unicode-based text protocols will be considered from now on.
Oracle is recommending Unicode as the format for all database text for new databases. So what happens when you cripple Unix so that it can't handle Oracle data in default form?
AT&T considers Unicode the future of Unix (cf. Plan 9), Sun has made the conversion to full Unicode fundamental to the future of Solaris, and as we speak the Free Standards Organization is preparing to do the same for an upcoming version of the LSB (Linux Standards Base) common core that all major Linux vendors have committed to.
It's unfortunate that so many Unix users still think that ASCII was good enough for grampa, so it should be good enough for every Unix user on earth from now on, but fortunately those who drive the standards have abandoned that kind of thinking forever.
"Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
ISO 8859-1 is "West European". A quick web search seems to indicate it covers about half of the European languages.
And this is who in Europe actually "uses" it.
As a software developer, it pays of in spades because I don't have to answer any questions about languages: if they're in Unicode, they'll always be there.
At the expense of crippling the software.
No more wondering how to get this language to display in that web browser; it just works.
Software is not to "wonder how it will look in a web browser", it's to operate on data. Most of operations can be absolutely byte-value-transparent and they must never depend on charsets and languages in the first place, however ones that are dependent on them usually have to either use tags with metainformation (and there Unicode is not any better than anything else) or do some horrible guesswork -- say, which language is used in a certain chunk of text that contains some characters (what simply should never be done in the first place, but Unicode accomplishes nothing for it because characters are shared between languages). Displaying pretty characters is easy. So easy, one should never think about it. However a computer is not a typewriter, and therefore charsets and encodings should never be designed to simplify a tiny bit the simple task of displaying while turning any complex text processing into a complete hell.
If I want a character, I can look up the bytes; if I know the bytes, I can look up the character.
How can a computer LOOK UP the bytes? There is nothing BUT the bytes in the computer's memory so certainly it can't look them up. You can use "bytes" as an index in a font, or you can pass them to some processing routine.
No more "incompatibilities" than you'd get from bugs in any other library on your system.
There are no bugs in the font renderers already -- this is a non-issue. The problem is entirely in trying to force people to make huge amount of assumptions about the data's content in otherwise byte-value transparent operations, just to accommodate UTF-8 where it should not matter. It's a design issue, not implementation.
Contrary to the popular belief, there indeed is no God.