Slashdot Mirror
Unicode and the Unix Console?
Phactorial asks: "At it's current state, most UNIX consoles (not graphical terminal emulators, mlterm is out for this) I have dealt with do not handle unicode properly. This is essential when it comes to dealing with languages that require characters that are not in the current ASCII set. I was wondering if anyone out there is developing a solution for non-Linux platforms. I know the Arabeyes project is currently working on a project called 'Akka' which provides UTF-8 (kinda) support and even shaping and bidirectional code (essential for many languages in the East, the program works fine and I am working on getting a FreeBSD port out). However, I was pondering, how are other UNIX consoles doing? Do any of them fully support unicode, even bidirectional characters? shaping? (a great many of today's UNIX applications lack many if not all of these ;(). If you know of such applications or are working on support for a platform, could you give feedback as to your experiences and thoughts on the current state of the UNIX console?"
From text of question:
(not graphical terminal emulators, mlterm is out for this)
I was wondering if anyone out there is developing a solution for non-Linux platforms.
The answer "Sure, there's this graphical terminal emulator in a recent linux distro!" seems somewhat inappropriate to the question.
Solaris 2.6 supports 56 "locales" and is six or so years old now. Is this what you were asking about? I don't have experience with non-USA locales, but it seems the UNIX people have realized that there are countries outside of North America and have tried to accomodate them.
Vote in November. You won't regret it.
In short: Yikes! UNIX is a timesharing system for TTY terminals from 1979*.
That's a rather depressing outlook. We need to do better. This is supposed to be a discussion about that, not just another 'UNIX is UNIX because it is UNIX' polemic.
(* Just stating the facts. I connected to a SparcStation with a VT220 terminal as a serial console just last week- it was handy and it's cool that it works.)
If you start handling Unicode in files, then you need unicode in file names, because users will try to name them that way. If you allow unicode in file names, then you need to have make understand unicode, because someone will name all their .c files with Cyrillic characters. The shells then need it for completion. Soon you realize that you need the C compiler to understand unicode as well, so that you can have unicode variable names, etc.
So I think it would be best to bite the bullet and go all the way. It will require some planning -- the gcc folks would have to decide gcc will read unicode in 4.0, and Linus would have to decide linux 3.0 will be in unicode. Then the various distributions will have to come out with "unstables" or "rawhides" or whatever they call them, and slowly beat thousands of little apps each with their own presumptions on the size of a text character into submission.
plan9 is unicode inside and out. I'm not advocating it over simply improving good old linux, but it can be examined for lessons and ideas.
Not flamebait, just nonsense.
.Net are 100% Unicode. Windows and Macintosh are now all Unicode based.
Unix isn't byte based, it's text based. Of course one layer deeper, it's byte based, but so is every other OS, and below that it's transistor based, etc.
What distinguishes Unix from other OSes is an emphasis on working in text with text utilities, often thru windows (telnet clients) on other machines -- windows whose only supported datatype is text.
In Unix, as in XML, text is sort of considered the ultimate data type. Bytes are just the medium used to represent the text under the surface. If the bytes were what mattered, people would usually work in a hex editor and do hex I/O, but they don't. They work at the text level of abstraction most of the time. It's the text that matters, not the bytes used to digitize it.
For text to reach its full potential, though, you have to say goodbye to grampa's ASCII and move on to a rich, universal form of text: Unicode. It's ludicrous for someone to say that speakers of non-Western languages should never have the ability to use the full range of Unix the way Westerners can. People who make comments like that are usually unaware of the problems that even English speakers have with single-byte encodings. (The second most powerful currency on earth is the Euro. Where is the Euro sign in Latin-1? Where are the curly quotes used by almost all English-language press? What happens when a press release destined for Time Magazine gets piped thru a series of single-byte Unix utilities? Undefined!)
XML, another system that considers text the universal data type, is Unicode based. They understand the concept of "universal". Same for HTML now. More and more Web pages are going to UTF-8, even for English, to avoid weird problems with Macs vs. PCs, Euro signs, curly quotes, embedded non-English text, etc. Are such pages really supposed to be out of reach of standard Unix utilities?
Java and
IETF and W3C have made it clear that no non-Unicode-based text protocols will be considered from now on.
Oracle is recommending Unicode as the format for all database text for new databases. So what happens when you cripple Unix so that it can't handle Oracle data in default form?
AT&T considers Unicode the future of Unix (cf. Plan 9), Sun has made the conversion to full Unicode fundamental to the future of Solaris, and as we speak the Free Standards Organization is preparing to do the same for an upcoming version of the LSB (Linux Standards Base) common core that all major Linux vendors have committed to.
It's unfortunate that so many Unix users still think that ASCII was good enough for grampa, so it should be good enough for every Unix user on earth from now on, but fortunately those who drive the standards have abandoned that kind of thinking forever.
"Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
You can keep the byte orientation and still have Unicode support. See this.
However, I was pondering, how are other UNIX consoles doing? Do any of them fully support unicode, even bidirectional characters? shaping? (a great many of today's UNIX applications lack many if not all of these ;( ). If you know of such applications or are working on support for a platform, could you give feedback as to your experiences and thoughts on the current state of the UNIX console?"
Whoa there, cowboy. Let's work on getting the delete key to work properly before we try any of that fancy stuff! If I never have to type stty erase again, I'll be a happy bunny!
People who fear that a switch from US-ASCII to UTF-8 will break their existing programs should really read the Bell Labs document linked above, section 2.3 of the Unicode spec, or RFC 2044. UTF-8 was designed very carefully to make life extremely easy for people making that exact migration. There are amazingly few circumstances where it even matters that it is variable width. Those people who are suggesting UCS-2, UCS-4, etc. as alternatives in order to solve the nonexistant problem of UTF-8's variable width nature should really take a closer look at it.
But my grandest creation, as history will tell,
Was Firefrorefiddle, the Fiend of the Fell.
Well, you could start by looking at everybody who wrote that software you mention.
People who write that software never use their "internationalization" -- they see it as a "feature" to add in the list of marketing checkboxes.
Then add everybody who has to deal with more than just Ameri^H^H^H^H^HEnglish text on a day-to-day basis.
That will be me -- and I hate Unicode.
Probably took too small a survey, then. People in my lab write them every day. We write mostly in English (sometimes German), and refer to people, locations, and events in a dozen European countries. Using some pre-Unicode technique, like "codepages", would be a nightmare.
Almost all European languages, including English, are in a single iso8859-1 charset -- what happens to coincide with the beginning of Unicode table. People who use iso8859-1 can "switch to Unicode" and continue using just the same thing with longer bytes, getting no benefit whatsoever but pretending to have "internationalized" their software. For everyone else Unicode causes nothing but trouble, waste of resources and incompatibilities.
As for "code pages" this is a DOS/Windows kludge that is a dumb idea in its own way -- everyone else uses _charsets_ and those can be easily displayed in pretty much everything. The only problem is, no one bothered to make a usable (that means, not XML) tagged format that can include information about languages and charsets used in a document. MIME has charset information for parts of the document, and substrings in the header but not substrings in the document, so it isn't really usable either, however can be used as a proof of viability -- most of mail clients have it all implemented, therefore metainformation with charsets can be easily used.
Contrary to the popular belief, there indeed is no God.
ISO 8859-1 is "West European". A quick web search seems to indicate it covers about half of the European languages.
And this is who in Europe actually "uses" it.
As a software developer, it pays of in spades because I don't have to answer any questions about languages: if they're in Unicode, they'll always be there.
At the expense of crippling the software.
No more wondering how to get this language to display in that web browser; it just works.
Software is not to "wonder how it will look in a web browser", it's to operate on data. Most of operations can be absolutely byte-value-transparent and they must never depend on charsets and languages in the first place, however ones that are dependent on them usually have to either use tags with metainformation (and there Unicode is not any better than anything else) or do some horrible guesswork -- say, which language is used in a certain chunk of text that contains some characters (what simply should never be done in the first place, but Unicode accomplishes nothing for it because characters are shared between languages). Displaying pretty characters is easy. So easy, one should never think about it. However a computer is not a typewriter, and therefore charsets and encodings should never be designed to simplify a tiny bit the simple task of displaying while turning any complex text processing into a complete hell.
If I want a character, I can look up the bytes; if I know the bytes, I can look up the character.
How can a computer LOOK UP the bytes? There is nothing BUT the bytes in the computer's memory so certainly it can't look them up. You can use "bytes" as an index in a font, or you can pass them to some processing routine.
No more "incompatibilities" than you'd get from bugs in any other library on your system.
There are no bugs in the font renderers already -- this is a non-issue. The problem is entirely in trying to force people to make huge amount of assumptions about the data's content in otherwise byte-value transparent operations, just to accommodate UTF-8 where it should not matter. It's a design issue, not implementation.
Contrary to the popular belief, there indeed is no God.