Slashdot Mirror
Unicode and the Unix Console?
Phactorial asks: "At it's current state, most UNIX consoles (not graphical terminal emulators, mlterm is out for this) I have dealt with do not handle unicode properly. This is essential when it comes to dealing with languages that require characters that are not in the current ASCII set. I was wondering if anyone out there is developing a solution for non-Linux platforms. I know the Arabeyes project is currently working on a project called 'Akka' which provides UTF-8 (kinda) support and even shaping and bidirectional code (essential for many languages in the East, the program works fine and I am working on getting a FreeBSD port out). However, I was pondering, how are other UNIX consoles doing? Do any of them fully support unicode, even bidirectional characters? shaping? (a great many of today's UNIX applications lack many if not all of these ;(). If you know of such applications or are working on support for a platform, could you give feedback as to your experiences and thoughts on the current state of the UNIX console?"
Its Gnome 2 terminal can deal with any truetype unicode font, even those that are proportionally spaced such as the luscious, but now under-wraps, 'Arial Unicode MS'. RH 8's vim is also unicode savvy.
A major improvement for my line of work.
From text of question:
(not graphical terminal emulators, mlterm is out for this)
I was wondering if anyone out there is developing a solution for non-Linux platforms.
The answer "Sure, there's this graphical terminal emulator in a recent linux distro!" seems somewhat inappropriate to the question.
Solaris 2.6 supports 56 "locales" and is six or so years old now. Is this what you were asking about? I don't have experience with non-USA locales, but it seems the UNIX people have realized that there are countries outside of North America and have tried to accomodate them.
Vote in November. You won't regret it.
etc.
BeOS unicode native. I'd expect that FreeBeOs (or whatever it's called) is the same, and I think it's also Unix compatible?
(This will be considered flamebait, but someone has to say it.)
The way I see it, we shouldn't be cluttering a clear, simple and sane interface like the unix console with complexity like unicode. Unix is inherently byte based, and unix terminals are byte based. If it's not a byte, don't put it in a unix terminal.
This isn't to say that we shouldn't have other mechanisms for supporting foreign languages - but this particular path has been travelled before and it's not pretty. Look at the AS/400 - tables stored in the DB/FS are marked as being in a particular character set, and the OS tries real hard to fix up and convert from set to set as needed. This causes countless problems in the infrequent cases where there is no possible mapping between sets.
Another way to look at it - why don't we have unicode support for grep? Why aren't all files tagged with an appropriate character set, so we know what they're really supposed to look like? When you 'tail -n 20' a file, how does tail know that those line feeds and carriage returns aren't part of some unicode char?
In short, unix is byte based. All the unix tools are byte based. If you want to use unicode, build a unicode layer on top of the bytes, but dont screw with the existing stuff that already works perfectly well.
Alter Aeon Multiclass MUD - http://www.alteraeon.com
If you start handling Unicode in files, then you need unicode in file names, because users will try to name them that way. If you allow unicode in file names, then you need to have make understand unicode, because someone will name all their .c files with Cyrillic characters. The shells then need it for completion. Soon you realize that you need the C compiler to understand unicode as well, so that you can have unicode variable names, etc.
So I think it would be best to bite the bullet and go all the way. It will require some planning -- the gcc folks would have to decide gcc will read unicode in 4.0, and Linus would have to decide linux 3.0 will be in unicode. Then the various distributions will have to come out with "unstables" or "rawhides" or whatever they call them, and slowly beat thousands of little apps each with their own presumptions on the size of a text character into submission.
plan9 is unicode inside and out. I'm not advocating it over simply improving good old linux, but it can be examined for lessons and ideas.
Not flamebait, just nonsense.
.Net are 100% Unicode. Windows and Macintosh are now all Unicode based.
Unix isn't byte based, it's text based. Of course one layer deeper, it's byte based, but so is every other OS, and below that it's transistor based, etc.
What distinguishes Unix from other OSes is an emphasis on working in text with text utilities, often thru windows (telnet clients) on other machines -- windows whose only supported datatype is text.
In Unix, as in XML, text is sort of considered the ultimate data type. Bytes are just the medium used to represent the text under the surface. If the bytes were what mattered, people would usually work in a hex editor and do hex I/O, but they don't. They work at the text level of abstraction most of the time. It's the text that matters, not the bytes used to digitize it.
For text to reach its full potential, though, you have to say goodbye to grampa's ASCII and move on to a rich, universal form of text: Unicode. It's ludicrous for someone to say that speakers of non-Western languages should never have the ability to use the full range of Unix the way Westerners can. People who make comments like that are usually unaware of the problems that even English speakers have with single-byte encodings. (The second most powerful currency on earth is the Euro. Where is the Euro sign in Latin-1? Where are the curly quotes used by almost all English-language press? What happens when a press release destined for Time Magazine gets piped thru a series of single-byte Unix utilities? Undefined!)
XML, another system that considers text the universal data type, is Unicode based. They understand the concept of "universal". Same for HTML now. More and more Web pages are going to UTF-8, even for English, to avoid weird problems with Macs vs. PCs, Euro signs, curly quotes, embedded non-English text, etc. Are such pages really supposed to be out of reach of standard Unix utilities?
Java and
IETF and W3C have made it clear that no non-Unicode-based text protocols will be considered from now on.
Oracle is recommending Unicode as the format for all database text for new databases. So what happens when you cripple Unix so that it can't handle Oracle data in default form?
AT&T considers Unicode the future of Unix (cf. Plan 9), Sun has made the conversion to full Unicode fundamental to the future of Solaris, and as we speak the Free Standards Organization is preparing to do the same for an upcoming version of the LSB (Linux Standards Base) common core that all major Linux vendors have committed to.
It's unfortunate that so many Unix users still think that ASCII was good enough for grampa, so it should be good enough for every Unix user on earth from now on, but fortunately those who drive the standards have abandoned that kind of thinking forever.
"Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
However, I was pondering, how are other UNIX consoles doing? Do any of them fully support unicode, even bidirectional characters? shaping? (a great many of today's UNIX applications lack many if not all of these ;( ). If you know of such applications or are working on support for a platform, could you give feedback as to your experiences and thoughts on the current state of the UNIX console?"
Whoa there, cowboy. Let's work on getting the delete key to work properly before we try any of that fancy stuff! If I never have to type stty erase again, I'll be a happy bunny!
Shell scripting language.
Pixels keep you awake!
I don't want a BSD port.
I want my OpenBSD to be native utf-8, nothing else.
Currently it is not locale/NLS aware (which I consider
A Good Thing(tm)), but handles eight-bit I/O as if
it was iso-8859-1. I want it to change that to utf-8
because more characters ( comes to mind) can
be handled that way.
My Karma isn't excellent, damn it! (And
Kterm is xterm with double byte support. It's been available since before unicode, but you shouldn't have any trouble hacking it to use a unicode font. http://packages.debian.org/stable/x11/kterm.html
even the c source code
http://plan9.bell-labs.com/plan9
okok, It's a graphical OS but bitmap terminals are hardly hard to come by
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
Sometimes it can get a little ugly, though. e.g. footnote 52 from the standard:
C99 is also the first time the extended character set is required to map to UCS, too (see Annex D). Pretty crazy.I don't believe gcc has put in extended identifiers yet, though (it's gaining C99 compliance pretty slowly).
To be honest I don't know why the difference in numbering. You have UTF-32, but UCS-4? What's the deal? Oh well.
FWIW, I believe that on some Crays, bytes are 32 bits in size. They had to come up with a pretty wack C implementation to get 8-bit chars. I believe 8 bits per byte is a POSIX standard?
The question was explicitly not about terminal emulators (xterm, mlterm, kterm, whatever). However, if you want a unicode-supporting xterm, then why not use gen-oo-wine xfree86 xterm? Yes, xterm indeed supports unicode (at least since Xfree4.2), if you use the -u8 switch. Debian's xterm package even comes with a script, uxterm, to set up variables and resources as appropriate.
Classification 5 in http://acharya.iitm.ac.in/iitmsoft.html seems related
Seriously, I have yet to see a person (other than Martin Duerst who apparently made a career of stuffing Unicode into everything he notices) willingly using Unicode, as opposed to being forced to do that by some software that requires it. The "internationalization" of documents is a strawman -- at this point in history no non-linguistic document contains more than two languages, local charsets handle that perfectly, and linguists went far beyond what Unicode can provide already, so they have to use different formats anyway. If and when true internationalization will be necessary people will need one simple thing -- language/charset tagging. Tagging is also important because it makes those texts "machine-readable" -- programs will know what parts of text they should interpret using rules that apply to different languages and charsets, and pass "as-is" everything that is in the languages they don't know.
XML already allows language attribute in all tags, and if charset attribute will become valid everywhere where the language is valid, problem will go away immediately and without mandatory Unicode adoption everywhere because everyone who can read a language has a font of a charset that is used with it, and everyone who doesn't shouldn't have problem with occasional
"can't display this section of text, ( ) download "klingon fixed" font to make it readable, (x) show as block, do not edit, ( ) display/edit in hex".
Obviously, a console, or any other kind of program can easily be modified to do that if necessary, and there will be no loss for people that, like myself, simply use their native language + ascii charset, and switch to all other charsets using nice xterm font menu.
Contrary to the popular belief, there indeed is no God.
If I remember correctly, the text modes have the font stored in a 2048 bytes array, with every character having a byte per line, and 8 lines per character. I don't think there's any way of squeezing more chracters into a text mode, unless video card designers come up with some extension.
So probably if Linux is made to support Unicode correctly this will only work in framebuffer modes, where it's possible to have as many characters as you want. That would mean a lot of improvement is needed in this area. For example the rivafb driver/nVidia X driver would need to be fixed to coexist. Sure vesafb can be used, but it's painfully slow, and some really old cards don't support it.
Here is a neat paper describing how Plan9 made the full transition to Unicode. Not exactly an easy feat, although it was harder than necessary for them because they decided to do it back when Unicode was still being standardized. And of course, ASCII isn't going away anytime soon, as there are plenty of systems that don't need it but do need all the memory they can get.
In the great CONS chain of life, you can either be the CAR or be in the CDR.
I prefer the Unix console, myself.
Sex - Find It
ac, you fail it.
Sex - Find It