Slashdot Mirror
Migrating Large Scale Applications from ASCII to Unicode?
bobm asks: "We've been asked to migrate our newer applications to Unicode. My biggest issue is that if we start storing user data in unicode we will no longer be able to provide complete updates the legacy (pure ASCII) systems. This is important in that we are currently updating > 25k customers a day and managment does not want that to be affected. I also haven't found a clean way to provide multilanguage data mining that can return a single language output. This doesn't even begin to address issues like data validation and display issues. (note: we currently handle the web pages in multiple language sets but require the data to be in ascii form.)
I've spent some time on Unicode.Org but I really haven't found any real world discussions on people doing this on a large scale (>1Tbyte databases)."
You don't mention any specifics, so it's hard to give details in response. What databases? How free hands do you have?
I'd suggest a message oriented XML based system. You can model to your hearts content in XML, languages, charset etc. You can design near anything around that, and have various backends convert the XML messages (SOAP possibly) to the kind of data that's useful for the given backend.
Unable to read configuration file '/bigassraid/htdig//conf/14229.conf'
Geocrawler error message.
All decent databases have unicode support and allow you to convert the data on the fly. What's the problem here? And if you use UTF-8 encoding you have ASCII combatiabilty (sp)...
J.
Why not encode the data using XML... that way most of your data already maps to the real data.
This would be without the XML tags, of course. Just the encoding of the data...
Thus, you will be using UNICODE, and encoding it in XML text.
Hmm... at some places you may need an XML to unicode translator.
The fact that you are still storing and transfering your data in ASCII, does not mean it's a ASCII system... it's only your communication medium. This way systematic migration may become more possible.
What might be useful is to read how StarOffice, did their unicode and internationalization changes to an existing large code base at sun.com
C.
I sometimes write stuff
No you don't.
Your comments record with all those -1 scores indicate that you're just a lameass.
In fact I don't even know why I am wasting my time replying to this. Gotta go, see you never.
For older clients, simply send a question mark or similar for any character not in the ASCII character set; this is extremely trivial to add to your back end. New clients get unicode and all the trappings that go with it. Be sure your support people are trained to explain that updating the client provides the new multinational functionality and eliminates the question mark placeholders.
Regarding your question about different languages/encodings - you may need to include the language per record all the way through to the client end. Without knowing more about your output system, it's difficult to say what the display issues are, but it's difficult to believe many display libraries would limit you to a language per session.
Hehe... seems like someone else had the same idea already.
And, I must also insist that more domain specific information be given to aid in giving a solution.
PS: By no mean do I think XML is the begin and end of all things... just that it may actually be useful here...
;)
http://www.ebxml.org/
You should really set your moderation threshold higher... replying to thoses posts are pointless, and cause follow-up posts like this etc. etc.
;)
(I saw the post because I wanted to see if who was first with a specific suggestion...)
</offtopic>
If your application returns results in XML you can always encode "safely" parts of the text using character entities (&#nn;). An other solution is to return not one but several results, in various encodings (you would have either to store the native encoding of a text or to figure out what it could be)
And I hope this kind of practical discussion can help to raise the level of interest in Unicode amongst application coders.
Although a lot of "core" coders (as in people who write languages and tools) are really into Unicode and trying to get their code to process it properly I found that most "application programmers", people who use those tools, are not at all interested. They tend to think that all software should support their favorite encoding natively. They also tend to curse alot when they get data in a different encoding ;--) Usually they view Unicode as yet another curse thrown upon them by an irresponsible buzzword-worshipping management.
In fact Unicode is certainly hard an painful to implement, but it is a standard and at least written by people who know what they're doing. It solves problems that most of us either have had to deal with (oh the agony of dealing with odd characters in SGML data) or will have to deal with,:face it people, there are more and more people whose names include funny characters, even in the US, to leave that market untapped.
So please view Unicode as a chance, and if the poster can do it on a terabyte of data, you can certainly do it on much less, especially as the tools are coming (yes, even Perl!)
Look, that's why there's rules, understand? So that you think before you break 'em. (Terry Pratchett)
Why not stick with the most used languages on the planet then? Chinese or Spanish?
Actually the easiest thing would be for everyone to learn chineese. There are at least twice as many people who speak chineese as there are people who speak english; hence don't have to teach that many people chineese as compared to english.
We do not live in the 21st century. We live in the 20 second century.
Latin-1 accomodates Western Europe and the Americas. It doesn't work for Eastern Europe or Asia. With Latin-1, you're cutting out potential profits from Greece, Russia, Arab countries, China, and Japan. For an international company, Unicode IS about making money.
Check this standart for unicode compression.
It compresses 16 bit unicode chars to 8bit using some reserved tags to switch the character windows. Sample java implementation is avaiable. The best thing is that most of the standart ASCII chars will still be encoded as 8bit ASCII after the compression. So you can still store all your data in 8bit ASCII and convert it to unicode before displaying it. And you don't have to modify your old data!!!
Interesting read... but it's all just logic and truth.
It won't find an open-minded intelligent audience (At least not a large one) in the States.
Especially not now.
Besides posting things like this on a forum such as slashdot.org will rally less support and more iritation.
The title of this article alone was enough to give me horrible flashbacks to working with EBCDIC/ASCII conversions and IBM's weirdly proprietary and immutable standards. Thank GOD for Larry Wall, is all I have to say about that.
Free music from Jack Merlot.
A very useful resource on Unicode is this page, written by Markus Kuhn. In particular you may be interested in How do I have to modify my software?; while it does concentrate on Unix, the general principles should be the same on any OS.
Steven Murdoch.
web: http://www.cl.cam.ac.uk/users/sjm217/
What's the problem? If you use the UTF-8 encoding
for Unicode, all your data will be ASCII compatible.
Much easier to do that sort of thing in C
...... };
char get_ascii(char ebcdic)
{
char map[256]={
return map[ebcdic];
}
Whats the big deal?
Oracle 8i, UTF8 character set. Compatibility with both Unicode and ASCII character sets. What're the problems? Well, clients that think that Unicode is UCS2, is one to watch out for, or forgetting that there's more to life than Western European ISO.
Basically, 90% of the problems you will encounter is in converting between character sets to integrate with other things. If you can use Java (Unicode native) and PL/SQL for as much as possible, you'll have fewer problems. If your client is Excel (don't ask) that complicates matters. If you can assume that everything in the database is US7ASCII you're all set, because you won't need to do any data cleansing. If you have to convert stuff that's already there, then you will run into problems, what happened to me is that we had a Western European encoding, but people were entering Cyrillic data. It all came out fine on their desktops, which were configured for that character set, but the actual data in the database was gibberish as far as the queries were concerned. Non-trivial to fix.
Good luck!
IMHO XML isn't just "the flavor of the month", it's been around sometime now and probably will be (at least I hope).
XML data can be bloated by using verbose tags, but nobody is forcing you to use descriptive tags. If you want just use tags like <a> thru <zzzzzz>
Considering using UTF-8 for export instead of direct Unicode. As long as the legacy systems are 8-bit clean, you can feed UTF-8 back to them without too many problems. There will be no issues at all for ASCII data since 7-bit ASCII is the same in UTF-8. You just need to convert front end applications to be UTF-8 aware. You need not convert legacy backends to understand Unicode, they will just store UTF-8 as some weird 8 bit characters. The beauty is you'll be able to convert them in phases and ASCII never stops working.
Why has this been modded as insightful?
It should be fairly obvious to anyone who is knowledgeable about technology (I hope that includes most Slashdot moderators) that this guy doesn't know what he is talking about.
Because chinese
(a)has only a standardised written form, not spoken form
(b)that written form is especially annoying to represent digitally.
(c) it is a tonal language, and therefore not very easy to learn unless you have been raised from birth speaking it, since your brain won't have developed the requisite pitch analysis. There are many more non-tonal than tonal language speakers in the world, so standardising on a tonal language would place ALL of them at a disadvantage. It's easy for a tonal language speaker to go the other way though.
spanish:
(a) everyone would be spitting all over eachother. That's just the way the language is.
(b) It has bizarre gender constructions. Gendered nouns, again, are easy to learn from birth, but going from a non-gendered to a gendered language is difficult, since the brain's from-birth language database hasn't allocated a row for "gender".
(c) It has annoying verb tense constructions. In english, one can easily construct new tenses to deal with problems encountered when talking about time travel/relativity in physics. "He would have been going to do that last week". That's a pain in the ass in spanish. Hence, native spanish speakers have a much shakier grasp of the concept of time.
We should really standardise on conlang like lojban. Then everyone would be at a roughly equal disadvantage, the language would be totally sanely constructed, amenable to computer parsing, and representable as ascii.
Flamebait? I thought this was fairly funny (and assume that it was intended that way).
but something like 60% of all people who speak chinese never speak to anyone else who speaks another langauge.
Mod point free since 2001
How do you know you were the first post?
Also, can't someone else post quickly while you were typing away?
Just curious...
Unless you mean British English, in which case ASCII is not sufficient to produce a £ sign...
Unselfish actions pay back better
In the development todo for mySQL 4, they have a list of "Things that must be done in the real near future". Quite far down on that list I found:
"* Add support for UNICODE."
That's great, because mySQL 4 is about to be released any day now.
As a PHP developer I wanted to know if php supports unicode. This is what I found:
Strings:
"A string is series of characters. In PHP, a character is the same as a byte, that is, there are exactly 256 different characters possible. This also implies that PHP has no native support of Unicode."
The way I understand this, you have old clients, new clients, and a server that must handle both. And the server and new clients should support Unicode.
First, although this is probably obvious, I should note that if your data is primarily text, then you're looking at a 2Tb database when you start using Unicode (depending on the encoding).
This is sortof like supporting German language entry, and wanting to display it on English clients. Its not easy, but it can be done, to some extent. Most Unicode you encounter will have an equivalent ASCII representation; there are acceptable conversions for almost all non-Eastern character sets. You can serve up a converted representation to your ASCII clients.
DO NOT listen to the bullshit about serving up UTF-8 to ASCII clients. They can't understand it any more than I can understand German ; it will seem to work only for low-ASCII characters, but break for all others.
As for data validation, you are going to have to have two rulesets. One will be client-side ASCII; the other a unicode ruleset used by both the new client and the server. Incoming ASCII from the old client should be converted to equivalent Unicode (that's the easy part) before being validated.
Sorry, no realworld information here either ; certainly not on database that size.
i-name =twylite [http://public.xdi.org/=twylite], see idcommons.net
spanish: (a) everyone would be spitting all over eachother. That's just the way the language is.
I live in Spain and speak spanish. I've never found people spitting on each other a problem, perhaps you're thinking of a particular country in South America.
(b) It has bizarre gender constructions.
Bizarre?? Lots of languagues, perhaps the majority, have this.
English has many idiosyncracies, one of the worst for people that are learning it are that it isn't pronounced as it is written. In this respect, Spanish is much more sensible and easier to learn. Also, phrasal verbs are a nightmare for anyone trying to learn English. In this respect Spanish is also easier.
(c) It has annoying verb tense constructions. In english, one can easily construct new tenses to deal with problems encountered when talking about time travel/relativity in physics. "He would have been going to do that last week". That's a pain in the ass in spanish.
This is relatively obscure. All working languages have their idiosyncracies, including English.
Hence, native spanish speakers have a much shakier grasp of the concept of time.
Is this a joke?
If you store it using UTF-8 (there are lots of options for storing Unicode) your problem may not be that bad. I'm assuming your system is in C or a derivative. UTF-8 avoids the obvious breakage of embedded null bytes. You might need to add an output filter to make sure you don't ship out any characters numbered higher than 127 to non-Unicode-savvy customers.
On the other hand, if you've got deep assumptions that strlen(whatever) == numberOfCharsIn(whatever) then you're pretty well hosed.
To bring it back into computing terms, it would actually be best for everyone to do English - the main reason being the small set of glyphs required to express full words and sentences (i.e. only 26 letters and the various punctuation, unlike european lanquages with all the accents and chinese et al with the 10,000 symbols or however many...)
Because Chinese-speaking people can speak English but not many non-Chinese can speak Chinese. Hell, most Chinese can't talk to most other Chinese due to two primary written forms (old, simplified), 31 major dialects (Mandarin now primary, Cantonese losing place) and hundreds of "minor" dialects.
Spanish-speaking people can also speak English [1] but most English-speakers can't handle much beyond "Yo quiero Taco Bell".
All over the world, you see many people of different nationalities -- none of whom have English for a Mother Tongue -- talking in English to each other, albeit with accents and of varying quality. It is the de facto "lingua franca", which, incidentally, means "French language", which used to be the world-wide basis for talking. Hell, during the Napoleonic Wars, the British and Germans fighting the French often had to speak in French to each other.
English took over more than fifty years ago. Remember that English is a bastard language which came together a a mixture of many different languages (primarily Old Norse [like Icelandic], Old German and French). It is adaptable. It is easy to grasp the basics and communicate your intended meaning despite incredibly bad construction and grammar, unlike in most other languages. So lay off.
woof.
[1] excludes New York, Texas, California, Florida
Actually I used XML for 3 months. WHy is it when
anyone disagrees with someone on here they're
immediately labelled as ignorant? Insecurity
perhaps?
Make sure you use UTF8. Firstly because unlike UCS2 (16bit) it can encode all the characters not a subset of them. Eventually 16bit won't be enough for you. Secondly its 7bit ASCII equivalent so there is no real problem with migration over time.
Thirdly since ascii 7bit is UTF8 ascii space there isnt any data migration to be done to set this up.
What we generalize as chinese is actually many different related languages (dialects) that cannot be mutally understood when spoken, but can be mutally understood when written in the standard manner.
I may have missed the point, but Unicode is a character set. Once you have converted the characters in Unicode, you still have to store them. Instead than using UCS-2 (two bytes per character), you may store them in UTF-8, where codes (0-127) are represented exactly like in ASCII.
I agree!
- I've got a swedish keyboard at home, american and a swiss-german keyboards at work and from time to time I have to use the french keyboard. People around me speak english, high-german, swiss-german, italian, and french.
CAN'T WE JUST STOP THIS MADNESS AND JUST USE ENGLISH! - It's the one language everyone understands!
> Unicode is 2 bytes per char, ascii is 1. A > simple converstion program is trivial to write, > you simply have to find the mappings. This is not entirely true, there are many different ways to store unicode: utf-7,utf-8,utf-16,utf-32,ucs-2,ucs-4,etc... I believe you are talking about encodings, Unicode is like ascii, one numeric identifer = one character. Now how you store your numeric identifer is another matter...
Actually I used XML for 3 months
So you know almost nothing then ?
I keep getting involved in usenet flames over XML because I'm still a newbie (not quite 3 years) - and the other guys have something like 10 years experience (they're SGML dinosaurs). Flaming XML is fun - that's why the interesting work has moved beyond it - but if you're going to do this, then attack the real issues with XML, tell us what they are, and tell us what your solutions are.
After all, if you know everything about XML from just 3 months experience, then you're obviously much smarter than we are.
Read Usenet and c.i.w.a.h. You'll get flamed to a crisp by them (they're a little dysfunctional, to put it mildly), but there are a couple of people thereabouts who know how to do this right.
And if you keep zipping a file, it will compress to zero bytes :-)
XML would make it take up several more terabytes if you used it for storage...
This is expensive and slow...
How about Esperanto instead? It is an artificle language as opposed to "natural". It is a rule-based language derived from Latin. No exceptions to the rules the way that Natural languages have (English in particular). Very amiable to computer parsing.
If you use the UTF-8 encoding, of which ASCII is a subset, then you minimize the amount of code and text that has to change -- only the text that isn't expressable in ASCII changes, using multiple bytes per character, and ASCII string manipulations still "just work".
You first have to examine carfully the chracter set your current application can deal with. Is it ASCII? Or just the printable range? Or do most routines treat everything as sequences of 8-bit characters? Is the null character permitted in data? And so on.
After that, you have to identify the operations which are character set specific. This can be quite a bit of work. Character set specific operations include case conversion, collating, normalizing, measuring string length and character width (for formatting plain text), text rendering in general, and so on.
Now you look at your tools. Do they prefer some kind of Unicode encoding? For example, with Java or Windows, using UTF-16 is most convinient (some would say: mandated).
Now you put the pieces together and look for a suitable internal representation (not necessarily "Unicode", i.e. UTF-8, UTF-16, or UTF-32), identify points at which data has to be converted (usually, it is a good idea to minimize this, but if you want to fit everything together, there is sometimes no other choice), and modules and external tools which have to be replaced because adjusting them or adapting to them is too much work.
Your web page generation tools probably need a complete overhaul, so that they are able to minimize the charset being used (for example, German text is sent as ISO-8859-1, but Russian text as KOI8-R or something like that), since client-side Unicode support is mostly ready, but many people don't have the necessary fonts.
Actually I used XML for 3 months. WHy is it when anyone disagrees with someone on here they're immediately labelled as ignorant? Insecurity perhaps?
I complained about your posting being modded as 'insightful' because the posting contained dumb comments:
XML is just the current flavour of the month
This is a dumb comment. XML is built on top the experience of SGML, which has been around for a long time. If you understand the issues involved in software integration across multiple systems then you should understand why XML is a very important standard.
Unicode is 2 bytes per char, ascii is 1. A simple converstion program is trivial to write, you simply have to find the mappings.
Saying this is dumb in the context of the orginal question and also demonstrates a lack of understanding of what's involved in enterprise level software development.
Actually I used XML for 3 months.
So? I am fluent in Spanish. That doesn't mean that I am qualified to make comments about South American politics.
Seriously, there's a huge difference between someone with trivial experience and someone who has worked on major projects at an enterprise level. So I stick by my original comment - you don't know what you are talking about in this context.
But if your database is currently dominated by ASCII or even typical Latin-1 text, that's a reasonable tradeoff; no increase for ASCII text, a slight increase for Latin-1 text (100% on a minority of the characters in actual text; anyone have actual stats handy?), 50% increase for the rest of the 16-bit range, and the same maximum character size (U+10000 - U+1fffff take 4 bytes in both UTF-8 and UTF-16). And then you have the other advantages already mentioned: compatibility with 7-bit ASCII, NUL-terminated C strings, and ordinary 8-bit clean text channels. If you're currently in the ASCII or Latin-1 domain the question isn't even what you expect to store in the future, so much as how much cheaper disk space will be when you finally need to store it.
char ascii;
int unicode;
unicode = (int)ascii;
You are aware of the difficulties of writing on computers in Chinese aren't you? Japanese manages to avoid this as you can break any kanji down to it's sound components (hirogana), which can be spelled via the keyboard (romanji), and then select from a small list. From my understanding, Chinese is far more complex, due to how the same english equivalent spelling can be pronounced a few different ways in Chinese. The alternative is to start changing every keyboard on the planet and redo every programming language (that's in english anyway) so that it can be easily done in Chinese. Either way it would be a mess.
Well, soon we won't need that pound currency symbol either. It'll be the Euro...
Do you think a million users around the world are staring at their screens at a (hash | square | # | £ | some other symbol) and wondering what the hell we're talking about?
Note to ACs: I won't mod you up, even if you are being funny or insightful. So take a chance! It's not real life!
Because Chinese-speaking people can speak English but not many non-Chinese can speak Chinese.
Most Chinese (the majority living in P.R.China) do _not_ speak English.
Hell, most Chinese can't talk to most other Chinese due to two primary written forms (old, simplified), 31 major dialects (Mandarin now primary, Cantonese losing place) and hundreds of "minor" dialects.
(a) Given that a person has a good grasp of either simplified or traditional characters, it is normally not a major issue for them to read the other character set. (b) Mandarin (Putonghua) is the Chinese national language. People retain their spoken dialects from speaking with their parents and local community, but learn Putonghua in the PRC.
(a)has only a standardised written form, not spoken form
If you consider the standardised written form to be "simplified characters", then the standardised spoken form is Putonghua (Mandarin).
(b)that written form is especially annoying to represent digitally.
Do you mean to store, or to input the data? Both are easy. There are many popular input schemes used (based upon personal preference) and a proficient typist will have no issue with this. As for storage, I believe the most popular encoding atm (for simplified chinese) is GB2312.
c) it is a tonal language, and therefore not very easy to learn unless you have been raised from birth speaking it, since your brain won't have developed the requisite pitch analysis. There are many more non-tonal than tonal language speakers in the world, so standardising on a tonal language would place ALL of them at a disadvantage. It's easy for a tonal language speaker to go the other way though.
I've met plenty of people who have had no issues in learning Chinese due to their "non-tonal" upbringing. Hell, there seems to be plenty of Mormon missionaries walking around Chinatown speaking with near perfect Mandarin.
XML isn't a character set encoding.
Blar.
ASCII is a 7 bit standard encoding. UTF-8 uses those first seven bits EXACTLY the same as regular ASCII.
The 8th bit of every btye is used to encode higher unicode characters.
It was designed, in part by Ken Thompson, to allow unicode in existing ascii software (such as the unix family of os's) without code changes or even recompilation.
Spanish-speaking people can also speak English [1]
[1] excludes New York, Texas, California, Florida
Ha. That's funny. What you mean to say is that most latin americans living in some parts of the USA can speak English. The world is a big place you know. The vast majority of the people of this world cannot speak English.
Why don't you take a trip to China or Columbia and see how you get on only speaking English?
I'm assuming converting to Unicode would double the size and we would have to introduce some sort of compression to fit it on a CD-ROM?
8-bit Ascii, Code 156 is the pound symbol.
Does your application support multiple languages now? If it does, it probably has a default language for everything that should be present in case the specific language asked for is missing. Rather than have that be "en_us" (or whatever), make that "US English ASCII-friendly". You can then add a new language "US English Unicode". Then alter your mandate so that everything has at least that language. I'd add Unicode and ASCII flavors for all other languages too, although anything that doesn't use high chars can just be stored as ASCII with the Unicode encoding generated (if space is that much of an issue).
If your application database is not multi-lingual already, then you have some serious architechture work to do. I'd look at it from that standpoint though -- there is a wealth of reference material describing how to add language support to existing data and apps. Think of Unicode as another language.
Concentrate on these issues, and let the technical issues (such as encoding scheme) be decided after you know what you want to do. As far as that specific one goes (seems to have the most interest on this page so far), just use whatever you DBMS supports most natively.
-Richard
You're special forces then? That's great! I just love your olympics!
The issue of International addresses (city names like 'Petrakalinosorvabad') was STICKY. Further complication: conversions between ASCII and EBCDIC !! So, the whole Unicode problem is thus even further afield.
This illustrates the size of the problem with huge amounts of legacy code that runs well, is debugged, but is out of date in international markets of today. UNICODE would solve some of this with addresses being printed in the actual national language. Imagine delivering mail to Saudi Arabia with the address in (gasp) ARABIC !
Americans presume that foreign postal workers read Engligh characters.
Mainframes that speak Linux and run RDBMS's are a first step to rewriting / converting this legacy code to the new international age. There's a lot of room for better service and greater efficiency - by encouraging non-U.S. postal workers to not have to speak english and therefore deliver our packages faster!
Unicode will solve problems, but create them, too.
Unitarian Church: Freethinkers Congregate!
While I know XML is a favored silver bullet by the popular press and developers, I still haven't decided if the infatuation with a complicated packaging scheme is really worthwhile. It's nice in a sense that there are off the shelf readers that can interpret the data for you, sure, but ultimately it's still up to your code to pull out the data in a meaningful way. A good XML reader will do two things for you: 1) provide a regular format for all data, and 2) handle string conversions to and from various encoding schemes.
It seems to me quite silly to bother dealing with all sorts of encoding schemes if you can control the data from the get-go. Convert from whatever your input data is to UTF8 as early as possible. With that, you immediately have support as if you wrote everything as wide characters, but don't have to change much, if any of your code. UTF8 is narrow, with reserved codes for multi-byte encoding. UTF8 doesn't require changing your string functions* that depend on a single terminating null, and you never really have to think about the encoding again. We've migrated from ASCII to UTF8 and now support whatever languages come in as an XML input format, but we immediately convert to UTF8 and forget the XML once we hit our database.
* Caveat: Poorly encoded UTF8 can represent the same wide character in many ways. For this reason, a straight byte comparison of UTF8 strings is sometimes incorrect. Either you should test all strings at conversion time to see if they are minimally encoded, or convert to UCS2 and back again, just so all strings go through the same manipulative process, and give you the same byte stream. I learned this the hard way. With that out of the way, it's just like using normal ASCII.
Any connection between your reality and mine is purely coincidental.
Sure, Tcl isn't as buzzword compliant as Java, and it isn't popular with the "in" crowd, but it's a good workhorse, and Unicode and UTF-8 are two things it has been doing well for a while. Check it out at www.tcl-tk.net.
I've also heard it said that English is the hardest language in the world to learn, I'm not saying it it, but that's what I'm told by exchanges students and their ilk.
Just my 3.14159 cents.
This sig isn't original enough, it's time to come up with something witty...
Except the British aren't that keen on the €, are they? And both the € and the £ symbol require an "8bit clean" charset.
Unselfish actions pay back better
Don't offer an upgrade path, just require users to buy the new software and create all new apps/databases for your new software.
For old clints that dont' return a capablility level, find a backwards compatible way for them to indicate a capability level (it may simply be in the form of them doing a query in a specific form-- those that don't do it are considered to be 'ancient'.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
Just in case any of this work is being done on Microsoft Windows, you should avoid "#define UNICODE", TCHAR, and _T(). These are mainly legacy tricks used to help Windows 3.1 developers cross-compile their code for NT. Microsoft themselves doesn't use them, and insted goes with pure Unicode through the app. Even COM in Win32 since the first release of Windows 95 is all Unicode (BSTRs).
Of course, this would preclude you from using MFC, but then again, many think that avoiding it is a good thing (again, Microsoft is among those who avoid using it). But aside from other benefits, you'd end up with not needing to build two separate binaries: one for Windows NT/2K and one for Win9X.
Oh, and one other thing. If you are doing any portable code, remember that the Microsoft documentation lies and that wchar_t is not always 16-bit like they say. In fact, the spec recomends that it be 32-bit, and most other platforms (Linux included) define it thus.
I've also heard it said that English is the hardest language in the world to learn, I'm not saying it it, but that's what I'm told by exchanges students and their ilk.
Try learning German. Then you will think that English if fairly easy to learn. Even German kids have a lot of problems with all the rules.
I am a German native speaker, and frankly I like (British!) English a lot better than German.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted and ignored otherwise.
...and a swiss-german keyboards at work...
Ugh. When I saw the first of these I really had some problems using it. The next thing I did was to get an US (actually EU) one.
Why is it that most non US/EU keyboard layouts are pretty bad for programming, i.e. []{}\/ and the like are hard to type?
Most ACs are not even worth the keystrokes to insult them. Be generically insulted and ignored otherwise.
EBCDIC!
sulli
RTFJ.
There's another aspect to "going Unicode" that should be brought up, and it has a strong bearing on the business case for doing so.
It's not just for 'going international' ("We're not going to be selling in Europe, so don't bother"), or even for multilingual ("We don't care if the Spanish-speaking market in the US is big, stick to English"). It's also for cross-platform support.
Western versions of MS Windows uses CodePage 1252, which is close enough to Latin-1 for most marketing types. However, Macs use (suprise, suprise) MacRoman, and the upper half of the character set is wildly differnt. Go Unicode, and any servers or applications can easily support Windows, Linux, Unix and Macintosh data. And especially with Mac OS X being how it is, suddenly you have an extra market in the US to sell into.
Here are some of the advantages and disadvantages of UTF-8:
- UTF-8 allows you to encode any character in the entire ISO-10646 character set (which is potentially much larger than Unicode since it is a 31-bit code, rather than Unicode which is only a little over 20 bits, or 17 * 65,536 code points). This is probably not of great interest since it is not expected that the ISO character set will ever need to define any characters outside the Unicode range.
- Strings encoded in UTF-8 can be processed by standard C language routines. A binary 0 embedded in the string can be used as a string terminator just as in 1-byte character sets. Note that routines like strlen() will return the number of bytes rather than the number of characters in a string.
- UTF-8 preserves the Unicode sorting order so that string comparisons work the way you'd expect without having to convert to Unicode to do the comparison (but note that the Unicode sorting order is not likely to be a useful "language sensitive" sorting order if that matters for your application, so you may still need some way to perform that kind of sort).
- If you have an arbitrary byte in a string, it is possible to determine unambiguously whether it is the starting byte for a character, and if not you can probe backwards for the starting byte. This is not true of all multibyte character set encodings. This can be very useful for some applications and not at all for others of course.
- Characters within the ASCII range (00-7f) are transmitted unchanged.
- Most alphabetic characters (including Hebrew and Arabic characters) are transmitted with only 2 bytes - the same as if you'd stored them as UCS-2 or UTF-16, but not as compact as if you'd stored them with their corresponding ISO 8859-x character set.
- Ideographic characters and the remaining rare alphabetics within Unicode Plane 0 are transmitted with 3 bytes, which is 50% larger than if they'd been stored with UCS-2 or UTF-16 or (often) with their native computer character set like Shift-JIS.
- All other Unicode characters (mostly historical Chinese and Japanese characters and character sets for dead languages) can be transmitted in at most 4 bytes.
- Depending on your display systems, you may need transformation routines to convert to and from other formats used by those systems. For example, many printers or computer fonts that support large character sets might be arranged for use as Shift-JIS or Big5 rather than for Unicode.
- Because it preserves a certain degree of compatibility with 1-byte character streams, many existing programs and subsystems can coexist with UTF-8 with little or no modification. That does not mean you can count on UTF-8 being safe anywhere that ASCII is safe; you need to evaluate each system on its own merits. However it is quite likely to make your conversion easier.
Even if you don't use UTF-8 for the external storage format, many projects have found that its advantages make it ideal for processing data in memory. Other times using a fixed-with (16 or 32-bit format) is desirable; fortunately the conversion between UTF-8 and the fixed-width Unicode formats is quite easy and quick.Um, no. There are three major Romanization standards for Chinese, and pinyin is the standard for the PRC. It's possible to use accent marks and such to indicate pronounciation. Changing keyboards isn't really necessary - there are several input methods. Programming languages don't necessarily have to change, but it's not impossible to do so - Algol68 supported several non-English languages.
Life is a psychology experiment gone awry.
Use UTF-8 fuckwit. Why do STUPID questions get posted to slashdot?
Why don't you take a trip to China or Columbia and see how you get on only speaking English?
I don't know about China, but it's very easy to get along in Colombia while speaking only English.
It's not as easy as Belgium, and perhaps not as easy as Germany, but one who speaks only English should have no trouble taking a trip in Colombia.
There's some interesting articles on the subject, among them:
http://www.hastingsresearch.com/net/04-unicode-lim itations.shtml
It sounds like part of your system is using code pages to communicate is various languages like a web baised application. The data portions is not the linguistic text but just items that can be represented in ASCII. Some of you application can only support ASCII and all the data in your database is ASCII. If it is truly ASCII 0 - 127 (0x7F) (7 bit clean)then you data can often just redefine the database to declace that it contains UTF-8 (Unicode) data. But you must be sure that is is 7-bit clean first. Ont of the best Unicode support packages for C/C++ code (I assume that this is C) is ICU. http://oss.software.ibm.com/icu/ ICU uses UTF-16, but there is xIUA http://www.xnetinc.com/xiua/ which is also free open source software that add UTF-8 support to ICU. Even better it will allow you to add support and still run in code page first and then you the same code to support Unicode. It makes it easy to develop hybred application that may use Unicode in one part of the application and not in another. It will also allow you to use UTF-8 for database access. UTF-32 to interfece with Linux Unicode wchar_t and a mix of code page and UTF-8 requests to a browser.
E.g. we had that with two different japanese kanji encodings (on Sun workstations and Windowze boxes). Both encodings converted to Unicode and back, but they both had characters not present in the other encoding. So if you created, say, a filename on one system, converted the string to unicode and back to the other encoding on the other system, then all you got was a lot of gibberish.
So storing your data in unicode alone doesn't solve all your problems. All the clients that access that data need to support the same encodings used. (e.g. your american windowze box cannot handle unicode with kanji stuff unless you have the right language pack installed)
Essentially it boils down to: all your clients and servers must use the same encoding, wether you use unicode or something else.
Idempotent operation: Like MS software, wether you run it once or often, that doesn't make it any better.
We converted Bridge.com to unicode a couple of years ago. I don't remember all the specifics. We had to change encoding on a few characters. It wasn't that big of a deal. The only catch I remember is that for one of the Chinese translations we couldn't use Unicode for some wierd reason.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
I've found CF a bit cumbersome to use by itself. A wrapper in an OO language like C++ or Objective-C is very convenient. Your Objective-C wrapper is commonly called the Cocoa Foundation framework :)
It's been ported to Linux and FreeBSD, and I'd recommend it to anyone doing Unicode or XML work. The parser is currently non-validating, but there are so many other 'gifts' that come with CF that makes it worthwhile.
Hey, it was good enough to build an OS on.
Moderators should have to take a reading comprehension test.
He's got a point. Ever watch the Spanish channel? They use so many words they never stop speaking. In English, inflection can be used to indicate all sorts of meaning. But when you are talking 100 words a second, you speak monotone. Also, using 2 or three words of Spanish to every word in English is bad for conceptualization. Everything is 'con' or 'de' of something else. No complex word constructions necessary in science and engineering. Like the physics thing. When your language inhibits higher conceptualization, you're disadvantaged.
A good point. The lack of accent marks and similar stuff makes written English simple. The spelling is quite difficult, making it hard to learn, but it also makes it a very rich language. English has more words than any other language. Advantage? I think so. You can say more things more ways, and I like that a lot. I personally like to spell wonderful as wunderful. I think it's wonderfully wonderful, with some extra wonder on top.
This whole discussion reminds me of that (inane?) Unicode article by Hastings Research that got flamed to death here on Slashdot some time ago. Here's the link (to the article) if you're interested in reading it again.
We make b2b2c ecommerce applications and frameworks. We are currently internationalizing all our current CRM, SCM, etc applications to enable them to run in any locale. Converting all to Unicode is the smaller task, as we found out. To maintain compatibility to legacy char sets and code pages you need to update using both ways, a big hassle. Chances of running into Unicode encodings that do not have a legacy encoding is slim. ...... oh, and the 3rd party SW nightmares for webservers .... and Java is different everywhere too. C++ also has it's issues, we are moving to ICU for Java and C++, since Java doesn't cut it for internationalization.
The bigger challenges are database improvements to handle multiple concurrent languages and to have them sorted per locale. Each piece of data that gets transfered contains info about its locale, especially important when you have to handle multiple currencies at once (like with multiple supplier quotes from different countries). Not even to mention when and how the exchange rates get updated. Oh, and when does an offer expire, in the client timezone or in the server timezone?
We have to support multiple platforms - MS NT & Win2000, Solaris, AIX, HP-UX, Oracle, DB2, IE 5x, Netscape
Converting all to Unicode and maintaining compatibility with legacy installations was the easiest task of them all.
Dave
English is one of the hardest languages to learn CORRECTLY, but no one learns to use it correctly, so it doesn't matter.
English is really easy because you don't have to obey _any_ of the rules to be understood. It's just getting everything correct to impress someone that's hard.
I did try german, for two semesters anyhow. not going back unless i have to...vocabulary and phonics wasn't so bad for me, but i couldnt deal with all the gender-dependants stuff.
If you can't see the value in jet powered ants you should turn in your nerd card. - Dunbal (464142)
woof.
You accept, however, that there are major differences and that people grouped as speaking "one language" do not do so. Why did I write "Putonghua" instead of "Mandarin"? How many people besides the few pedantic linguistics like us know it? Three? Hell, they wouldn't use that one for the million-dollar question!
"Given that a person has a good grasp..."
Not quite given. China has a bit of a literacy problem outside of the demonstration/showcase cities. Almost anyone can read the "chu" char (box with piercing line meaning "through" or "centre") but reading the "xiao" character may be a bit beyond the average farmer, much less beyond the abilities of most non-Asians.
woof.
posted anon to save mods wasting their points. you're welcome.
How about converting UTF-8 to ASCII. Are there any free tools to do that?
+ absolute ASCII compatibility.
+ ASCII is a 7 bit standard encoding. UTF-8 uses
+ those first seven bits EXACTLY the same as
+ regular ASCII.
This IS informative.
&# 20006; &# 20016; &# 20026; &# 20036; &# 20046; &# 20056; &# 20066; &# 20076; &# 20086; &# 20096; &# 20106; &# 20116; &# 20126; &# 20136; &# 20146; &# 20156; &# 20166; &# 20176; &# 20186; &# 20196; &# 20206; &# 20216; &# 20226; &# 20236; &# 20246; &# 20256; &# 20266; &# 20276; &# 20286; &# 20296; &# 20306; &# 20316; &# 20326; &# 20336; &# 20346; &# 20356; &# 20366; &# 20376; &# 20386; &# 20396; &# 20406; &# 20416; &# 20426; &# 20436; &# 20446; &# 20456; &# 20466; &# 20476; &# 20486; &# 20496; &# 20506; &# 20516; &# 20526; &# 20536; &# 20546; &# 20556; &# 20566; &# 20576; &# 20586; &# 20596; &# 20606; &# 20616; &# 20626; &# 20636; &# 20646; &# 20656; &# 20666; &# 20676; &# 20686; &# 20696; &# 20706; &# 20716; &# 20726; &# 20736; &# 20746; &# 20756; &# 20766; &# 20776; &# 20786; &# 20796; &# 20806; &# 20816; &# 20826; &# 20836; &# 20846; &# 20856; &# 20866; &# 20876; &# 20886; &# 20896; &# 20906; &# 20916; &# 20926; &# 20936; &# 20946; &# 20956; &# 20966; &# 20976; &# 20986; &# 20996; &# 21006; &# 21016; &# 21026; &# 21036; &# 21046; &# 21056; &# 21066; &# 21076; &# 21086; &# 21096; &# 21106; &# 21116; &# 21126; &# 21136; &# 21146; &# 21156; &# 21166; &# 21176; &# 21186; &# 21196; &# 21206; &# 21216; &# 21226; &# 21236; &# 21246; &# 21256; &# 21266; &# 21276; &# 21286; &# 21296; &# 21306; &# 21316; &# 21326; &# 21336; &# 21346; &# 21356; &# 21366; &# 21376; &# 21386; &# 21396; &# 21406; &# 21416; &# 21426; &# 21436; &# 21446; &# 21456; &# 21466; &# 21476; &# 21486; &# 21496; &# 21506; &# 21516; &# 21526; &# 21536; &# 21546; &# 21556; &# 21566; &# 21576; &# 21586; &# 21596; &# 21606; &# 21616; &# 21626; &# 21636; &# 21646; &# 21656; &# 21666; &# 21676; &# 21686; &# 21696; &# 21706; &# 21716; &# 21726; &# 21736; &# 21746; &# 21756; &# 21766; &# 21776; &# 21786; &# 21796; &# 21806; &# 21816; &# 21826; &# 21836; &# 21846; &# 21856; &# 21866; &# 21876; &# 21886; &# 21896; &# 21906; &# 21916; &# 21926; &# 21936; &# 21946; &# 21956; &# 21966; &# 21976; &# 21986; &# 21996; &# 22006; &# 22016; &# 22026; &# 22036; &# 22046; &# 22056; &# 22066; &# 22076; &# 22086; &# 22096; &# 22106; &# 22116; &# 22126; &# 22136; &# 22146; &# 22156; &# 22166; &# 22176; &# 22186; &# 22196; &# 22206; &# 22216; &# 22226; &# 22236; &# 22246; &# 22256; &# 22266; &# 22276; &# 22286; &# 22296; &# 22306; &# 22316; &# 22326; &# 22336; &# 22346; &# 22356; &# 22366; &# 22376; &# 22386; &# 22396; &# 22406; &# 22416; &# 22426; &# 22436; &# 22446; &# 22456; &# 22466; &# 22476; &# 22486; &# 22496; &# 22506; &# 22516; &# 22526; &# 22536; &# 22546; &# 22556; &# 22566; &# 22576; &# 22586; &# 22596; &# 22606; &# 22616; &# 22626; &# 22636; &# 22646; &# 22656; &# 22666; &# 22676; &# 22686; &# 22696; &# 22706; &# 22716; &# 22726; &# 22736; &# 22746; &# 22756; &# 22766; &# 22776; &# 22786; &# 22796; &# 22806; &# 22816; &# 22826; &# 22836; &# 22846; &# 22856; &# 22866; &# 22876; &# 22886; &# 22896; &# 22906; &# 22916; &# 22926; &# 22936; &# 22946; &# 22956; &# 22966; &# 22976; &# 22986; &# 22996; &# 23006; &# 23016; &# 23026; &# 23036; &# 23046; &# 23056; &# 23066; &# 23076; &# 23086; &# 23096; &# 23106; &# 23116; &# 23126; &# 23136; &# 23146; &# 23156; &# 23166; &# 23176; &# 23186; &# 23196; &# 23206; &# 23216; &# 23226; &# 23236; &# 23246; &# 23256; &# 23266; &# 23276; &# 23286; &# 23296; &# 23306; &# 23316; &# 23326; &# 23336; &# 23346; &# 23356; &# 23366; &# 23376; &# 23386; &# 23396; &# 23406; &# 23416; &# 23426; &# 23436; &# 23446; &# 23456; &# 23466; &# 23476; &# 23486; &# 23496; &# 23506; &# 23516; &# 23526; &# 23536; &# 23546; &# 23556; &# 23566; &# 23576; &# 23586; &# 23596; &# 23606; &# 23616; &# 23626; &# 23636; &# 23646; &# 23656; &# 23666; &# 23676; &# 23686; &# 23696; &# 23706; &# 23716; &# 23726; &# 23736; &# 23746; &# 23756; &# 23766; &# 23776; &# 23786; &# 23796; &# 23806; &# 23816; &# 23826; &# 23836; &# 23846; &# 23856; &# 23866; &# 23876; &# 23886; &# 23896; &# 23906; &# 23916; &# 23926; &# 23936; &# 23946; &# 23956; &# 23966; &# 23976; &# 23986; &# 23996; &# 24006; &# 24016; &# 24026; &# 24036; &# 24046; &# 24056; &# 24066; &# 24076; &# 24086; &# 24096; &# 24106; &# 24116; &# 24126; &# 24136; &# 24146; &# 24156; &# 24166; &# 24176; &# 24186; &# 24196; &# 24206; &# 24216; &# 24226; &# 24236; &# 24246; &# 24256; &# 24266; &# 24276; &# 24286; &# 24296; &# 24306; &# 24316; &# 24326; &# 24336; &# 24346; &# 24356; &# 24366; &# 24376; &# 24386; &# 24396; &# 24406; &# 24416; &# 24426; &# 24436; &# 24446; &# 24456; &# 24466; &# 24476; &# 24486; &# 24496; &# 24506; &# 24516; &# 24526; &# 24536; &# 24546; &# 24556; &# 24566; &# 24576; &# 24586; &# 24596; &# 24606; &# 24616; &# 24626; &# 24636; &# 24646; &# 24656; &# 24666; &# 24676; &# 24686; &# 24696; &# 24706; &# 24716; &# 24726; &# 24736; &# 24746; &# 24756; &# 24766; &# 24776; &# 24786; &# 24796; &# 24806; &# 24816; &# 24826; &# 24836; &# 24846; &# 24856; &# 24866; &# 24876; &# 24886; &# 24896; &# 24906; &# 24916; &# 24926; &# 24936; &# 24946; &# 24956; &# 24966; &# 24976; &# 24986; &# 24996;
Thank you.
Why is it that most non US/EU keyboard layouts are pretty bad for programming
Oh, it must be that the evil humanists are trying to convince programmers that they should be working to make other people's life easier and not turn it to a bobdamn nightmare.
Who do you think use keyboards more, programmers or non-programmers? Hm? Do they possible need to type in ä or ö frequently?
I really hope your name included a foreign character so that you would think who's typing in your name in your paycheck.
Unicode does not solve any problems with multilingual text processing -- what it solves is not a problem (having non-iso8859-1 native language, I am qualified to testify that displaying and respresenting data in various languages wasn't a problem for at least 30 years already), and real problems -- rules, matching, hyphenation, spell checking, etc. remain problems with Unicode just like they are without it.
To make it possible to process, transfer and store the data in multiple languages one does not need Unicode -- in fact Unicode usually only adds additional step that requires some knowledge of language context that may be unknown, unavailable for some kind of processing, or simply not disclosed by end-users. What is necessary is byte-value transparency, so text in multiple languages at least will not be distorted by "too smart" procedures that cut the upper bits or make some other ASCII-centric assumptions. If/when users will care about marking languages in a way more advanced than iso 2022, they probably will find byte-value transparent channels to be suitable for whatever they will use.
However if/when real usable languages-handling infrastructure that will solve those problems will be created, it won't need unicode because it will have language metadata attached to the text already, and without metadata, text, in unicode or in native charsets, is not usable for most of applications if it's not somehow already known what language it is supposed to be in.
Contrary to the popular belief, there indeed is no God.
I know several people who do this to declare an array:
buf = (char *) malloc (size_array)
when it should be this:
buf = (char *) malloc (size_array * sizeof(char) );
Won't a switch to unicode break all the code that looks like scenario #1 above? Is there any way to switch this on a system-wide level? I guess redefining malloc as a variable-length argument function or an elaborate #define might work. I dunno, just a core dump here but I haven't seen any of the followups address this issue. Even if this isn't an issue, could somebody please followup with a solution.
Now that I'm aware that a unicode switch in the future might be inevitable I try to do the right thing, but I've seen this in soooo many places, especially old-school C hacker types.
-- Who needs windows and gates in a world w/o walls and fences?
Do you really need to update it?
I'll reply with a quote from Heinlein:
... Its very variety, subtlety, and utterly irrational, idiomatic complexity makes it possible to say things in English which simply cannot be said in any other language.
"English is the largest of the human tongues, with several times the vocabulary of the second largest language - this alone made it inevitable that English would eventually become, as it did, the lingua franca of this planet, for it is therby the richest and most flexible - despite its barbaric accretions
--- even the safest course is fraught with peril
UTF-8 and Unicode FAQ for Unix/Linux
[off-topic, but trying to make a point]
"Everyone should program in BASIC. It's the language that everyone understands."
"MS Paintbrush should be the standard, because almost everyone has it on their PC."
Etc., etc.
English is a terribly primitive language when you compare it with German, Portuguese, etc. In fact, that's why everyone can speak it, just like everyone can leanr BASIC quickly. But there are some things you just can't do right in BASIC. I speak seven languages (four fluently) and I think that fact plays a major role in my ability to program computers. Everyone should learn to speak at the very least two different languages.
As someone said, language gives shape to your thoughts and determines what you can (and cannot) think about.
That's a brilliant idea. You can't cope with the multilingual environment you've put yourself in, the craftsmen don't want headaches, and the businessmen don't want to spend money on the markets they are going to suck all the money out of. The solution: Make everyone use English. We are really solving the world's problems here.
Fucking brilliant. Especially if you consider the world only Europe and certain former British colonies...oh, and if you 'forget' that even in these countries people don't necessarily speak English.