Slashdot Mirror

I can't help but notice that there's nothing mentioned about unicode. I don't see how a major web development language, especially one made by Japanese designers, can go so long without adding comprehensive unicode support.

What's up is that there's massive disagreement on whether Ruby's standard strings and characters should become Unicode. There are quite a few people used to the old world where a string was interchangeable with a vector of 8 bit bytes, and they don't want to let go. To add to the problems, Unicode is controversial in Japan. So, expect Unicode to continue to be painful and inconsistent in Ruby even after the next major release.

everything's just a bunch of bytes.

Or, to put it another way, no, it still doesn't.


  the opinion that "the best and most common way to use characters of other languages is UTF-8" is not universally held. It might be the most common, but many learned and respected individuals think it's not the best.


Indeed. For example, I prefer UTF-16 for most purposes, and I see that the MS and Java worlds agree with me; UTF-8 is primarily for Ugly Old Unix compatibility.

Unfortunately, Ruby's issue is not so much with UTF-8 as an encoding as with a perception that Unicode as a character set is insufficiently Japanese.

I know I have plugged this once before in a Ruby discussion, but anyway, my thoughts on attitudes to Unicode in Japan.

At the risk of plugging my own hopefully interesting page on the subject, I think this is one of the few places left in IT where the Japanese dislike of Unicode is still causing a problem in modern IT.

Yes, Ruby and it's author have an interesting attitude to string representation in general and Unicode in particular. It's partly what inspired me to write this:

Psychology of Unicode in Japan

It's really been a very interesting struggle between people's psychological and I.T. needs -- a struggle that's pretty well over now, but has left behind things like Ruby's way of doing things.

I used to say just what you are saying. In fact, I used to go round going "But don't you see, character X and character Y are seen as TWO SEPERATE CHARACTERS! You can't force people to use one!". And of course it's true that Han unification is a difficult job that can never be done perfectly.

But, it's a difficult job that Chinese, Taiwanese and Korean representatives were all able to see the need and true meaning of, and co-operate to do it well. The Japanese input into the discussion was clouded by their internal politics (the 'let's all speak Esperanto and ban kanji!' brigade versus the 'bring back Yukio Mishima!' brigade) and by bureaucratic inertia.

I actually did quite a lot of research into this sad bit of computing history, and the more I learned the more I realized that Western insensitivity hadn't been the real problem. I summarized the results here:

http://www.jbrowse.com/text/unij.html

(It's kind of aimed at people without much background knowledge. It also contains mistakes and simplifications.)

Sakamura is a special case, though -- I think his disregard for facts in favor of nationalistic grandstanding (and I don't just mean Unicode, although it's a good example) do indeed justify the term 'nutcase'. The fact that people like him formed a significant part of Japan's outward-facing IT organization is regrettable... but the MOF is still worse overall :)

Parent has it right. The Advanced Hex Editor (AXE) has this functionality. Lots of fun when looking at uncompressed graphic formats like icons stored in executables. :)

Grab a few megs from the start of the disk and use sox, the sond exchange to tack audio headers onto it, and try various codec conversions, endian swaps, etc.

There's every chance that the audio chip was interfaced to the drive very simply, as you theorized, without a filesystem. I'm aware of a product which lets you access an ATA device via RS232, it's called the StampDrive. As far as I can tell, it's a PICmicro that's been taught a basic subset of the ATA spec, and it acts as a storage broker for any device that can speak async serial.

People who build their own dataloggers have lots of experience with this sort of dirt-cheap interfacing. Your audio bug is, after all, just a specialized datalogger. A few minutes with a search engine should find plenty of info on the subject.

Post back with any success stories. :)

I think Axe (http://www.jbrowse.com/products/axe/screenshots.s html ) can do something like that.

I really wish I could switch to Mozilla (ok, Firefox). My co-workers are switching to Firefox. My users are switching to firefox. But I can't, because I have no idea how to implement my pet project as a mozilla-type plugin.

All it has to do is read in a dictionary file, then catch the 'new page loading' event, perform morphological analysis on the page, and edit the page as it loads to include ruby tags and/or something to display definitions in the toolbar. That's it! It's fairly computationally intensive and sometimes the right html to insert at a given point is a bit of a guessing game, but it's not rocket science. But HOW THE FORK DO I DO IT IN MOZILLA??

PS Yes I have rtfm and no I cannot implement the analysis algorithm usefully in javascript and yes I do have to insert ruby tags, as well as regular javascript that talks back to the plugin, into the page on the fly.

Considering the amount of research that seemed necessary to get it working in the minefield of IE, I expected that I would be quite capable of figuring it out in mozilla, but it just seems to be an order of magnitude harder.

I would be grateful for advice (eg a pointer to a similar project). Or failing that, remarks on the lines of 'if u cant use mozilla u r lame u lame wind0z3 lu20r hehe l8trz' would also be fine.

(plug)

To sort out these and other common misconceptoins about what Unicode is and does, why not refer to my Unicode Tutorial?

(end plug)

(repeat plug in .sig)

May I offer this guide to all things unicode:

Unicode terms, FAQs, and mistakes?

It helps clear up confusion between things like 'character sets' and 'encodings' and 'code points'.