Crowd-Source Translation Software For Free Content?

yahyamf writes "I have a lot of free educational content in the form of audio lectures and text, which I'd like to translate into as many languages as possible. I would also want to transcribe the audio and create audiobooks from the text. There are already several volunteers willing to contribute, but I need some web based software to manage all the work. Facebook is already doing something like this, but it is only for their content. I've also looked at Damned Lies, which is part of the Gnome project, but it doesn't seem to handle audio. Are there any other open source translation projects out there that I can customize and build upon?"

Comments

Question

[arizwebfoot]

Are they your lectures and who owns the copyright on the lectures? Does the university or do you? Since your work product was for hire . . .

Re:Question

[bcrowell]

Are they your lectures and who owns the copyright on the lectures? Does the university or do you? Since your work product was for hire . . .

Hold on there, cowboy. It's not that simple. In the US, work for hire status depends on three criteria, and those criteria are somewhat ambiguous as applied to university professors. Here is a more detailed discussion of the law. There isn't a clear legal precedent addressing the issue, but that's because the issue almost never comes up. The issue doesn't come up because there's a solid consensus in the world of education that the professor owns the copyright to things like lectures, textbooks, and journal articles. (Note that when it comes to articles, a journal that requires a copyright transfer asks the author, not the school, to sign it.) Regardless of the law, it's clear that there are overwhelmingly strong reasons (e.g., academic freedom) why universities know they shouldn't cross this line. It's sort of like Mia Farrow's famous remark that "you don't fuck the kids." Doesn't matter if it's theoretically legal to go there, you just don't go there.

More relevant questions to ask the OP would be (1) where we can take a look at these materials, and (2) whether he's put them under a free license such as CC-BY-SA. If the answer to #2 is no, then probably nobody will be interested in doing the translations for free.

In answer to the OP's original question, I know of two approaches that could be used. One would be to create a wiki of the English version and then allow translators to use the wiki to produce translations. Another would be to put the English version in some kind of format that's amenable to version control (e.g., plain text or latex), and use version control software such as git.

I have some experience with this because I wrote some CC-BY-SA-licensed physics textbooks, and over the years I've been contacted by roughly 10 people who were enthusiastic about translating them. None of those people ever translated any significant amount of text. It's a huge amount of work to do this kind of translation, and people's enthusiasm seems to evaporate quickly. A good example of the fragility of enthusiasm, in a slightly different context, is wikibooks, which is basically an abysmal failure, at least if you compare what it's accomplished over all the years of existence with its original stated goals, which were to revolutionize education. Writing or translating a book is just too much work for most people to tackle without some kind of financial or nonfinancial reward. It's not analogous to software, which is a functional product rather than a creative one.

Re:Question

[Anonymous Coward]

create a wiki of the English version and then allow translators to use the wiki to produce translations

You make it sound so simple. Do you know any translation-friendly wiki software? I've seen a lot of wikis, but none of them has even minimal translators' support (like edit side by side). The only available web interfaces for translations are for software or movie subtitles -- short strings, not much more than a grid of text inputs. And traditional TM software sucks, and it's not even free.

Oh no

[Dyinobal]

I've fallen behind in my web 2.0 buzz words. What the hell's a crowd source? I was thinking someone or something that draws crowds like Obama or double jointed Swedish twins. Unenlightened minds want to know!

Re:Oh no

[EdZ]

It means "post your work on the internet and get lots of other people do do it".

Re:Oh no

[Aladrin]

Outsource means to get someone 'out'side your company to do your work, usually for money.
Crowdsource means to get a 'crowd' of random people to do your work, usually for free.

Re:Oh no

[Anonymous Coward]

Isn't that called Linux?

Re:Oh no

[MBCook]

A crowd source is where crowds come from, like an apartment building.

A crowd sink is where the crowds go to, like a stadium event.

Standard EE terminology. So the poster must be looking for... a place to steal people from to force them to translate boring lectures? I'm not sure how electrical engineering applies here.

Maybe electroshocks as "encouragement" to do the work?

Re:Oh no

[telchine]

What the hell's a crowd source?

I think it might be a typo. Is crowd sauce what you might put on your crowd burger?

Re:Oh no

[mr_mischief]

You silly, this is where that "social engineering" comes into play.

You convince all these people they want to help you for free, then you sell the fruits of their labor for money.

Some see crowd sourcing as communist, but it's actually quite the opposite. It's capitalism in hyperdrive: you put up a little bit of capital, organize a whole bunch of ultra-cheap willing (so technically non-slave) labor, and you profit more from the higher margin. Those paid laborers can just keep working for those stupid enough to keep paying them.

Re:Oh no

[buchner.johannes]

A crowd source is where crowds come from, like an apartment building.

Even the stork theory is better than this... Do you tell that your children too?

Re:Oh no

[K. S. Kyosuke]

Wrong. A maternity hospital is a crowd source. A cemetery is a crowd sink. And apartment buildings connected to stadium events form oscillators.

Re:Oh no

[initialE]

Yes, other people will "doo-doo" it.

Re:Oh no

[CityZen]

An apartment building (and a stadium) is a source or a sink, depending upon its current state and the surrounding conditions.

So if the building is currently charged up and the proper conditions arrive outside of it (morning, nice weather), then the building becomes a source.

Similarly, when the building is nearly empty, other conditions (evening, bad weather) might turn things around and it becomes a sink again.

You can think of these things as capacitors.

Also, the interconnections between buildings (sidewalks, streets, transit) offer varying degrees of resistance, just like real wires do.

durrr...my head herts..

[nawcom]

http://letmegooglethatforyou.com/?q=speech+recognition+open+source&l=1

Now was that too hard?

Re:durrr...my head herts..

[Anonymous Coward]

http://letmegooglethatforyou.com/?q=speech+recognition+open+source&l=1 Now was that too hard?

Did you even read the post?

Re:durrr...my head herts..

[Kamokazi]

That's actualy a great idea. Just include this link with a copy of the English language versions:

http://www.lmgtfy.com/?q=learn+to+speak+english

Problem solved!

Open source audio translation?

[Red Flayer]

You've got to be kidding.

Currently there is software that can do parsing of speech into text not very well. Especially if you're dealing with multiple speakers, variable quality audio, etc.
Currently there is also software that can do translation of text between languages not very well. There's a reason professional translators are still in high demand (even for just written text).

You're looking for open source software that can combine both those into something effective? If you don't mind the translated audio being practically useless, then you might be able to find something.

I hate to be the bearer of bad news, but... good luck with that.

Re:Open source audio translation?

[Jurily]

You're looking for open source software that can combine both those into something effective? If you don't mind the translated audio being practically useless, then you might be able to find something.

My hovercraft is full of eels.

Re:Open source audio translation?

[fuzzyfuzzyfungus]

I'm pretty sure that the writer of TFQ is looking for software to coordinate a human speech to text effort(ie, manage volunteer accounts, serve audio clips for transcription/translation, receive results files from them, and so forth), not speech to text software.

He is, in essence, looking for an audio equivalent of the interface used by the Distributed Proofreaders project. With, perhaps, a side of translation mechanisms similar to the ones used on Ubuntu launchpad or equivalent. Neither are particular exotic technologically.

Such a setup is more or less prosaic in CS terms, no major breakthroughs need to be made; but it would constitute a somewhat specialized flavor of Content Management system. I honestly don't know if anything of the sort exists.

Re:Open source audio translation?

[MoonBuggy]

I believe you're misinterpreting the request - the translation is done by humans (the volunteers mentioned in the summary); what is needed is software to manage and coordinate the different files (text and audio), different languages and different users doing the translation.

Re:Open source audio translation?

[MrMr]

That one has been done already, now the rest of his lecture. http://www.omniglot.com/language/phrases/hovercraft.htm

Re:Open source audio translation?

[Anonymous Coward]

Do you want to go back to my place, bouncy bouncy?

Re:Open source audio translation?

[Anonymous Coward]

why is the entry tagged 'goodluckwiththat'

OLPC

[Anonymous Coward]

You can try OLPC, but they're too busy reinventing the wheel... er the GUI (Sugar) to be bothered about trivial things like "educational content"

One word:

[Anonymous Coward]

Plone.

An open source project for websites translation

[Anonymous Coward]

Hello,

At transposh we aim to create such a project, that will enable crowd-sourcing websites translations (and hence your scripts), no audio is planned though.

Currently we have a wordpress plugin, but a generic plug is being written, everyone is welcomed to help

Ofer

My "big idea"

[DriedClexler]

Here's what I think is the best way to facilitate "crowdsourced" translation: write a "semi-automatic" translator. That is, you have to spoonfeed it information about the grammatical function and meaning of all of the text, which signficantly simplifies the problem of automatically translating it. Then, you can turn over any text to crowdsourced translation. Instead of having to know two languages, all that the crowd has to know is what the text actually means, which then allows them to disambiguate it for the program.

It's relatively easy to do sanity checks too: The "clarifier" just tells it to translate into the target language and back, giving the worst (or "lowest probability") possible translation consistent with the disambiguation constraints, which tells the crowd what they need to further clarify the meaning of. Plus, you only have to do the clarification once for each text, instead of once for each target language.

I actually started developing this a few years ago, and even hired someone to develop it. (Just the interface, though, which allows you to easily mark up the text and see what you've done to it.) I had even contacted several law firms for a patent search, but strangely, all of them told me that this would would put them in a conflict of interest with a large corporate client. Too bad we haven't seen results from them...

So, who's implemented my idea already?

Re:My "big idea"

[xenocide2]

Launchpad's Rosetta tool does something different, but workable. Software usually doesn't have complete sentences as much as words or phrases sprinkled about the UI. So what they do is gather up a bunch of pre-existing translation strings and suggest them to a human for approval for a new context.

What you're proposing is basically an intermediate language, similar to how you can compile Java, Ruby, Python, Scala and so on into JVM bytecode, and let the JVM translate into the platform's specific language. So when you say "Instead of having to know two languages, all that the crowd has to know is what the text actually means, which then allows them to disambiguate it for the program," what you really mean is "Instead of having to know two language, all that crowd has to know is some intermediate representation language and the target language." Not exactly a great improvement over just using English as a lingua franca (it must make frenchmen turn in their grave to see English labeled the french language).

Re:My "big idea"

[DriedClexler]

So when you say "Instead of having to know two languages, all that the crowd has to know is what the text actually means, which then allows them to disambiguate it for the program," what you really mean is "Instead of having to know two language, all that crowd has to know is some intermediate representation language and the target language."

Whoa, not so fast! Under my idea, you don't need to know an intermediate representation language. (And you don't need to know the target language, just the source language, but I assume that's what you meant.) You just need to be able to disambiguate potential meanings to arbitrary precision, which you can certainly do if you understand it. Certainly, you have to learn how to "tag this as the verb", "pick which meaning of 'set' is being used here", "identify the antecedent of 'him' in this sentence", etc. But that's not an intermediate language, any more than knowing how to draw in PhotoShop is a language.

And it would most certainly be an improvement over English as a lingua franca. English is very difficult to learn, and everyone would have to know it. With my idea, everyone keeps their language. All that any text requires is that at least one person -- and that could be the author -- take the time to mark up the text. And the qualifications are low: you just have to know the language it's written in and learn how to use the markup program.

You seem to think the idea involves the crowd knowing the target language and making final approval. It actually involves the crowd only knowing the language it's written in. Checking the validity, like I noted, is performed by translating into the language and back. If it looks incomprehensible, you know you didn't disambiguate enough.

Re:My "big idea"

[Anonymous Coward]

The Rosetta project as part of Launchpad will hopefully be Open Sourced here in July. Maybe someone can use this as a starting point? I believe the underlying software behind it is still gettext.

Re:My "big idea"

[DriedClexler]

Okay, maybe some clarification is in order. My idea wasn't necessarily a replacement for what the submission was trying to do. It just tackles the general problem of "how to translate a lot of stuff". The idea is the program -- which has to have a special module for each target language -- PLUS crowdsourcing the easier work of marking up the source text. So you are correct that writing the program requires knowledge of the target language, and some kind of internal representation format. However, using it does not.

IMHO it's still a better idea any other I've seen for large scale translation of natural languages.

Re:My "big idea"

[Hognoxious]

"Lingua franca" means "language of the Franks", not "french[sic] language".

Just to confuse the issue further, the Franks were in fact Germans.

Re:My "big idea"

[SEWilco]

Yes, you do need to know an intermediate language. It's the language which you use to identify the exact meaning of the original. It's called transfer-based machine translation. The computer will have rules to emit facts, concepts, and actions into various languages, but you have to define the components so it understands them. You'll need to know exactly how to define that meaning to "arbitrary precision", and to do that you have to use a language (whether represented as text or graph images). You'll have to use a specific language and it doesn't matter whether the representation is "run (verb)", "run (v)", "run is a verb", "run (v):rapid motion on foot", or "run -> p(3) * 2a + node(18)b".

Re:My "big idea"

[Anonymous Coward]

That's a bad idea. Not even most people who know both languages are fit to translate. And given the sorry state of machine translation, it will make things worse because people will trust the software about the "possible translations" for the original text, most of which will be things only a computer can come up with. Have you ever tried correcting some of those google-translated Chinglish subtitles? Trust me, it's better to delete everything and start afresh -- if you happen to know Chinese. Otherwise, hands off.

Re:My "big idea"

[DriedClexler]

I guess my idea still isn't clear. The link you gave still describes, as best I can tell, fully-automatic translation which just happens to use an intermediate language to simplify the process. The article describes how the program has to infer, based on statistics and context, what each word means. That's the complete opposite of the "semi-automatic" translation I'm going for, which attempts to elimiate the need to guess by having someone who understands the source text in the source language assist the program in translating.

(Of course, my idea could certainly *make use* of known techniques to guess meaning, as that would save the user time, since they'd only have go change those that the "pre-parser" got wrong.)

Yes, you do need to know an intermediate language. It's the language which you use to identify the exact meaning of the original. ... You'll need to know exactly how to define that meaning to "arbitrary precision", and to do that you have to use a language (whether represented as text or graph images). You'll have to use a specific language and it doesn't matter whether the representation is "run (verb)", "run (v)", "run is a verb", "run (v):rapid motion on foot", or "run -> p(3) * 2a + node(18)b".

Do you need to know html to make some of the text in a document bold? People seem to make text bold in MS Word all the time without knowing html. Certainly, the programmer needs to have an internal representation language, but again, the user does not. Now, what if bold means "this is a verb"? What if short superscripts indicate the particualr meaning of the preceding word? What if arrows point to antecedents? Is the text in a "new language" now?

So yes, if you call "English text with formatting" a different language from "English text without formatting", then you're right. And likewise (as I said before) if you consider learning how to draw lines in PhotoShop "learning a new language", then you're right.

It's just that that's not how people use the term.

Google Translation Center

[jtolds]

This doesn't handle audio, nor does it seem to be up even, but this seems kind of like what you want:
http://blogoscoped.com/archive/2008-08-04-n48.html

A Helpful Comment

[WyerByter]

The people over at BOINC have a software called Bossa for distributed thinking projects (crowd sourcing). I am not sure of the current status of the project, but I have heard of at least one group that is trying to implement it.

use bugzilla

[Jessta]

soo...you need software to manage work being done by a large number of people?
Any bug tracker software will do the job.(bugzilla, tracker, etc.)

Create a bunch of bugs for the things you need done and assign them to people, people can discuss them, upload solutions and discuss those solutions, upload patches for issues, post new bugs for new required translations etc.

No need to create new software for something this simple and generic.

- Jesse McNelis

vSync tool

[Anonymous Coward]

We've built an online tool (vSync) that does exactly what you need. It's been used by the Stanford University (see http://ecorner.stanford.edu/authorMaterialInfo.html?mid=1532) and Cisco Institute.

Feel free to contact me directly at ogi (at) tunezee.com for more details.

99translations might work

[J05H]

consider using http://99translations.com/ - they have a good interface, several OSS efforts use them for internationalization and I'm pretty sure they have a "free" option. YMMV

Re:99translations might work

[Anonymous Coward]

The welcome email from 99translations:

Welcome to 99translation.com!

Our tools deliver the best expirience to software companies to find right translatior for their software and the integrate their work in their release cycle flawlessly.

If you have any questions feel free to ask them via e-mail to support@99translations.com or post to our forum at http://99translations.com/forums/4.

Our integration tools can be achieved at http://99translations.com/tools.

Feel free to post your translation requests to the web site message board: http://99translations.com/forums/1 or contact translatior that published their profiles at http://99translations.com/p.

Best Regards,
99translations.com

I especially like how they get their own domain name wrong, right at the top of the message.

Main problem with crowsourcing

[wasaty]

I'm an freelance translator and I'd like to warn you about the most serious pitfall of crowdsourcing - the quality. I've seen Facebook translation onto my language (Polish) and it's terrible. There are other projects done this way and most of them are of extremely poor quality.
Problem is - if you want quality content, you need professionals do the job. They don't necessarily have to be paid professionals (translators) - maybe just the people from your field, who wish to contribute for some reason or other. But in crowdsourcing you have to take into account a lot of poor translations and you have to introduce some form of quality control - best would be to hire editors, but maybe some kind of voting system would do.
Just don't let your content be translated without QA, because you won't sell much of it.

Re:Main problem with crowsourcing

[gdek]

Spoken like a professional translator.

Re:Main problem with crowsourcing

[westlake]

Spoken like a professional translator.

A subject poorly taught is poorly learned. You need clarity. You need consistently. A sense of style, a touch of humor.

Re:Main problem with crowsourcing

[Hognoxious]

You need clarity. You need consistently.

And, it seems, you need to know the difference between a noun and an adverb.

Re:Main problem with crowsourcing

[pjt33]

Facebook has a voting system for quality control, so it clearly demonstrates that even that isn't sufficient to get a halfway decent translation. I use Facebook in Spanish and the translations are often appalling.

Re:Main problem with crowsourcing

[Vintermann]

I think this is a consequence of self-selection for the translation job. The ones who are really fluent in English may feel less of a need for a Spanish translation.

For what it's worth, the Norwegian Facebook translation has become quite good with time.

Re:Main problem with crowsourcing

[pjt33]

I wasn't very clear in GPP. The main problem isn't so much the sense of the translation as the quality of the Spanish. The spelling is terrible and the verb tenses appear to be random.

MLM?

[American Terrorist]

"I have a lot of free educational content in the form of audio lectures and text"

Are you by any chance an Amway salesman trying to get attention?

Related: Open Source Mission

[dwheeler]

It handles texts, not audio, but Open Source Mission's Gospel Translations might be a useful model. They work with publishers/rights-holders (if any) to get the right to post works, then coordinate translations to a huge variety of languages. Once a translation is done, they post/host it for free. The translations are developed using a Wiki. Their focus is on Christian works, but I think the approach would work for any literature you want widely distributed in a variety of languages.

Re:Related: Open Source Mission

[cerberusss]

The translations are developed using a Wiki. Their focus is on Christian works, but I think the approach would work for any literature you want widely distributed in a variety of languages.

Such as lolcat-speek? From the LOLCat Bible, Genesis 1:

At start, no has lyte. An Ceiling Cat sayz, i can haz lite?

LIBRIVOX.ORG

[Anonymous Coward]

You may want to contact the folks at Librivox.org -- they're currently making audio books of the Project Gutenberg content and they have a system in place for handling the audio files, quality control --- it sounds very much like what you're looking to do. Perhaps they'd either let you use them to host projects or at least could give you pointers on how their software/processes work so that you could create something similar without completely reinventing the wheel.

From another pro translator

[zooblethorpe]

Spoken like a professional translator.

Indeed. I hope you don't mean that in a pejorative sense? When TFQ is asking about translation, it's perfectly appropriate for professionals in the field to chime in with their insights and expertise.

There was an article recently in the Japan Times about a project at the University of Tokyo to build a very similar system, though it is apparently just for texts being translated into Japanese. For the curious: http://search.japantimes.co.jp/cgi-bin/ek20090422a1.html. I don't agree with some of the pronouncements in the article (understanding the nuances of the source text and accurately conveying those in a fluently written target text does indeed take some skill, whereas the article and even the project name Minna no Honyaku suggest that 'anyone can translate!'), but the project itself looks interesting. The project site is http://trans-aid.jp/ (Japanese only).

Perhaps the TFQ submitter could contact the Professor Kyo Kageura mentioned in the article to find out more about the Minna no Honyaku system? It's basically crowdsourcing for translation projects that don't merit the time, money, and quality of professional translation, which kinda sounds like what they're looking for.

Cheers,

Transifex

[Ignacio]

One option worth looking at is Transifex. It's being very actively worked on, with the release of the next version imminent. Also, a hosted version is planned, so you eventually won't even need to maintain a server to run it on. It works with the big five FOSS VCSes, and the new version will be able to crack open tarballs as well. The Fedora Project has been using it for about a couple of years now, with great success.

Go to casting words

[Anonymous Coward]

www.castingwords.com

iCanLocalize.com

[SpzToid]

www.icanlocalize.com has some interesting offerings in the way of workflow & price, especially if you are already using a CMS like Drupal 6.

If you use Drupal for example, you can set it up so as soon as you 'publish' (or at least advance the content in the workflow-process) it is made available for professional translators to begin working on.

The pro translators have their own web-enabled interface and toolset (I think) which is similar to www.trados.com in translation memory function.

If you are a non-profit, the iCanLocalize allows your org. to use the translation interface yourself for free. Otherwise it costs .05 USD per word to use if you do it yourself. Translation in comparison is dirt cheap I think, at .07 per word. The last time I checked a few years ago, the same thing (less elegant workflow) cost .16 euro per word, in Northern Europe.

What about Mechanical Turk?

[foobarb]

See also http://www.meedan.net/

Also, Google has a translation widget that might be a reasonable stop-gap measure.

http://translate.google.com/translate_tools?hl=en

Social Translator

[Anonymous Coward]

Social Translator (http://socialtranslator.org) are already doing crowd soure translation and is becoming very popular. They are also introducing audio and API's which can hook into existing content management systems and forum systems. Check them out, they are probably exactly what your looking for.

socialtranslator.org

[Anonymous Coward]

http://socialtranslator.org is already doing this is very popular.

Speech to text

[SEWilco]

For speech-to-text, an obvious place to start is with the long-aged ViaVoice engine. If you can figure out how to buy it, as the page for that info is empty or broken.

Pootle web-based translation tool

[dwaynebailey]

You're probably looking for something like Pootle. This is used by a number of projects doing localisation including: Creative Commons, OpenOffice.org and others

It allows online translation and management of translation projects. It translates Gettext PO (for software localisation) and XLIFF (XML Localisation Interchange File Format), by using standard localisation formats it makes it easy to manage both online and offline translations. The Translate Toolkit can be used to convert various formats into PO or XLIFF for online translation.

I'm not sure exactly what you need to do in the audio, do you want to overdub or use subtitles? If you want subtitles you can use sub2po from the Translate Toolkit to convert subtitle files to Gettext PO. You'd still need to subtitle the files, then you could put those on Pootle to allow anyone to translate them into their language.

If you have documents in OpenDocument Format (ODF) then you can use the Toolkits odf2xliff converter to allow those to be translated on Pootle.

Pootle allows people to translate online or offline and they can commit their work directly to a version control system from within Pootle. This allows you to automate most of the process for you and your users.