The thing I'm sort of confused about, but maybe someone can explain why I shouldn't be, is, why does there seem to be no implication for language translation? Or is there but coverage is overwhelmed by the fascination with chatGPT?
In short, is machine language translation now a fully solved problem?
A couple years ago when I tested Google translate in a non-esoteric conversation with my Russian speaking girlfriend and, although it was useful, in terms of native fluency it failed pretty decisively.
But isn't this a much easier problem than that which chatGPT is being marketed (or at least covered in the media) as solving?
ChatGPT has been blowing every single translation task I've thrown it out of the water, even compared to other modern systems. I have no idea why more people aren't talking about that aspect of it either, other than the Anglosphere in general is kind of oblivious to things that aren't English.
For Russian, at least, sticking the article (bit by bit) into ChatGPT produces results that are broadly comparable to Bing and Google translators. It is somewhat more likely to pick words that are not direct translations, but might convey the idea better given the likely cultural background of someone speaking the language - for example, it will sometimes (but not always) replace "voodoo" with "witchcraft". However, the overall sentence structure is rather stilted and obviously non-native in places.
As others have noted, it doesn't seem to be fully language-aware outside of English. For example, if you ask it to write a poem or a song in English, it will usually make something that rhymes (or you can specifically demand that). But if you do the same for Russian, the result will not rhyme, even when specifically requested, and despite the model claiming that it does. If you ask it to explain what exactly the rhymes are, it will get increasingly nonsensical from there. I tried that after someone on HN complained about the same thing with Dutch, except they also noted that the generated text seemed like it would rhyme in English.
I wonder if that has something to do with sentence structure also being wrong. Given that English was predominant in the training corpus, I wonder if the resulting model "thinks" in English, so to speak - i.e. that some part of the resulting net is basically a translator, and the output of that is ultimately fed to the nodes that handle the correlation of tokens if you force it to talk in other languages.
I'm sure you're on the right track, regarding the % of the training corpus in English vs. other languages. It has done very well with colloquial Spanish as spoken in California, for example, which probably isn't too surprising.
What amazes me (and that you hint at) is that it still manages to pick more appropriate word/phrase choices, most of the time, even compared to dedicated translation software. I get the feeling (and I fully admit, this is just a feeling) that it's not using English, or any other language, as a pivot, but that there's some higher-dimensionality translation going on that allows it to perform as well as it does.
I worked as a translator for many years and have been following developments in machine translation closely. In my opinion, ChatGPT does represent a significant advance for machine translation. If you have the time to watch it, I made a video about the topic last week:
Hey you might this. Bilingual LLMs really are human level translators. I don't know why this frankly mindblowing fact isn't discussed or researched more but they are.
Thanks for posting that. The results do look good.
The examples are all short and from expository prose passages, though. Do you have any longer examples that include dialog, so the translator has to infer pronoun reference, the identities of speakers in conversations, and other narrative-dependent information? As I show in my video, that’s where ChatGPT is superior to Google Translate et al.—at least with Japanese to English.
That's a good point. I was just kind of randomly plowing through so i didn't pick any dialogue scene specifically. Don't think it'll fail there though.
This is a wonderful video, thank you for posting it. It had never even occurred to me to try ChatGPT for translation purposes. I wonder how well it does with slang? That's one area where all machine translate is lacking, probably because its training corpus doesn't contain it.
I think general translation is kind of solved when it comes to popular languages. Try DeepL.
I dont know how well it works for different language pairs to the languagesi know. I dont even know if deepl uses one of the newer large language models
What qualifies as popular languages in your opinion?
I use DeepL a lot as a first draft when translating stuff from Swedish (~10 million native speakers) or Dutch (~30 million native speakers) to English. While it's good enough as a starting point it regularly negates the meaning of fairly simple sentences, completely misses the use of popular idioms (often resulting in a non sequitur) and more often than not spits out grammatically incorrect nonsense for any sentence relying on implied context.
