What are the weaknesses of ChatGPT?

 For some aspects, ChatGPT does not offer any reliability:

1. He doesn't tell the truth. ChatGPT says what people want to hear from the information it has.
   
   
2.  He only knows a compartmentalized version of what people say about the real world, using that information to create compelling language to convey what he's learned.
3. He can't count. I tried to replicate Jonas Degrave's simulation at Engraved using a more complicated calculation and got the wrong result. It correctly tries to multiply the two numbers from this Python command that it knows how to simulate, which is quite remarkable. However, he simply does not know how to count.
   
   
4. He can't think. He often writes relevant descriptions and claims about what I asked of him and confidently misapplies them. He cannot reason; it is not a finite automaton (FSM).
   
   
5. His false modesty is somewhat embarrassing. ChatGPT writes with aplomb and authority, but shyly tells you he's only here to help, then embarrassingly apologizes when you tell him he was wrong.

What are the strengths of ChatGPT?

Having understood what ChatGPT is not or cannot do, let's now see what it can do as an interpreter and text generator. ChatGPT skills:

• He writes better than you. I gained this belief from the many conversations I had with ChatGPT. He writes with different levels of complexity and a very extensive vocabulary. 
  
  
• He follows instructions. He perfectly modifies the texts according to specific instructions, both in form and in content. More importantly, it maintains conversational context and understands your references to its previous actions, even in the most vernacular.
  
  
• It modifies the text while keeping the meaning. Whatever the text, ChatGPT makes changes in content, form, and style by applying your guidelines. It preserves the semantic content of the text or modifies it at your request.
  
  
• It manages multilingual terminology, one of the great challenges of translation. I don't know yet if this would be a realistic solution for a large-scale translation. However, I was able to see the effectiveness of ChatGPT: it introduces specific terminology when correcting previously translated documents, even when they are not original translations.
  
  
• It detects offensive terms. I submitted to ChatGPT excerpts of a pleading including racist and homophobic text messages, in the context of a federal criminal case, then I asked him to identify any offensive passages. He did a great job and was able to explain his reasoning.
  
  
• It can detect entities. I gave ChatGPT a typical task: to detect entities and surround them with tags. He omitted a few. With a few extra helpers, however, he easily identified them.
  
  
• It performs classifications according to taxonomic levels. One of ChatGPT's most amazing abilities is its ability to apply general knowledge to a specific situation.

As mentioned, ChatGPT's ability to state true facts or know what is correct is unreliable. (This means that content creators need to check that it's not talking nonsense.) However, once we have text containing a meaning that suits us, ChatGPT can manipulate or transform its form and content, while maintaining its intrinsic meaning. Let's take a closer look at the panorama of localization activities and how ChatGPT could change our current activities. Benefits of this generative AI tool:

• He is very good at translating. For languages ​​with a large corpus, ChatGPT rivals leading engines in producing out-of-the-box machine translation, even superior in some respects.
  
  
• He follows the terminological instructions with some efficiency.
  
  
• He can apply stylistic, specific, or general instructions. Conversely, machine translation engines struggle to perform this type of task correctly.
  
  
• It is strong enough to categorize things, especially at the text level, according to arbitrary taxonomies. This capability is an advantage for the localization industry, as it allows different specific instructions to be applied depending on the case.
  
  
• He excels in the revision of given texts, the keystone of quality localization. It performs quite well in the four main components of text editing and translation.
  
  
• It helps in content analysis, examining text for effective processing, improvement, or ROI. It can help anticipate or predict translation quality issues, adjust it to optimize its results in terms of reach, search engine optimization (SEO), and CTA/CTR performance, as well as improve the readability both at source and target level, among others.
  
  
• It helps to write and modify working code. Top programmers question ChatGPT's ability to create code at their level, and it seems that it doesn't. However, when I asked him to write XML content extraction code, he produced a Working Code which I managed to run. This makes code creation (and inherent learning) easier for non-coders.

ChatGPT will define new skills and practices

I started to understand the type of Prompts required by ChatGPT. This special technology only accepts natural language as input. To use this technology in production, we need to get better at building efficient Prompts. Specific content transformations will likely involve successive Prompts to perform each of the different tasks: cleanup, pre-and post-processing, etc. Learning how to use natural language Prompts as part of our automation workflows in a way that is both relevant and predictable enough in the results is going to be exciting.

And now?

We know we can no longer ignore this new generative AI. It will probably disrupt our sector. We must therefore take the tangent of this evolution toward the automation of languages. ChatGPT can transform and annotate text with a quality equivalent to a standard-skilled human reviewer, and even perform these tasks more efficiently. For this, he relies on various skills that hardly any human being has, and he can apply his knowledge to new situations.

More importantly, it shows the potential to solve some lingering location automation issues. Of course, it's one thing to have a conversation with him using fictional examples. It's another thing to imagine using it on a large scale to execute those actions. Going forward, we must:

• perform real-world, large-scale tests to assess error rates for each type of localization and review the task studied here;
  
  
• analyze the detailed journeys of macro and micro users within the localization value chains and identify the risks of disruptions induced by this type of test automation;
  
  
• understand how to solicit and provide relevant context to ChatGPT at scale, and document pitfalls and best practices;
  
  
• implement new automation and human review workflows, projecting what post-editing and QA will mean shortly with such AI in the process;
  
  
• design new automation and user experience (UX) interaction contexts for translators and clients at every possible improvement opportunity;
  
  
• ensure that the investment in licenses, deployment costs, and maintenance makes sense for our business.

Some thoughts on the language and practical use of ChatGPT

One of the most striking things about ChatGPT is that when faced with complex numerical operations, it gets almost correct but still wrong results. He doesn't cheat. He really learns everything about the language he is trained in. The fact that it almost manages to find the correct result of an operation beyond a certain order of magnitude (and the correct result for smaller numbers) tells us that a linguistic corpus of sufficient scale contains statistically significant insights into the real world. It also shows that dedicated formal systems (such as mathematics) are needed to produce relevant, reliable, and accurate information about the real world.

ChatGPT reminds us that a self-referential and self-consistent system cannot, by itself, carry the truth of the world, which exists independently. This echoes Gödel's incompleteness theorem. As conscious beings, we cannot detach our cognition from the formal and material systems that ground our understanding of the world in a reality that imposes itself on us and that we cannot define by language alone.