A Tutorial on Translation Quality Evaluation

August 2, 2025 11:25 pm

Who is This Tutorial For?

The intended audience for this tutorial is members of production environments for language services who are not necessarily professional translators, but are interested in learning about the processes and tools involved in evaluating finished translations. We include links to one possible system of tools that can be used to this end. The initial intended audience for this tutorial is the machine translation community at the 20th Machine Translation Summit in Geneva, Switzerland, taking place in June 2025.

Part A of this tutorial will aid anyone in understanding the basic principles of TQE and different roles in a TQE system (in particular the project manager’s and the technician’s) via a guided, concrete example, even if they will never fill the role of evaluator. Part B enables readers to contribute to TQE systems by preparing tools, specifications, and metrics for evaluators, and by producing quality ratings based on the error data received from evaluators.

After following this tutorial, a non-translator who wishes to conduct a TQE must find a professional translator to at least fill the role of evaluator. If the reader considers themselves already well-versed in the principles and theory behind conducting a TQE, they may proceed directly to Part B, referring to Part A as needed for examples.

Part A: TQE Basics with a Concrete Example

Although there are many methods with which to evaluate the quality of translations, this tutorial is focused on conducting a TQE (Translation Quality Evaluation) based on MQM (Multidimensional Quality Metrics). It consists of the following sections:

  1. What is Translation Quality (TQ)?
  2. Roles in TQE
  3. What is TQE?
  4. An introduction to a concrete example for Part A
  5. A system of tools for all steps in a TQE, applied to the concrete example

Note that this tutorial will provide information for using MQM in a TQE, but a full description of MQM of is available at the MQM website.

Part B: Expanded Theory & Your Own TQE

This section shows how to conduct a TQE on a text of the reader’s choosing by putting together the same steps and tools as in Section A5. It explains not only how to make each of the files, but also why each is necessary from a theoretical standpoint, and how the tools used to generate the files apply that theory. A technician’s view of each tool in this tutorial is at TQE Tools.

The subsections of Part B are analogous to the structure of Section A5:

  1. Preliminary Stage
    1. Formalizing Specifications
    2. Selecting a TQE Metric
    3. Preparing the Document for Annotation
  2. Error-Annotation Stage
  3. Automatic Calculation & Follow-Up Stage
Next: A1 What is Translation Quality (TQ)? >