Translate with Accuracy: How Content Testing Can Improve Your Product Terminology

i18n UXR - Part 2

Jan 28

Don't let a bad translation ruin your global expansion plans. Use Content Testing to ensure your localized product terminology is clear and tailored to your target audience's needs.

Importance of Content Testing

As companies expand to global markets and introduce new, innovative products, they face the challenge of effectively communicating their brand, messaging and product features to international audiences. One key aspect of this is the translation of key product terminology. The words used to describe a product can greatly impact how well the customer understands and interacts with it. The wrong translation can lead to confusion, frustration, and even mistrust.

So, how do we determine the best translation for such important product terminology? A/B testing can be useful, however, it may not be the most suitable method prior to product release. This is where Content Testing for translations comes into play.

Content testing is a thorough evaluation of the quality and effectiveness of the content on a website or application. By utilizing techniques such as 1:1 interviews, and surveys, content testing ensures that the content is clear, user-friendly, and tailored to the needs of the target audience. While these methods are traditionally used during product development in the source language, they can also be applied to localized content.

Real-World Example: “Guardian”

Let's take a specific example to illustrate the importance of accurate translations for product terminology, using the term "guardian".

In the context of a product like ByteDance’s TikTok or Facebook's Messenger Kids, it might refer to a parent or legal guardian. But in the context of virtual reality headsets like Meta Quest or Pico, it might refer to a system that helps users navigate their physical environment safely while using the headset.

As you can see, the way these terms are translated can make all the difference in terms of customer comprehension, brand alignment, and target audience engagement.

Content Testing Methods

In this section, we will tackle some of the most commonly used methods for content testing:

Text Highlighting
Stack Ranking & Word Comparison
Rating / Likert Scale
Word Association
Open-Ended Questioning
In-Context Test

These methods can help assess the accuracy and success of language translations and ensure they are meeting their intended goals. Regardless of the method always provide context for the participants, e.g. via clear definitions of the terms being translated or visuals.

1) Text Highlighting

In a text highlighting test participants are asked to evaluate translations based on pre-defined objectives or criteria, using green and red markers to indicate their findings. Green markers represent passages that meet the criteria, while red markers indicate areas where the translation falls short.

Some common objectives for text highlighting tests include ensuring that the text is easy to understand, matches language style expectations, and supports confidence in a product or feature. The results of the test provide valuable insights into the emotional response to the translations, helping to identify areas for improvement and ensuring that the intended message is effectively conveyed to the intended audience.

2) Stack Ranking & Word Comparison

The Stack Ranking test is a qualitative test, where participants are asked to rank different translations of a term or phrase in order of their preference.

The test can be applied to both smaller sets, with as few as 2 items, and larger sets, such as 5 or 7 items. When working with just 2 options, it is recommended to present the test as a comparison rather than a ranking.

Example

Rank these [German] translations for "controller" by personal preference:
A) Steuergerät, B) Controller, C) Kontroller, D) Bedienungseinheit, E) Steuerung.

It is important to note that the results of a Stack Ranking test are subjective as based on participant preferences, which can be influenced by a variety of factors including cultural background, language proficiency, and personal bias. Therefore, it is essential to carefully consider the limitations of this method and to ensure a large enough sample size to accurately represent the target audience.

3) Rating Scale

The Rating Scale test is a method to evaluate the accuracy, relevance, perceived sentiment, and other vital characteristics of translations. Participants are presented with various translations and asked to provide a rating using a Likert Scale.

Here is an example to probe for the accuracy of a translation:

Completely Inaccurate
Mostly Inaccurate
Neutral
Mostly Accurate
Completely Accurate

This allows for the gathering of numerical data that can be used to compare the different translations and understand which is most preferred by the participants.

Understanding Tone & Style Preferences

You may conduct customer perception tests to assess the tone and style of your content. Such assessments not only aid the translation in question but also provide valuable insights into your overall approach. Samples:

Empathy: Strongly Disinterested, […], Neutral, […], Strongly Empathetic
Personalization: Very Impersonal, […], Neutral, […], Very Personal
Sentiment: Strongly Negative, […], Neutral, […], Strongly Positive
Informal/Formal Tone:
- Very Formal, Formal, Neutral, Casual, Very Casual
- Stiff, Proper, Neutral, Relaxed, Lively

It is important to compare the results of these assessments with your objectives. Ensure that the perceived tone aligns with your intended message.

Additionally, consider soliciting feedback from customers on their preferred tone and evaluate the gap between your current language and the customer's preference. For example, if your goal is to maintain a very formal tone but the customer perceives it as too rigid and prefers a more informal approach, consider revising your language to better meet their needs.

4) Word Association

Word Association is a technique used to evaluate participants' perceptions and associations with a specific term. Participants are asked to list the first words that come to mind when they see or hear the term. It provides insights into participants' thoughts and experiences and can reveal biases or preconceptions. However, participants may not respond consistently, their responses may be affected by their background, culture, or experiences, and the method only provides a snapshot of their immediate thoughts. The results are subjective and rely on participants' own interpretations and personal associations, which may not be representative or accurate.

5) Open-Ended Questioning

In Open-Ended Questioning, we invite participants to provide their own translation for a specific term or to describe the meaning of that term as they understand it. If participants encounter difficulty in answering, providing examples of how other participants have responded to the question may be helpful. It should be noted that the open-ended nature of this method can make it challenging to compare responses across participants, as each individual is able to provide their own interpretation.

6) In-Context Test

During an In-Context Test, participants will be engaged with a simulated version of the product or website, evaluating the functionality and suitability of the translations within the intended use case. Provide participants with a defined set of tasks or allow for self-directed exploration.

In addition to monitoring the participants' behavior, gain an understanding of their experience by asking a few questions after the test. The following questions serve as a reference:

Can you describe your experience using the product/website in this scenario?
Did you encounter any confusing or unclear translations?
Did you understand the meaning of [term] in this context?
Was the translation of [term] appropriate for this usage scenario?
Would you have preferred a different translation for [term] in this context?

Note that this approach may require collaboration with the engineering team, leading to increased costs and extended timelines.

Execution Settings

The execution of Content Tests for Translations can occur in a range of settings. This post will concentrate on:

Surveys
1:1 Interviews
Focus Groups

1) Surveys

The participants are requested to complete a survey that inquires about their thoughts and feelings regarding the copy, such as ease of understanding, emotional impact, and others. These surveys may incorporate various Content Testing methods, including Stack Ranking, Open-Ended Questions, Rating Scales, or Word Associations. It is not advisable to employ Text Highlighting through this method. When executing In-Context Tests consider utilizing a specialized services like Userzoom (see “Tip: Userzoom” below).

Advantages: Broader sample size, offers quantifiable data.
Disadvantages: Limited depth, lacks qualitative context.

2) 1:1 Interviews

Individual participants are interviewed in a private setting, either in person or through remote sessions, to facilitate an open expression of their thoughts and perceptions regarding the translations without external influence. These interviews afford the opportunity to employ all listed Content Testing methods.

Advantages: Offers in-depth insights, and facilitates prompt clarifying questions.
Disadvantages: Resource and time-intensive, limited sample size, may necessitate a dedicated physical space.

3) Focus Groups

A group of participants is brought together and facilitated by a moderator to engage in a discussion and provide feedback on the copy. This approach provides a cost-effective and efficient means of gathering feedback from a larger participant sample in a relatively shorter time. Suitable methods for Focus Group settings include Stack Ranking, Word Comparison, and Rating Scales. Open-ended questions and word associations may also be useful, but they may be more difficult to execute and to be analyzed post-completion.

Advantages: Facilitates discussion, cost-effective, modestly larger sample size.
Disadvantages: Limited sample size compared to surveys, reduced depth, and group setting may affect results.

Roles and Responsibilities

The effective implementation of a content testing project requires the active participation of the following roles:

Language Owners
Content Owners
UX Researchers
Localization Program Managers.

1) Language Owners

The language owners (e.g. language managers or linguists) are an integral part of the content testing process. They research possible translations, interpret user feedback, and provide cultural and linguistic context to the team. They also assess study results to ensure that the translations align with the target audience and brand. They ultimately decide on the best translations.

2) Content Owners

Content owners - including roles such as content designers, engineers, and product designers - are responsible for providing context on the source copy, product functionality, goals, and intents to inform the translation and testing process. In addition, they may create mockups, alternative source language, and other materials for testing, and work in conjunction with linguists to evaluate the study results.

3) UX Researcher

The expertise of user experience research managers can greatly enhance the content testing process. They provide leadership and guidance to the team by utilizing research methods such as 1:1 interviews, surveys, or behavioral observations. They also impart their knowledge to other team members, enabling them to execute studies independently. If necessary, they coordinate with external partners and vendors to facilitate the research process, including sourcing participants. They are involved in evaluating study results. Furthermore, they may integrate content testing studies into other related UX projects to increase efficiency and optimize the overall research process.

4) Localization Program Manager

The localization program manager is responsible for overseeing the content testing project, from start to finish. They coordinate the efforts of different languages and roles and are the primary point of contact for stakeholders. They ensure that the project is completed on time and within budget, and work closely with engineering and design teams to implement product changes following the study.

Best Practices for Selecting Translation Options

When determining the appropriate variations for translations, it is important to adopt a strategic and thoughtful approach:

Establish clear objectives:
Define the clear objectives and purpose behind your chosen option. Analyze the sought-after tone and categorize your options as appropriate. Example categories: Persuasive or welcoming? Contemporary or conservative?
Diversify your approach:
Ensure your translations are diverse to gain a deeper understanding of the customer's tone and style preferences. In addition to carefully selecting the most suitable translation that is in line with cultural and linguistic context, it is recommended to experiment with variations that deviate from the norm. A monotonous approach to tone and style can undermine the significance of the data you collect.
Keep the length consistent:
Ensure that all variations have around the same length and structure so that the only variable is the copy.

Sample: “Buy for a friend”

The following presents a range of tonal options for the phrase "Buy for a friend", including a neutral tone, persuasive tone, welcoming tone, and motivational tone. Each option comes with its corresponding German and English translations. Using a variety of contrasting options will help us to understand better which tone is preferred by the audience we're trying to reach.

Neutral tone: 
- DE: Für einen Freund kaufen
- EN: Buy for a friend
Persuasive tone:
- DE: Schenke heute eine Überraschung!
- EN: Gift a thoughtful surprise today!
Welcoming tone: 
- DE: Schenke einem geliebten Menschen Freude.
- EN: Gift joy to a loved one.
Motivational tone: 
- DE: Verbreite Begeisterung mit einem Geschenk.
- EN: Spread positivity with a present.

Executing a Test: Step-by-Step Guide

Define the test goals
Identify the specific content elements of the product or website that require an evaluation and determine the linguistic objectives, such as targeting a specific demographic or prioritizing comprehension over style or brand.
Recruit participants
Recruit native speakers of the target languages as participants for the test. You should strive to have a representative sample of users in terms of demographics, technical proficiency, and other relevant characteristics.
Prepare test materials
- 1:1 / Focus Group:
  Prepare a script that outlines the tasks that participants will be asked to complete, as well as any instructions or materials that they will need.
- Survey:
  Create a detailed script outlining the tasks and instructions for participants, as well as any additional materials they may need to complete the study. It is recommended to utilize a user-friendly and easily accessible format, such as a microsite or PDF. Utilize survey creation tools, such as Google Forms or Qualtrics, to transform the script questions into a comprehensive survey format.
Conduct the test
- 1:1 / Focus Group:
  Have participants complete the tasks while being observed and recorded. For in-context testing, using testing software can help you to capture screens, videos, and audio from the participant and device.
- Survey:
  Administer the survey to the selected participants using the tools noted above.
Analyze the data
Once the test is completed, analyze the data collected during the test to identify preferred translations or any issues or problems that originate from localized content and UX.
Report findings
Document your findings in a report that includes an overview of the test goals, participants, tasks, and findings, as well as recommendations for improvements.
Iterate
Based on the findings and feedback, consider implementing changes and testing again with a new group of participants or with the same group.

Tip: Userzoom

Consider utilizing specialized software and services, such as Userzoom. They offer help to execute in-context and usability testing, conduct interviews, or administer surveys. They may also offer advanced methods like tree testing or card sorting. They can also support optimizing the participant recruitment process by providing access to a panel of pre-qualified participants, which can help to ensure that research studies are conducted with a representative sample of users.

Conclusion

In conclusion, content testing for translations is crucial for ensuring that key product terminology is translated accurately and effectively for international audiences.

By utilizing techniques such as text highlighting, stack ranking, or open-ended questioning, content testing can provide insight into the quality and effectiveness of localized content.

It's important to have a team with different roles, such as language owners, content owners, UX researchers, and localization program managers, to run an effective content testing project for translations.

Different execution settings, such as surveys, 1:1 interviews, and focus groups, can be used to gather valuable feedback from participants, but it's important to consider the trade-offs of each method in terms of time and resources required, sample size, and level of depth.

By implementing content testing for translations, companies can ensure that their products are well-understood and well-received by global audiences, ultimately leading to increased customer satisfaction and brand loyalty.

Developing Strong Working Relations between UXR and Internationalization Teams

Product Localization Frameworks