Cultural Fitness in the Usability of U.S. Census Internet Survey in Chinese Language

Introduction

A culturally fitting survey instrument provides an appropriate tool for surveying a population of specific ethnicity because a less fitting survey instrument could result in compromised usability or comprehension, and consequently increased measurement errors. The International Organization for Standardization defines usability as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (International Organization for Standardization 1997). An Internet survey instrument is both a questionnaire and a software product. Poor usability of a survey instrument would increase the likelihood of erroneous data entries and survey break-offs, which could affect the accuracy of survey results. Three usability factors are particularly important to survey instruments: ease of comprehending survey questions, ease of entering data, and ease of navigating the instrument.

There is a body of literature that attempted to address those questions by studying the interactive and visual features and question wording of Web survey design (Couper 2008; Dillman, Smyth, and Christian 2009; Schaeffer and Dykema 2011; Sha et al. 2016; Tourangeau, Conrad, and Couper 2013). However, those studies primarily deal with Web survey design in English. There is a gap in research on linguistic and cultural aspects of non-English Web survey design and usability. Using findings from a usability study of the Chinese version of the 2016 Census Test Internet Instrument, we attempted to characterize cultural-specific and linguistic-specific usability issues and to propose recommendations for addressing those issues.

Methods

The 2016 Census Test Internet Instrument

Among an increasingly diverse U.S. population of around 330 million people in more than 140 million housing units (U.S. Census Bureau 2016), Chinese is the second most-frequently-spoken non-English language. To provide an accurate count of this population, it is being planned to have a Chinese version of Internet instrument in the 2020 Census. In 2016, the U.S. Census Bureau developed an Internet data collection instrument for the 2020 Census (2016 Census Test Internet Instrument) and incorporated Chinese translation in the instrument to make a Chinese language version available. The Chinese translation was provided by a third-party translation firm under contract with the U.S. Census Bureau. The Chinese version of the 2016 Census Test Internet Instrument followed the design of its English source instrument. In this approach, the Chinese version had the same look-and-feel and navigation as the English version.

Usability Testing

The usability testing participant completed the Chinese Internet Instrument using a laptop or tablet computer (screen size range: 7–13 inches). The participants were instructed to use their real-life information to complete the survey and to think aloud while responding to the survey. While the participant was completing the Census Test, a test administrator (TA) observed and took notes on the participant’s performance including errors, difficulties, and comments. After the completion of the survey, the TA conducted a debriefing with the participant about his or her experience with the instrument. The entire session was audio- and video-recorded.

Participants

Thirteen native Chinese speakers participated in this study in February 2016 in North Carolina, Illinois, and the greater Washington DC area. All participants were proficient with using the Internet and had access to a mobile device or laptop to complete the survey. Ten participants participated in a 60-minute session and received $40 of honorarium each and the other three a 90-minute session and $60 because they completed an additional self-assessment. Their demographic characteristics and language proficiency are presented in Table 1.

Data Analysis

We summarized and categorized observation and debriefing notes taken by the TA, in conjunction with abstraction of the audio and video recordings.

Findings and Recommendations

Sentence-Level Chinese Text Presentation

The Chinese language has different syntax from that of English and does not conjugate verbs or use tenses. However, some questions in the Chinese version of the 2016 Census Test Internet Instrument were presented in an English sentence structure, presumably, to accommodate the original software programming that was based on English grammar. Such practice resulted in inadequate Chinese sentences, and in some cases, altered meaning. We present two examples below that read natural in English but not in its Chinese translation.

Fill Placement. As shown in Table 2, the English source question had three fills: the future tense [will be], the address [4600 Some Street, Apt 101] in the middle of the sentence and the date [April 1, 2020] at the end of the sentence. The sentence structure was somewhat complex itself.

Table 2  English source question compared with untailored and tailored Chinese display.

English	Of the people who will be living at 4600 Some Street, Apt 101, who will rent the house, apartment, or mobile home on April 1, 2020?
Untailored Chinese presentation	, 4600 Some Street, Apt 101, ,
	(Of the people who will be living, 4600 Some Street, Apt 101, the house, apartment, or mobile home on April 1, 2016, who @ rent?)
Tailored Chinese presentation	4600 Some Street, Apt 101 ,
	(Those living at 4600 Some Street, Apt 101, among them on 2016 April 1, who rent this residence?)

When the Chinese translation was displayed, even though all the words making up the sentence were correct, Chinese speakers had trouble understanding this question because it was in the English language structure. As shown in Table 2, middle row, the address fill was placed right after the translation for “Of the people who will be living at” like in English, but it was grammatically incorrect in Chinese. The date was displayed in the order of month, day, and year ( 2016) that did not follow the Chinese convention of year, month, and day. In addition, in the place of the fill for the future verb tense, the @ symbol appeared in the Chinese text. Chinese verbs do not change with time and therefore do not need any fills.

A naturally sounding Chinese question would require tailored programming where the fills are inserted in the grammatically appropriate location in the Chinese sentence, the date placed in the order of year, month, and day and that the programming does not fill a future tense. The tailored Chinese text presentation is shown in the last row in Table 2.

Context Placement. In this Internet survey instrument, its landing page includes the following sentence: Welcome to the United States Census Test, to resume, please select an item from below. It was translated into Chinese: (Welcome to come to the United States Census Test. To continue, please select an item among below.) Even though all the words making up the sentence were correct, the context was incorrect and therefore did not sound natural to the participants. First, the translation for welcome was literal, , meaning welcome for your arrival at a physical place, but not welcome for your participation. To welcome participation, an appropriate wording would be . The context of participation is necessary in the Chinese translation because the word “welcome” ( ) itself does not convey the intent of this sentence. In Chinese, the reader needs to know what they are being welcomed to do. In addition, the translation of the last clause “please select an item from below” ( ) follows the English syntax but not Chinese. It placed the adverb “below” at the end of the sentence which did not sound natural in Chinese. A more natural expression would place the adverb in the middle of the sentence, like this: “ .” (Please, among the items below, make a selection.).

Word-Level Chinese Text Presentation

The convention for a Chinese person’s name places the surname first, followed by the given name. For example, if a respondent entered his or her last name in Chinese characters as [Zhang] and first name as [Dajun], the order that he or she expected to see would be [Zhang Dajun]. However, in the Chinese version of the 2016 Census Test Internet Instrument, names entered in Chinese characters by the participants were displayed as a given name followed by a surname, like this: [Dajun Zheng]. Such presentation followed the English convention, which not only looked inappropriate in Chinese but also confused the participants about whether they entered the name in the correct name fields. However, if names were collected in English, Dajun Zhang would be displayed in an appropriate order. We are aware that when writing their name in the English language, Chinese speakers in the United States generally adopt the convention of placing given name in front of the family name to avoid confusion in everyday life and that is how their name is usually displayed on their ID card. Based on these findings, we recommend that survey designers consider the data needs when collecting names. The respondents’ natural assumption of using a non-English instrument is often that non-English language responses are accepted. If Chinese characters are to be accepted for names, a culturally fit instrument would display the name in the Chinese convention, i.e., surname followed by given name, with no space in between. If names are to be collected in English only, provide explicit instructions, such as: (Please use English to fill in name).

Data Entry

In the Chinese version of the 2016 Census Test Internet Instrument, respondents were required to set up security questions. To ensure that a valid answer was entered, there was a validation check on the minimum length of the answer: at least three characters in an answer to each of the four security questions. The combination of the choices of security questions and the setting of the validation check introduced confusion to Chinese respondents. Some respondents typed an answer of fewer than three Chinese characters, for example, [Beijing], [red color]. Because these answers were interpreted as two characters by the instrument software, the validation rule did not allow the respondents to proceed. In Chinese, the names of people, locations, or items often consist of two characters.

If the validation check of the response to the security question is required for setting up the access to the website, we recommend changing the validation rule to require only one character. However, this recommendation is made with Chinese language in mind. We are not aware of English language words that would likely be answers to any of the security questions and would consist of only one character, and further testing may be needed to confirm whether the same validation rule can apply to all languages.

Navigation

During the usability evaluation, we observed participants experiencing difficulties in understanding Chinese wording used for navigation. One contributing factor was that some Chinese words used to describe those navigation actions either were not accurate for the survey context or did not fit cultural convention. Below are three examples and our recommendations for improving the usability through using Chinese translation that is in context and culturally appropriate.

Button Labels. The word “Review” was used in the English version as a label on the button which the respondent could click to go over the completed responses before submitting them. The translation used in the Chinese version, , basically meant “inspect” which was not indicative of “review” in the survey context. While completing the Census Test, participants found themselves confused with this Chinese term because it was not clear to them what type of inspection would be needed. Since there is not a single counterpart word of “review” in Chinese for this context, we recommend adding more contextual information in the Chinese wording to accurately describe the functionality of this button. One wording suggestion would be: (Review and examine what you have entered).

Headings_._ The original meaning of “household questions” heading in the English version was “the survey questions about your household.” When it was literally translated into Chinese as , its meaning became “domestic problems.” We recommend adding more context to the Chinese wording. For example, the wording could go like this, (Census questions about this household).

Help Link. The “help” link was meant to tell the respondent that one could click this link to get help on understanding survey questions. However, the Chinese wording, (survey question helps) was confusing. Our recommended wording would be just (Help), which is consistent with the common practice that we are aware of on Chinese language websites.

Discussion

In the Chinese version of the 2016 Census Test Internet Instrument, we identified cultural fitness challenges in four areas: sentence-level Chinese text presentation, word-level Chinese text presentation, data entry, and navigation. These challenges involve lexicon, syntax, and semantics. By examining wording and syntax structure in the Chinese version of the instrument and the programming algorithm, it appears that the problems originated from applying English language conventions to the Chinese instrument. For two languages with fundamental linguistic differences like Chinese and English, this approach of filling Chinese characters into English sentence structures often results in sentences in Chinese characters but not in the Chinese language. To create a survey instrument in authentic Chinese language that respondents can understand, it is necessary to apply Chinese cultural and linguistic convention in survey instrument programming. Here are two examples that may result in unnatural, grammatically incorrect, or inaccurate Chinese language text: coding fills that are based on English sentence structure and translating navigational text without considering how it will be used in the survey context. In addition, the Chinese language does not conjugate verbs or use tenses. Fills designed to indicate past, present, or future tenses are not always necessary. Therefore, the use of fills or the location of the fills should be determined by conventions in the Chinese language.

Disclaimer

Any views expressed are those of the author(s) and not necessarily those of the U.S. Census Bureau.

Acknowledgement

The authors thank Kuopei White, Gianna Dusch, Joanne Pascale, and Paul Beatty for their valuable comments, and many colleagues at U.S. Census Bureau and RTI International including Patrick Hsieh, Jiyoung Son, Hyunjoo Park, and the test administrators for their contributions to this project.

Conflict of interest

The authors declare the absence of any conflict of interest.