Many multi-nation surveys use samples that are not representative of the nations compared. Examples include surveys of students or families of students in different nations (M. A. Straus 1968; J. H. Straus and Straus 1968), of nurses (Glazer and Gyurak 2008), and of residents of the largest city in each nation in the study (Lynch 2008). None of these studies used nationally representative samples, yet they draw conclusions about differences between nations. This article provides empirical data on whether valid cross-national conclusions can be made using these types of unrepresentative samples when the samples are drawn from the same subpopulations in each country and using similar study designs. It reports empirical tests of the cross-national validity of data obtained for a 32-nation study using convenience samples of university students – the International Dating Violence Study or IDVS.
The IDVS sample is not representative of the 32 nations because university students differ from the general population in important ways. In addition, because the sample consisted of students in classes taught by the member of a research consortium in that nation and in classes where other teachers permitted the questionnaire to be administered, the students in IDVS are not random samples of students in their nation, or even of students in their own university. Members of the consortium conducting the study administered the study questionnaire in enough of their own and classes taught by others.
Concurrent validity of nation-level measures based on non-probability samples is measured by the degree to which variables from the non-probability samples are correlated with measures of the same variable based on presumably representative national statistics. The degree of validity is indicated by the size of the correlation. Table 1 summarizes the results for 18 concurrent validity tests. Because of space limitation, only the first can be discussed here. More complete analyses, including evidence on “construct validity” (Campbell and Fiske 1959) can be found in Straus (2009).
Dominance of the male partner figures prominently in all theories of partner violence (M. A. Straus 2008). Therefore, it is fortunate that a test of the concurrent validity of the Dominance scale in the IDVS (Straus et al. 1999 (Revised 2007)) was possible because a widely respected measure of a closely related concept is available for 29 of the 32 nations in the IDVS: the Gender Empowerment Index in the Human Development Report (United Nations Development Programme 2005). The standardized regression coefficient is –0.69 (p<0.01) in the first row of Table 1 and the scatter plot show that the more Gender Empowerment, the lower the Dominance score of the men in the IDVS sample. For example, Tanzania has the lowest Gender Empowerment score and the highest Dominance score, and Sweden has the highest Gender Empowerment score and the lowest IDVS Dominance scale score. This correlation is remarkably high given that one is the report of male students about their dating relationships and the other is a compilation of national statistics, such as the female percent of seats in parliament; female proportions of senior officials, managers, professional and technical; and the ratio of female to male earnings.
This paper reports on tests of the cross-national concurrent validity of 18 variables measured by a 32 nation study using unrepresentative convenience samples. All 18 analyses found correlations with measures of the same or analogous variables measured using representative samples and different methods of measuring the same variable. The mean correlation of 0.51 is as high or higher than is typically found for correlations between different measures of the same construct within the same sample (Cronbach 1990). Given that these results are for correlations between different samples, one of which is a non-representative convenience sample, and given the major differences in the indicators used to measure the same variable in IDVS and in the nationally representative data, it can be concluded that an average correlation of 0.51 indicates a high degree of concurrent validity.
The National Context Effect
I suggest that the results in this article reflect what can be called a national context effect. This is the idea that each national context exerts an influence on residents of that nation, regardless of whether they are farmers, doctors, or university students. Thus, the students in each of the 32 nations, even though they are not representative of their nation, were mostly born and socialized, and are living their lives in one of the different national social contexts. The differences between nations based on student data parallel differences between nations in behavior and beliefs of representative samples from those nations.
The results reported indicate that valid conclusions about differences between nations can be reached using unrepresentative convenience samples within each nation, provided a comparable unrepresentative population sector is studied in each nation and similar data collection protocols are used. In this study the unrepresentative samples consisted of university students, but it probably could be samples of many other sectors, such as construction workers, residents of small villages, or residents of the largest city in each nation.
Although this article has provided empirical evidence on the validity of cross-national results based on unrepresentative convenience samples, cautions are needed. For example, although the same unrepresentative sector may be sampled in each of the nations studied, that sector may play a different role in the national socio-cultural contexts of different nations in the study. Being a university student is an elite status in many societies, but only a moderately privileged status in other societies. Being a member of a labor union in a society with government-sponsored unions is likely to be quite different than in a society with independent unions. It is also possible that both the measure based on the convenience sample and the measures based on the representative sample do not adequately measure the phenomenon that is intended to be measured. That is, both could be low in construct validity (Campbell and Fiske 1959). The evidence on that type of validity for the variables in the IDVS is presented elsewhere (M. A. Straus 2009).
Nationally representative probability samples remain the preferred method whenever possible. Nevertheless, when that is not possible, these results support proceeding with convenience samples of specific categories of the population, as long as they focus on roughly the same category of population in each country (such as students, urban dwellers, construction workers, etc.) and provided the data is collected in comparable ways across nations. In addition, including a few questions that can be correlated with pre-existing nationally representative data can enable tests of validity such as those in this article. Finally, these results may open the way for funding agencies to support more cross-national research. It is hugely expensive to conduct representative sample studies in ten, twenty, or forty nations, and very few such studies have been funded. Multi-nation studies using convenience samples can be done for a fraction of the cost, but grant review committees are reluctant to approve studies that use unrepresentative convenience samples. The evidence in this article showing the cross-national validity of research based on convenience samples might facilitate grant review committees approving meritorious cross-national studies which, in the absence of this evidence, might not be approved because the design does not include representative samples within each nation. If that were to happen, it could contribute to substantially increasing the number and scope of multi-nation cross-national comparative research.