The CCS-T-7 conceptualizes traditionalism as a facet of Conscientiousness. In a study aimed at developing a comprehensive lower-order taxonomy of Conscientiousness, Roberts, Chernyshenko, Stark, and Goldberg (2005) identified traditionalism as one of six facets of this personality dimension. People who score high on traditionalism "tend to comply with current rules, customs, norms, and expectations; they dislike changes and do not challenge authority" (Roberts et al., 2005, p. 122). Integrating traditionalism into the Big Five framework, these authors found that it correlated not only with Big Five Conscientiousness but also (negatively) with Big Five Openness to Experience.

Item generation and selection

The items of the scale were drawn from the Chernyshenko Conscientiousness Scale (CSS; Chernyshenko, 2003; Hill & Roberts, 2011). Based on Roberts et al.'s (2005) comprehensive taxonomy of conscientiousness, Chernyshenko (2003) developed a pool of traditionalism items from which Hill and Roberts (2011) derived a 10-item traditionalism scale. We in turn obtained the CCS-T-7 scale from Hill and Roberts’ (2011) 10-item scale by dropping two items to simplify the scale for low-literacy adults and dropping a further item to achieve an acceptable model fit of a unidimensional measurement model. The items of the scale were then translated from English into German, French, Spanish, Polish, and Japanese based on the multistep team translation approach TRAPD (Harkness, 2003). In each country, two independent professional translators translated the scale from English into the target language. Then an expert on the traditionalism construct reviewed those translations and finalized them in exchange with the professional translators. Scale development and translation were coordinated by an OECD expert group on non-cognitive skills.

Samples

The scale was administered in two online surveys conducted by the OECD: First in the USA and then in Germany, France, Spain, Poland, and Japan. The initially collected samples were quota samples based on gender, age, and education according to the country-specific distributions. Table 2 shows sample sizes and sociodemographic characteristics after data cleaning and case exclusion.

The data doesn’t comprise any missing values on the CCS-T-7 items, the sociodemographic variables, and the BFI-2 scales used for construct validation of the scale. It comprised very few missing values on the cognitive ability scale used for further scale validation. To deal with those missing values we applied list-wise deletion (see Table 5).

Table 2

Sample characteristics

Item analyses

We investigated the factorial structure of the items using multi-group confirmatory factor analysis (CFA). Because of the big sample size we used ML as estimator but checked that fit indices and standard errors estimated with MLR didn’t deviate severely. We used a unidimensional measurement model developed for the United States specifying one general traditionalism factor. In all countries, factor loadings and item intercepts were freely estimated, whereas the variance of the latent traditionalism factor was set to 1. The structure of the measurement model is sketched in Figure 1 below. While the chi-square test proved significant c²(84) = 755.384, p ≤ .001, more suitable fit indices indicated an acceptable to good fit of the multi-group model, with RMSEA = .077, CFI = .931, and SRMR = .036 (Hu & Bentler, 1999; Schermelleh-Engel, Moosbrugger, & Müller, 2003; Schweizer, 2010). The model parameters are shown in Table 3.

Figure 1. Measurement model for the traditonalism construct. Factor loadings were estimated freely, whereas the variance of the latent traditionalism factor was set to one. Error terms are not displayed.

Item parameter

Table 3 shows the standardized factor loadings and the unstandardized item intercepts of the multi-group structural equation model. The squared standardized factor loading indicates the item variance explained by the latent factor. An unstandardized item intercept indicates the difficulty level of an item (the lower the intercept the more difficult the item). On average, the standardized factor loading was .52 which suggests that the items reflect the latent traditionalism variable well. Most factor loadings were >.30. The third item (“I believe that people should be allowed to take drugs, as long as it doesn’t affect others”) revealed loadings >.30 in Germany and Spain, but smaller loadings especially in Poland and Japan. This suggests that this item is less associated with traditionalism in these countries. Alternatively, the reverse-keying might have provoked a slightly different response behavior for this item than for the other six items.

Objectivity

The scale is applied in a self-administered online assessment. The written instructions, the scoring rules, and the reference statistics ensure the objectivity of the measurement.

Reliability

We calculated Cronbach’s alpha and McDonald’s omega (McDonald, 1999) to assess the reliability of the manifest scale score (mean or sum score). Cronbach’s alpha is reported because it is commonly used, even though it is an unbiased reliability estimate only for at least essentially τ-equivalent measurement models in which all factor loadings are restricted to be the same. McDonald’s omega is reported because it is more appropriate for τ-congeneric measurement models that may show a heterogeneous structure of factor loadings. The coefficients displayed in Table 4 are >.70 in the USA, Germany, France, and Spain which suggests sufficient reliability of the manifest scale score for group studies (Danner, 2015). In Poland and Japan, Cronbach’s alpha and McDonald’s omega of the mean or sum score are >.60 which is described as sufficient for exploratory research (Hair, Black, Babin, Anderson, & Tatham, 2006).

Validity

To examine the convergent construct validity of the traditionalism scale, we calculated correlations (based on the manifest scale scores) between traditionalism and the Big Five Inventory 2 (BFI-2; Soto & John, 2017), which measures the Big Five personality domains Extraversion, Agreeableness, Conscientiousness, Negative Emotionality, and Open-Mindedness. The BFI-2 items were translated from English into the other five languages together with the CCS-T-7 items. Furthermore, we explored the correlations between the traditionalism scale and age, educational level, and cognitive abilities (ICAR verbal reasoning test; The International Cognitive Ability Resource Team, 2014), which were also included in the OECD surveys. The data collection within the OECD surveys didn’t allow for a hypothesis-guided investigation of the scale’s discriminant validity or criterion validity, which should be caught up on in a study with a more flexible data collection. The correlations are shown in Table 5. According to the guideline of Gignac and Szodorai (2016) we considered correlations of .10 as small, .20 as typical and .30 as relatively large.

As expected, there were large correlations between the traditionalism scale and the Conscientiousness scale in most countries. Only in Japan, the correlation was only typical. There were also typical to large correlations between traditionalism and Agreeableness in all countries. The BFI-2 Open-mindedness scale displayed null correlations or small correlations with the traditionalism scale across countries. Traditionalism was conceptualized as a lower order facet of Conscientiousness. Thus, the positive association of the CCS-T-7 with the BFI-2 Conscientiousness scale supports the convergent validity of the measure. The positive association between the CCS-T-7 and BFI-2 Agreeableness might be mainly due to the Agreeableness facet of Respectfulness. The null correlations or small and partly positive correlations between the CCS-T-7 and BFI-2 Open-Mindedness are not in line with earlier findings (e.g. Roberts et al., 2005), but can be explained by the item selection of the CCS-T-7 that operationalizes traditionalism mainly through rule-obeying and the respecting of authorities. Hence, scoring high on the CCS-T-7 might not preclude also scoring high on the BFI-2 Open-Mindedness facets of Aesthetic Sensitivity, Intellectual Curiosity and Creative Imagination.

We also found a typical to large negative correlation between the CCS-T-7 and cognitive abilities in the USA. In Germany, France, Spain, and Poland there were small negative correlations between traditionalism and cognitive ability.

Descriptive statistics (scaling)

Table 6 shows reference statistics of the traditionalism scale for each of the investigated countries. As can be seen, the scale scores were approximately normally distributed around the scale midpoint.

Table 6

Reference values for each country

Further quality criteria

We investigated the measurement invariance of the scale across the US, German, French, Spanish, Polish, and Japanese sample. We specified and compared configural, metric, and scalar invariance with multi-group structural equation models. In the configural model, we specified the same factor structure for each country, which suggests that the basic structure of the construct is equivalent across countries. In the metric model, we additionally restricted factor loadings to be identical in all countries, which suggests that the latent factors have the same meaning across countries. In the scalar model, we additionally restricted item intercepts to be identical in all countries, which suggests that the difficulty of the items is identical across countries, too. We evaluated measurement invariance based on differences in the χ², CFI, RMSEA, and SRMR between the four models. According to Chen (2007), a difference of ΔRMSEA ≤ .015, ΔCFI ≤ .010, and ΔSRMR ≤ .030 between the configural model and the metric model suggests metric invariance and a difference of ΔRMSEA ≤ .015, ΔCFI ≤ .010, and ΔSRMR ≤ .010 between the metric model and the scalar model suggests scalar invariance. Table 7 shows the fit indices of the configural, metric, and scalar traditionalism models and the level of measurement invariance they each imply. Moreover, the results of the chi-square difference tests between the models are displayed.

As shown in Table 7, the root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) imply metric invariance of the traditionalism scale, whereas the comparative fit index (CFI) and the chi-square difference tests imply only configural invariance. The chi-square difference test is usually considered as too strict, which is why we give less weight to its results in the determination of the appropriate level of measurement invariance. Unlike the RMSEA and the SRMR, the CFI computes the model fit based on a comparison of the specified model with a model assuming no intercorrelation between the items – the so-called independence model. The better the fit of the independence model, the more the CFI declines with the level of invariance.  As shown above in the section Item analyses CFI and RMSEA imply only acceptable model fit for the configural multi-group model specifying a unidimensional measurement model. This implies a certain degree of independence between the items. Thus, the strong decline in CFI from the configural model to the metric model is due to the limited fit of the unidimensional measurement model. RMSEA however even slightly improves from the configural model to the metric model implying that a metric model fits the data not substantially worse than a configural model. Therefore, we decided to give less weight to the CFI and to accept metric measurement invariance for the traditionalism scale. Metric measurement invariance allows for the cross-national comparison of the strength of correlations between the latent variable and other variables (e.g., correlations with an external criterion) but not of (1) the latent group means and (2) the manifest group means (as reported in Table 6).