The use of information and communication technology (ICT) has been on the rise for decades and has become an inevitable daily practice in multiple areas of life, particularly since the COVID-19 pandemic lockdown (Richter & Mohr, 2020; Rizun & Strzelecki, 2020). The way people feel and behave when confronted with ICT also depends on evaluations of their own self-perceived ICT competence in general and in specific competence domains (i.e., ICT self-concept, ICT-SC).

Self-concepts belong to the group of competence self-beliefs (Marsh et al., 2017) and describe the individuals’ mental representation and evaluation of one’s own competences (Brunner et al., 2010), which are formed by experiences, feedback from significant others, and interactions with the environment (Shavelson et al., 1976). Scientific research on self-concept comes mainly from the educational context. Here, decades of research (Byrne & Gavin, 1996; Marsh et al., 1988; Marsh, 1990c; Shavelson et al., 1976) showed that academic self-concept (ASC) is a multidimensional and hierarchically structured set of self-beliefs. The nested Marsh/Shavelson (NMS) model reflects the latest structural model of ASC (Arens et al., 2021) stating a general ASC at the apex and nested competence domain-specific ASCs (in school context subject-related self-concepts, e.g., math self-concept), representing the multidimensional nature of ASC (corresponding to the structure of ICT-SC in Figure 1).

Embedded in a nomological network, domain-specific ASC shows manifold health-promoting and motivational relations with affective (e.g., anxiety), behavioral (e.g., choice), and cognitive (e.g., perception of control) constructs (cf., Schauffel et al., 2021). Regarding its measurement, widely used validated scales are found in the English language (e.g., [Academic] Self Description Questionnaire I-III, SDQ I-III: Marsh, 1990a, 1990b; Marsh et al., 2005; Marsh & O'Neill, 1984, ASDQ: Marsh, 1990c) and German language (e.g., Arens et al., 2011; Schöne et al., 2002; Schwanzer et al., 2005 ), and they sufficiently represent the multidimensional and hierarchical structure of ASC (Byrne, 1996).

Based on the theoretical assumptions and research findings on ASC, the ICT-SC can be described as individuals’ mental representations and evaluations of their own competences in dealing with ICT, and these representations and evaluations are multidimensionally and hierarchically structured in a general ICT-SC and domain-specific ICT-SCs, which are comparable to general ASC and domain-specific ASCs. ICT as a generic term refers to multiple digital devices (e.g., computers, tablets, and smartphones) as well as software (e.g., messenger tools and programs such as Microsoft Office). Regarding the domain specificity of the ICT-SC, research shows that if objective competence demands are multidimensional, then the corresponding self-concepts are structured multidimensionally (Brunner et al., 2010).

The European Digital Competence Framework for Citizens (DigComp, Ferrari, 2013) and its updated versions DigComp 2.0/2.1 (Carretero et al., 2017; Vuorikari et al., 2016) represent a comprehensive framework that integrates and extends former models of ICT competence, including the Canadian Digital Skill Framework (Chinien & Boutin, 2011) or the components of ICT literacy by Katz (2007), among others (e.g., Eshet-Alkalai, 2004; Ferrari, 2012; Martin & Grudziecki, 2006). Therein, ICT competence is systematized alongside the five competence domains of information and data literacy, communication and collaboration, digital content creation, safety, and problem-solving, to which 21 competences are assigned. The applicability of DigComp 2.1 has been demonstrated across ages, contexts, and countries and it is widely used in practice and research (Carretero et al., 2017; Laanpere, 2019; Vuorikari et al., 2016). Consequently, domain-specific ICT-SCs should reflect the five domains of ICT competence according to DigComp 2.1.

Apart from theoretical and empirical arguments supporting multidimensionality, there is also practical support for a multidimensional ICT-SC that differentiates between competence domains (domain-specific) rather than between the digital tools used (tool-specific). In modern environments, the same tasks (e.g., booking a flight) can be accomplished with different digital tools (e.g., tablets, smartphones, and computers), and a specific task (e.g., online banking) may require a set of digital tools (e.g., computers and TAN[1] on a mobile phone). Thus, application requirements are primarily associated with qualitatively different competence domains instead of the tool used.

To simultaneously account for a multidimensional and hierarchical structure of the ICT-SC, the general ICT-SC and domain-specific ICT-SCs are not located on the same level (i.e., first-order-correlated-factor model [FOCF model]) but nested (see Figure 1), with a superordinate g-factor representing the general and domain-unspecific ICT-SC. Domain-specific ICT-SCs represent ICT-SC in specific competence domains (e.g., communicate).

Figure 1. Structural understanding of ICT-SC

Note. ICTSCg = g-factor general ICT self-concept (ICT-SC); SCCO = domain-specific ICT-SC communicate that refers to an individual's evaluation of his or her own competences related to communicating with relevant others through the use of ICT; SCPS = domain-specific ICT-SC process and store that refers to an individual's evaluation of his or her own competences related to processing and storing digital data, information, and content; SCGE = domain-specific ICT-SC generate content that refers to an individual's evaluation of his or her own competences related to creating digital data, information, and content; SCSA = domain-specific ICT-SC safe application that refers to an individual's evaluation of his or her own competences related to protecting digital data, information, and content as well as an entire system. It also refers to the responsible use of ICT; SCSP = domain-specific ICT-SC solve problems that refers to an individual's evaluation of his or her own competences related to successfully solving emerging technical problems of ICT as well as his or her competences related to solving content-related tasks, challenges, and problems using ICT; SCGL1-5 = Items that refer to the general ICT-SC.

Item generation and selection

We conceptualized the ICT-SC based on an extensive literature review on the topics of ICT competence and domain-specific self-concept, including literature databases and publications from psychology, educational science, economic, and socio-political disciplines. As a starting point, we developed a scale to measure ICT-SC in a specific application area, namely, the workplace, as a first pre-version, whereas the final scale is intended for a broad target group with regard to age (18-69), context, and countries.

We generated items based on ICT competences specified in the DigComp 2.1(Carretero et al., 2017). To ensure content validity, representatives of heterogeneous industries, including health services and manufacturing (N = 4) qualitatively evaluated the competences and derived behavioral indicators (e.g., safe application: screen look and anti-virus software). Only competences for which all four representatives could derive behavioral indicators were considered in the item construction.

The linguistic item formulation took into account aspects of linguistic comprehensibility and consistency (for details see Moosbrugger & Kelava, 2012) as well as widely used instruments of domain-specific self-concept research with good psychometric properties of domain-specific self-concept research in the English (SDQ I-III: Marsh, 1990a, 1990b; Marsh et al., 2005; Marsh & O'Neill, 1984; ASDQ: Marsh, 1990c) and the German language (e.g., Arens et al., 2013; Schöne et al., 2002; Schwanzer et al., 2005). Using formulation principles of well-proven self-concept scales, we aim to establish distinctiveness from other competence beliefs, such as self-efficacy.

We began with a large set of items generated by the authors independently from each other. Items were cognitively pre-evaluated via an iterative process (Pretest I: N = 5, Pretest II: N = 3) with the method of thinking aloud (van Someren et al., 1994) by subject-matter experts (SMEs, Cortina et al., 2020) and novices as representatives of the scale’s target population. Redundant and otherwise problematic (e.g., ambiguous) items were adapted or discarded, remaining 43 items. Seven items measured the general ICT-SC, and six items with comparable item strains (e.g., “In the work context I am good”/“I learn quickly”/“I have always been good”) measured each domain-specific ICT-SC. One item per facet (e.g., SCCO) was negatively formulated (“In the work context, I am not good”). Two items per facet considered the social reference norm (“Compared to my colleagues”) because social comparisons are a central source of self-concept (Bong & Skaalvik, 2003).

We choose a six-point fully-labeled Likert-type rating scale as the response format, ranging from strongly disagree (1) to strongly agree (6). A neutral middle category was omitted to reduce the response tendency towards the middle.

The pre-version of the final scale was first piloted in a sample of German employees (S1). To winnow the initial set of 43 items, we examined item characteristics, scale characteristics, and the factorial structure (CFA). We used the statistic software Mplus 8 (Muthén & Muthén, 1998-2017). For CFA, we used maximum likelihood estimation (MLR) because of its robustness against mild deviations from the normal distribution (Hox et al., 2010) and the full information maximum likelihood estimation procedure (FIML) to handle missing data, because all missing values could be classified at least as missing at random (MAR, Enders, 2010).  Item means ranged from 3.44 to 5.15 and all standard deviations were higher than 0.83, indicating appropriate variability (Stumpf et al., 1983). The response range was strongly disagree (1) to strongly agree (6). The results of the item and scale analysis showed excellent reliability (α = .82 to α = .90). Item selectivities were good (r_it = .54 to r_it = .81), except for the inverse items (“I am not good”, r_it = .27 to r_it = .46); therefore, these items were excluded from further factorial analysis.

The results of CFA supported a multidimensional and hierarchical factorial structure and indicated the exclusion of the comparative items (“Compared to my colleagues”) that refer to the social reference norm. The best-fitting model was an incomplete bifactor model (i.e., NMS model). Here, model fit without the comparative items (χ² = 453.110, p < .001, χ²/df = 1.849, CFI = .955 TLI = .940, RMSEA = .055, SRMR = .035) was better than with these comparative items (χ² = 1955.484, p < .001, χ²/df = 3.320, CFI = .824, TLI = .801, RMSEA = .091, SRMR = .122). In the best-fitting model, all items loaded significantly on the higher-order g-factor. At least 75% of the newly developed items loaded significantly on the respective factor. The item selection and adaptation process from the pre-version to the final scale ICT-SC25 was performed based on the findings of the piloting (for details see ESM 2 of the original article Schauffel et al., 2021). In summary, based on these analyses, we excluded inverse items (e.g., “In the work context, I am not good at communicating with others via digital systems”) and comparative items that referred to the social reference norm (e.g., “Compared to my colleagues, I am better at handling digital systems”). Apart from educational context (e.g., class), referencing a comparison group is not trivial. In the work context, multiple team memberships may exist, and in private life, the social reference group is even less clear. Items with poor factor loading were reformulated (e.g., changed the verb from “call up” to “find”). To widen the scale’s applicability beyond the work context, we further adapted the item strains (e.g., from “In the work context, I can” to “I can”), and the instruction (e.g., “digital systems in the work context” to “digital systems”). The final 25-item scale ICT-SC25g and its English translation (ICT-SC25e) resulted (see Table 1). The translation procedure was as follows: First, the items were translated into English by two researchers (mother tongue German; fluent in English). Second, item translation was examined by experts in psychological diagnostics who are fluent in both English and German and back-translated to German. Third, the translated items were discussed and adapted in iterative workshop cycles with experts in the field of content.

Samples

We used data from five independent samples (total N = 2024). Table 2 displays demographic information about the participants of the five samples. In each survey (i.e., sample), our research goals were outlined, participation was voluntary, and withdrawal was possible at any given time of the survey.

Sample 1 (S1) consists of 280 employees (66.8% female, 32.5% male, 0.7% divers). Survey participants were acquired via multiple channels (e.g., social media, career portals). The sample was heterogeneous with regard to personal and occupational characteristics. Fielding took place in summer 2019 using Unipark (Questback, 2020). As an expense allowance, cash prizes were raffled among the participants. Data were used to pilot the pre-version of the final scale.

Sample 2 (S2) represents a quota sample (N = 486, 48.8% male, 51.2% female) that reflects the heterogeneity of the adult population in Germany with regard to age, gender, and educational attainment. Only native speakers were recruited. Fielding took place in January 2020 using a web-based survey (computer-assisted self-administered interviewing [CASI]) by the online access panel provider respondi AG. Data collection was performed as part of the investigation of the quality of several questionnaires. Participants were financially rewarded for their participation. A subsample was reassessed after approximately two to three weeks (Mdn = 14 days). Data were used to validate the final scale ICT-SC25g.

Sample 3 (S3) represents a second German quota sample (N = 571, 49.0% male, 51.0% female), which reflects the heterogeneity of the adult population with regard to age, gender, and educational attainment. Data were collected in August 2020 following the same procedure as in sample 2. Data were used to replicate previous findings of this study and to further validate the ICT-SC25g. We used a planned missingness three-form design (Graham et al., 1994), in which a large number of items (i.e., survey questions) were divided into four subsets, including a common block (X) and three partial blocks (A, B, and C). Items in the X set were asked first and to everyone (i.e., demographics, frequency of ICT use). In addition, the item sets in the three forms A, B, and C were rotated so that different item sets appeared last in each form. As a result, one-third of the participants did not answer the questions in sets A, B, and C (i.e., ICT-SC and all other validation measures).

Sample 4 (S4) includes data from 204 German undergraduate psychology students (12.7% male, 87.3% female). Data collection was performed as part of longitudinal monitoring of digital teaching in spring 2020. Measures used in our study were assessed in the middle of the semester (May 2020). Participants were rewarded either financially or by credits. Data were used to validate the ICT-SC25g in a specific application context.

Sample 5 (S5) represents an English quota sample (N = 483, 48.4% male, 51.6% female) that reflects the heterogeneity of the adult population in the United Kingdom with regard to age, gender, and educational attainment. Only native speakers were recruited. Fielding took place in January 2020 following the same procedure as in S2. Data were used to pilot the English scale version ICT-SC25e.

Table 2

Demographic characteristics of the participants in the five samples

Note. ^a Age group instead of exact age collected. Age groups ranged from 18 to 25 years to 65 to 74 years. ^b Frequency distribution of age groups was as follows: 11.1% were 18 to 24 years old, 45.4% were 25 to 34 years old, 9.6% were 35 to 44 years old, 19.6% were 45 to 54 old, 13.9% were 55 to 64 years old, 0.4% were 65 to 74 years old. ^c The equivalent UK and German educational levels are as follows (from low to high): ohne Bildungsabschluss/Hauptschule [no educational qualification/lower secondary leaving certificate], mittlerer Schulabschluss [intermediate school-leaving certificate], (Fach-) Hochschulreife [higher education entrance qualification].

Item analyses

We run a multisample-multiphase process to ensure high psychometric properties of the developed scale (Cortina et al., 2020). In phase 1, a pre-version of the final German-language scale was developed. In phase 2, the pre-version was piloted, and the final item set was selected and adapted based on item, scale, and factorial analyses (ICT-SC25g). These phases are documented in the section “Item generation and selection”. We then translated the scale into English (ICT-SC25e). In phases 3 to 6, we validated the ICT-SC25g/e. In phase 3, we tested the factorial validity of the ICT-SC25g/e using confirmatory factorial analysis (CFA) and tested its measurement invariance (MI) across countries. In phase 4, we analyzed additional psychometric properties, such as scale internal consistencies and test-retest reliability. In phase 5, we examined the construct validity (i.e., convergent and discriminate validity) of the scale by investigating the ICT-SC25g/e in a nomological network. In phase 6, we examine the comparability of the ICT-SC25g/e scale across gender. Table 3 summarizes how the five samples were used throughout the six phases.

Table 3

Structured overview of samples, phases, and validation measures

Throughout the six phases (Table 3), we conducted several (partly replicating) validation and reliability analyses. The methods of analysis were confirmatory factor analysis (CFA), reliability analyses, correlation analyses, and measurement invariance testing.

To investigate the assumed multidimensional and hierarchical structure of ICT-SC, we specified four models that are discussed in ASC research: a g-factor model, a FOCF model, a second-order model, and an incomplete bifactor model that represents the NMS model. See Schauffel et al. (2021) for the model illustrations and the methodological review by Arens et al. (2021) for an in-depth discussion of these models. The models were evaluated and compared in terms of theoretical relevance and model fit information. A good model fit was indicated when the comparative fit index (CFI) and the Tucker-Lewis index (TLI) were > .95, the root-mean-square-error-of-approximation (RMSEA) was < .06 and the standardized root-mean-square-residual (SRMR) was < .08 (Hu & Bentler, 1999). The relative fit indices Akaike information criterion (AIC), Bayesian information criterion (BIC), and the sample-adjusted BIC (Adj. BIC) were used for model comparison at a descriptive level. Smaller values indicated a better model fit (Hu & Bentler, 1999). We considered standardized factor loadings λ ≥ .30 to be substantial (Floyd & Widaman, 1995). We evaluated correlations between the latent variables (factors) of the ICT-SC25g/e using the conventions of Gignac and Szodorai (2016) for correlations with personality traits (small: |r| ≥ .10, typical: |r| ≥ .20, large: |r| ≥ .30).

Item parameters

Table 4 shows the descriptive statistics for each of the 25 items of the ICT-SC25g/e. Thereby samples 2 to 4 refer to the German scale version (ICT-SC25g) and sample 5 refers to the English scale version (ICT-SC25e).

Table 4

Item and scale characteristics of the ICT-SC25g/e

Note. Scale in italics, item code not. SCCO = domain-specific ICT self-concept (ICT-SC) communicate; SCPS = domain-specific ICT-SC process and store; SCGE = domain-specific ICT-SC generate content; SCSA = domain-specific ICT-SC safe application; SCSP = domain-specific ICT-SC solve problems; SCGL = first-order factor general ICT-SC.

Descriptive statistics (scaling)

Descriptive statistics across the five validation samples are displayed in Table 4.

Further quality criteria

Test economy: Due to the short completion time per subscale (< 1 min) the ICT-SC25g/e can be seen as a flexible and economic instrument to measure ICT self-concept.

Test fairness: We ran multigroup CFAs to test whether the best-fitting model was invariant across gender (male vs. female) and countries (Germany vs. UK). We tested increasing levels of MI (i.e., configural, metric, scalar, and strict) against each other, starting with the least constraint solution (step-up approach, Putnick & Bornstein, 2016). The pattern of factor loadings is equal across the user groups if configural MI holds. If metric MI also holds (“equal factor loading”), a comparison of variances between latent variables is also possible. Furthermore, the comparison of latent scale means is possible if scalar MI (“equal intercepts”) holds (Chen et al., 2019). If strict MI (“equal residual variances”) holds, then further comparisons of manifest means and analyses with manifest scale scores are allowed (Vandenberg & Lance, 2000). We follow the recommendations of Chen (2007) and examine increasing levels of MI by assessing the change in CFI and RMSEA since the chi-square difference test is sensitive to the sample size. If the decrease in CFI is less than .010 and the increase in RMSEA is less than .015, a higher level of MI is accepted. The results were comparable for the German (ICT-SC25g, S2/S3) and English (ICT-SC25e, S5) scale versions and supported strict MI across gender and across countries within both structural models (NMS/FOCF). The Satorra-Bentler corrected chi-square (SBχ2) difference test was either nonsignificant or the changes in CFI and RMSEA were below the critical threshold of ΔCFI < .010 and ΔRMSEA < .015, as recommended by Chen (2007).

Further literature

The ICT-SC25g/e was first published in Computers in Human Behaviors reports.

• Nathalie Schauffel, Trier University, Business Psychology, Universitätsring 15, 54296 Trier; E-Mail: schauffel@uni-trier.de
• Isabelle Schmidt, GESIS–Leibniz Institute for the Social Science, P.O. Box 12 21 55, 68072 Mannheim, Germany, E-Mail: Isabelle.Schmidt@gesis.org

The data and code used throughout this scale development and validation is available online on the GESIS SowiDataNet | datorium repository (https://doi.org/10.7802/2343).