The following descriptions of theory and the validation process in sections 3 to 5 are largely taken from the published paper by Böhn and Deutscher (2021).

With regard to vocational in-company training, training quality can be defined as the “subjective perception of characteristics of vocational training that are conducive to certain outcomes” (Klotz et al., 2017). This view relies of a widely accepted approach, the 3-p model of workplace learning by Tynjälä (2013), describing learning processes on three dimensions: presage, process, product (e.g., Biggs, 1999; Seyfried et al., 2000; Visser, 1994). Based of Biggs (1999), Tynjälä (2013) emphasises the individual’s perspective and considers all learner factors and aspects of learning context within the presage (input) dimension. The process dimension includes characteristics of workplace learning such as the structure and performance of work tasks and the individual’s interaction with others. The product (output) dimension covers all learning related outcomes with a focus on the individual’s personal and professional development (Tynjälä, 2013, p. 14).

Besides this broad consensus on the processual model, the ideas about specific characteristics in training reality that define these dimensions vary widely. There is a clear lack of conceptual clarity regarding vocational training quality, which is ultimately reflected in the vast amount of test instruments developed in this area that have partly little intersection. Our research objective was to organize all existing test instruments in the VET context by using a qualitative meta-synthesis, to ultimately develop a survey instrument that comprises all relevant characteristics for workplace learning in the context of VET, excluding learning in vocational schools. The survey items identified were then organized and grouped into the model of workplace learning by Tynjäjä (2013) as a theoretical framework. Figure 1 shows the resulting categories (new items were only created when necessary, see section “Item generation and selection” below).

Fig. 1. Integrative category system based on a qualitative meta-synthesis of test instruments in the context of dual VET (Böhn & Deutscher, 2021, p. 30)

As the German VET system is special with regard to its duality, consisting of state schooling and private training companies, large amount of research and test instruments focus on related aspects such as learning venue cooperation (e.g., Brooker & Butler, 1997; Dwyer et al., 1999; Ebner, 1997; Feller, 1995; Fink, 2015; Heinemann et al., 2009; Keck et al., 1997; Nickolaus et al., 2015; Prenzel et al., 1996; Ulrich & Tuschke, 1995; Virtanen et al., 2014; Walker et al., 2012). Since the goal of vocational training is the attainment of a formal occupational qualification, trainees’ performance needs to be assessed throughout the process. For this reason, work tasks characteristics and the training personnel are of particular relevance for in-company training (Brooker & Butler, 1997; Fuller & Unwin, 2003; Virtanen et al., 2009). Therefore, existing surveys strongly focus on questions related to those aspects (e.g., Beicht et al., 2009; Brooker & Butler, 1997; DGB, 2008; Ebner, 1997; Ernst, 2016; Feller, 1995; Gebhardt et al., 2014; Heinemann et al., 2009; Hofmann et al., 2014; Keck et al., 1997; Koch, 2016; Nickolaus et al., 2015; Prenzel et al., 1996; Velten & Schnitzler, 2012; Virtanen et al., 2014; Zimmermann et al., 1994). We focused on test instruments in the VET context, however, there is contentual overlap to general workplace learning instruments.

Item generation and selection

A qualitative meta-synthesis was conducted to collect and categorize all existing survey instruments on training and learning quality in the area of VET. Thereby, the instrument integrates the full range of research findings on relevant quality criteria and item formulations (for detailed information on literature research, selection, and categorization see the full papers: Böhn & Deutscher, 2019; 2021). Mayring’s (2004) qualitative content analysis (including generalization, selection, and bundling) was conducted to inductively determinate categories for the 3356 items, excluding school-related items. Items with similar contents were grouped (e.g., the items ‘Have you previously held a full-time job?’ [NCVER, 2000] or ‘Do you have previous work experience?’ [Virtanen & Tynjälä, 2008] were grouped under the keyword ‘personal background’). Subsequently, keywords were merged to categories to achieve a reasonable summary of contents, which was recorded in a codebook. The procedure was repeated by the second researcher, each together with a third person (intracoder reliability = .984, intercoder reliability = .926).

Overall, 43 test instruments (oldest from 1994) with 3356 items were classified into 30 categories. Focusing on workplace characteristics, 2343 items (and their categories) could be arranged within the framework model by Tynjälä (2013). User frequencies of items were analysed. The test instruments showed immense nominal and substantive heterogeneity regarding their items. Within the corresponding studies, validation results were not reported in all cases but indicated satisfactory to excellent levels of internal consistency or model fit.

New scales were constructed by relying on frequently used items and some newly developed items. When existing scales and items showed little validation or occupation-specific formulations, they were adapted to a standard format or new items were created. Overall, 82 items could be largely accepted. All other items were reformulated in an occupation-independent manner and designed based on the underlying category codebook. Additionally, the phrases ‘in my company’ or ‘in my department’ were added when necessary to clarify the company focus. The initial German version was translated to English by two researchers and checked by a native English speaker and is thus available in two languages.

A first German version of the VET-LQI, comprising 166 items, was pretested anonymously in three vocational schools in Germany (Baden-Wuerttemberg) in 2017. The sample consisted of 393 trainees in 15 different occupations from the commercial and business management domain. Afterwards, items that caused difficulties in understanding or that did not improve the scale reliability while being dispensable regarding content validity were removed.

The main validation of the German VET-LQI, with 139 items and 31 scales (including 5 person-related and 2 framework scales), then took place anonymously in three other vocational schools in 2018, with 428 trainees from 7 occupations within the commercial and business management domain. For the documentation of the item selection (difficulties etc.), see section “Item Analysis” below and table 24, which is based on confirmatory factor analysis (CFA) due to the theoretical substantiation of the factor structure.

Samples

The sample was recruited via three German vocational schools (PASI mode), where complete school classes of training courses were asked to participate. The sample consisted of 428 apprentices, aged between 16 and 37 (M = 20.5). 54.6% of the participants were female, 45.4% were male. Half of the sample was in their first year of training, while 30% were in their second year and 20% in their third year of training. Apprentices were trained in 7 commercial occupations e.g., Retail Salesman (33%), Industrial Management Assistant (24%) and Office Management Assistant (20%). Compared to the statistical population of the analyzed occupations (using Data by the Institute for Vocational Education and Training), t-test and chi²-tests showed no significant differences with respect to age (p = .426), gender (p = .586) and education (p = .100). 41% possessed a higher school certificate, nearly 50% owned at least a secondary school certificate (see Böhn & Deutscher, 2021).

The number of missing values were low and stayed < 5% for the vast majority of items, except for five items (items 26 and 64: 6%; items 91 and 129: 8%; item 90: 9%). Initially, the items were maintained within the following analyses since the missingness was not excessive and important contents of the respective scales would be missing, if the items were deleted (in case of items 90 and 91, the whole scale 14 would no longer exist). 

Item analyses

As stated above, the version of the VET-LQI that resulted from the pretesting comprises 31 scales, including 5 person-related and 2 framework scales (‘Vocational Training Framework’, ‘Company Framework’). These person-related and framework scales, as well as 2 additional output scales (‘Completion and Final Exam’, ‘Career Choice’) were not part of the following validation process (and thus remain unmentioned in tables 1–22) as they rather represent formative than reflexive theoretical concepts and do not meet the requirements of factor analyses (Bollen & Lennox, 1991; Diamantopoulos & Winklhofer, 2001). Furthermore, the content of these scales can be adjusted and supplemented individually, depending on the context of the survey usage.

As a first step, the internal reliability of the remaining 22 scales (99 items; see tables 1–22) and their item discrimination power were assessed. 15 items could be removed due to low scale correlation (discrimination power < .3; Ebel & Frisbie, 1986). 9 of these items (items 32, 44, 47, 55, 62, 82, 96, 128, 135) also showed missing value ratio above 5%, while 6 items (items 28, 29, 66, 69, 87, 139) improved the respective scale reliabilities.

The resulting scale reliabilities were good or excellent for 19 scales (Cronbach’s alpha > .7), while 2 scales ‘Training Requirements and Ability Level’ (.684) and ‘Overall Assessment and Satisfaction’ (.657) indicated at least acceptable values (Cronbach’s alpha > .6; DeVellis, 1991; Nunnally, 1978). The latter scale could be improved by deleting item 124 but in order to keep school-related satisfaction within the instrument, the item was maintained for the factor analysis. Only the scale ‘Relevance of Tasks’ showed problematic consistency of .447 (Cronbach’s alpha) as it included two items (out of three) with discrimination power < .3 (items 59 and 61).

Additionally, for content-related reasons four items (33, 52, 88, 124) that otherwise improved the scale consistency if deleted and five items above the 5% threshold of missing values (items 26, 64, 90, 91, 129) were initially kept for the next step of factor analysis. As indicated above, in cases of such minor “weaknesses”, decisions to initially keep the items were made, in line with Schermelleh-Engel and Werner (2012), in order to maintain the broad approach of the VET-LQI in terms of content validity.

CFA was conducted for the 22 scales (and 84 remaining items) using the free statistics software R and lavaan package, applying the common principal axis extraction and varimax rotation method. As the model structure was based on the extensive meta-synthesis of training quality instruments, the number of factors and the related indicators were already theoretically substantiated. Additionally, three exploratory factor analysis (initial starting point with 99 items, model 1 [84 items] and model 5 [76 items]) were conducted to enable a comparison of how the items could be structured as factors on a mere statistical basis versus our theory-driven approach. In the beginning (99 items), an EFA with principal-axis extraction suggest 25 factors (eigenvalues > 1), when we theoretically substantiated 22 factors. Then, for model 1, the EFA results suggest 17 factors, whereas the final model 5 would suggest 16 factors only. However, as we substantiated our factor solution theoretically by an extensive literature research and synthesis of 43 instruments, we propose our 22-factor model in order to consider more specific quality criteria of in-company training (the EFA factor matrix can be obtained upon request from the authors).   

In the following, our proposed model was tested in a CFA with 22 factors. The model fit indices as well as item characteristics (uniqueness, factor loadings, discrimination power) were checked and weak items were deleted, leading to more parsimonious models, as presented in the following.[1] MLM was used as a more robust version of Maximum Likelihood (e.g., Curran et al., 1996).[2]

The local goodness-of-fit indices indicated possible improvements for the initial 22-factor model (table 23). Regarding the factor loadings, loadings should be high and unambiguous directed on one factor.[3] Indicators with low communalities or high values for uniqueness were treated with caution for values > .6 (uniqueness). Lastly, all indicators were significantly related to their factor and there were no high correlations (> .8) between variables, therefore multicollinearity was no issue.

Following the presented thresholds, two items (59, 124) were eliminated from the base model for high uniqueness (> .6), low factor loadings (< .4) and low discriminatory power (< .3).[4] The critical item 61 was kept as it was part of the same scale as item 59, thus only the worst item should be deleted to test the results in the next model (table 23, model 3).

For model 4 (table 23), the items 70 (discrimination power: .291) and 133 were removed due to critical values (correlation with item 132 >.8 and high loadings on two factors). For the final model (table 23, model 5) four items were eliminated: Items 21 and 46 showed high loadings on more than one factor, while item 99 correlated highly with item 98. Item 129 could be excluded for its amount of refused answers (above 5%) without harming the scale reliability.

Table 23.

Global fit indices for CFA.

Models allow covariances between factors. *Correlations between factors set to one.

Comparing the models’ fit indices (table 23), the Chi²/df ratio indicated an acceptable fit for model 1 and the following more parsimonious variations (< 2; Byrne 1991). Not surprisingly, the p-value stayed significant for all models due to their complexity. As CFI and TLI deteriorate with model complexity, the indices did not reach an acceptable fit (Hu & Bentler, 1999) but indicated an improvement from model 1 to 5 due to the item exclusions. RMSEA and SRMR indicated a good fit for model 1 and improved further until model 5 (Browne & Cudeck, 1993; Hu & Bentler, 1995). Therefore, model 5 generally shows the best fit indices while being the most parsimonious (see also fig. 2).

Item parameters

Regarding the final item parameters, please see fig. 2 above for item factor loadings and table 24 below for item communalities, means, standard deviations, and discrimination power.

Fig. 2. Factor loadings based on final model 5 (taken from original paper by Böhn and Deutscher [2021, p. 42])

Table 24

Final Scale Reliabilities and characteristics of the final Manifest Items

Note. Scale ranging from 1 (strongly disagree) to 7 (completely agree), N = 428.
*Scale 7 (‘Relevance of Tasks’) did not show sufficient reliability but can be included by researchers who want to check for work task relevance despite the weak values. 

Objectivity

The scales objectively represent the most crucial quality criteria of vocational training identified in the literature. During the application of the instrument, the researcher does not have a negative impact on the objectivity as trainees do not need any assistance or guidance in filling out the easy understandable instrument. Concerning the evaluation and interpretation, it has to be considered that trainee responses represent their subjective perception of vocational training within a specific training company and thus do not constitute an objective measurement. A fully objective measurement does not seem suitable for vocational training reality since there are no objective quality standards available for most criteria and scientific observation of large numbers of training processes does not appear to be economically feasible.

Nevertheless, on an aggregated level survey responses can give valuable insights into the state and changes of training quality with regard to specific companies, specific training occupations, or fields of vocational training. The one-sided measurement approach via trainee perception has to be considered. In this regard, the instrument can easily be adjusted to an application for trainers or trainees’ colleagues to achieve a multi-perspective picture (as was done in Krötz & Deutscher, 2021a/2021b; 2022).     

Validity

Content validity can be assumed by the preceding qualitative meta-synthesis and the theoretical considerations in organizing and reformulating the existent (validated) survey instruments. Besides testing the VET-LQI’s structural validity via CFA (see above), the average variance extracted (AVE) criterion yielded good results (> .5) for 16 of 22 factors regarding the amount of variance explained, while following Fornell and Larcker (1981) five factors were just slightly below (> .4). The scale ‘Relevance of Tasks’ (scale 7), that already showed low internal consistency (see table 24), only achieved a value of .373. Furthermore, Intercorrelation between the 22 factors and the Fornell-Larcker criterion were assessed to measure the degree to which the different quality criteria are unrelated. 13 factors met the Fornell-Larcker criterion (Fornell & Larcker, 1981). The factor correlations in table 25 show that nineteen factors had acceptable moderate correlation values. Only factors 19, 20, and 22 indicate problematic correlations of > .8 (e.g., Evans 1996).

Table 25. Correlations between latent factors

 Reliability

Reliability in terms of Cronbach’s Alpha of the final scales slightly changed after the item eliminations, in most cases implying slight decreases in Cronbach’s Alpha. However, except for scale 7 (‘Relevance of Tasks’), the final values achieve a satisfactory level (see table 24 and section 4 ‘Scale Development’).

Descriptive statistics (scaling)

Descriptive information on the scales and items is shown above in table 24. The data is mostly left-skewed for the relatively high agreement on the items, corresponding to relatively high training quality. 

Further quality criteria

The maximum processing time for participants amounts to 45 minutes. However, as an experience value from several further applications, most trainees finish within 30 Minutes.In our experience most trainees tend to give honest responses due to the anonymity in the survey design. Only a small and neglectable share used response behaviours due to laziness as e.g., crossing the same response for the whole survey page. Faking towards socially desirable values was no issue since the majority of trainees appears to be glad for having the opportunity to evaluate the training reality in their company, stating the perceived strengths and weaknesses as opposed to being the ones who usually become evaluated by the training personnel. There are no other biases known. Invariances across different person/group characteristics are potential research purposes of the survey as, for example, relations between specific characteristics (e.g., education) and distinct perceptions of training quality could be revealed.

• Prof. Dr. Viola Deutscher, Chair for Economic and Business Education, Competency Development and Training Quality,  University of Mannheim L4, 1 68161 Mannheim; Mail: viola.deutscher@uni-mannheim.de
  

Further literature [original validation:]

Böhn, S., & Deutscher, V. (2019). Betriebliche Ausbildungsbedingungen im dualen System – Eine qualitative Meta-Analyse zur Operationalisierung in Auszubildendenbefragungen [Training Conditions in VET – A Qualitative Meta-Synthesis for the Operationalization in Apprentice Questionnaires]. Zeitschrift für Pädagogische Psychologie, 33(1), 49–70. doi:10.1024/1010-0652/a000234

Böhn, S., & Deutscher, V. (2021). Development and validation of a learning quality inventory for in-company training in VET (VET-LQI). Vocations and Learning, 14, 23–53. doi:10.1007/s12186-020-09251-3

Further studies the instrument has been (partly) used for:

Krötz, M., & Deutscher, V. (2021a). Differences in perception matter – How differences in the perception of training quality of trainees and trainers affect drop-out in VET. Vocations and Learning, 14, 369–409. doi:10.1007/s12186-021-09263-7

Krötz, M., & Deutscher, V. (2021b). Betriebliche Ausbildungsqualität – Eine Frage der Perspektive? [Quality of in-company training – A matter of perspective?]. Zeitschrift für Erziehungswissenschaft: ZfE, 24, 1453–1475. doi:10.1007/s11618-021-01041-4

Krötz, M., & Deutscher, V. (2022). Drop-out in dual VET: Why we should consider the drop-out direction when analysing drop-out. Empirical Research in Vocational Education and Training, 14, 1-26. doi:10.1186/s40461-021-00127-x