According to Deary (2012, p. 348), “intelligence predicts important things in life.” In particular, the relevance in job-related fields and education had been disclosed even decades ago. In an early paper, Harrell and Harrell (1945) demonstrated the association between intelligence and specific occupations. Additionally, intelligence is highly correlated with job performance and job training success, which was proven in a meta-analysis by Salgado and colleagues (2003). In another meta-analysis, Poropat (2009) had shown significant associations between intelligence and academic success.

Intelligence must be understood as the mental ability to learn and solve problems. In other words (Gottfredson, 1997, p. 13):

Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly, and learn from experience. It is not merely book learning, a narrow academic skill, or test-taking smarts. Rather, it reflects a broader and deeper capability for comprehending our surroundings “catching on”, “making sense” of things, or “figuring out” what to do.

The Cattell-Horn-Carroll Model (Schneider & McGrew, 2012) allows us a more detailed look at the theoretical structure of intelligence. It distinguishes general intelligence (see also Carroll, 1993), as well as abilities on the broad (e.g., Reading & Writing Ability, Comprehension-Knowledge, and Fluid Reasoning) and narrow level (e.g. of Fluid Reasoning: Induction, General Sequential Reasoning, and Quantitative Reasoning).

According to Carroll (1993), figural matrices primarily measure induction (which is the narrow level of Fluid Reasoning): The test taker’s task is “to inspect a set of materials and from this inspection induce a rule governing the materials, or a particular or common characteristic of one or more stimulus materials, such as relation or trend” (p. 211). Schneider and McGrew’s (2012) definition of induction is quite similar: It is “the ability to observe a phenomenon and discover the underlying principles or rules that determine its behavior” (p. 112). Accordingly, we conclude that the HMT is a test of induction. Induction, in turn, is defined by Schneider and McGrew (2012) as a narrow ability. Further, it is the core of the broad ability of fluid reasoning (Schneider and McGrew, 2012). Thus, the HMT is a test of induction as well as of fluid reasoning. One may now argue that there are plenty of well-designed, validated, and established tests of intelligence and the HMT would be redundant. However, this is the core point of the benefit: The aim of the HMT was to develop a web-based, economic, and valid alternative which is not very different from other tests using matrices, but: it is free of cost.

If not otherwise indicated analyses were computed with SPSS (IBM).

Item generation and selection

The development of the HMT was part of the doctoral dissertation of the first author. Items were generated and selected in three pilot studies conducted from 2009 to 2011. In a fourth study the results of which are presented below the final version of the HMT was validated. The matrices were designed by using different operations: horizontal and/or vertical addition or subtraction and/or varying the positions of separate elements, i.e. rotation or movement. The original version was introduced in German. A professional translator translated the original version into English and Spanish. Because of the non-verbal tasks and existing samples without back translation or other procedures.

Samples

The sample for the validation study of the HMT was recruited in the year 2011 and consisted of 1,339 B.Sc. psychology students from the University of Hagen, a German distance learning university. 76% were female. The mean age was 32.2 years (SD = 8.97). Participants were recruited via email and the university’s online-studies webpage. They received course credit for their participation. Initially, 3,405 students registered. Several steps of data cleaning were performed. 1,384 participants abandoned the questionnaire at some point. Of the remaining 2,021, 119 participated multiple times and were excluded. Another 563 students were removed because they had taken part in at least one of the preliminary studies so that the final sample only included first-time participants. All tests were performed online except for the Intelligence Structure Test 2000 R (I-S-T 2000 R; Liepmann et al., 2007) which was administered as proctored paper-and-pencil test.

Item analyses

To determine the dimensionality of the HMT, a parallel analysis (PA; Horn, 1965; based on O’Connor, 2000) (with 9,000 data sets using principal component eigenvalues), the minimum average partial (MAP) test (Velicer et al., 2000, based on O’Connor, 2000), the comparison data (CD) technique (Ruscio & Roche, 2012, using R 2.15.1) and the scree test (Cattell, 1966) were performed. All four analyses suggested a two-component solution. Hence, a principal component analysis (PCA) with two predefined factors was conducted. The two factors explained 46.1% of the variance, 35.1% were described by the first factor. All loadings on factor 1 were greater than .30. The loadings on factor 2 decreased continually from a = .53 (item 1) to a = -.59 (item 19). The correlation of the loadings on factor 2 with item difficulty was r = .85 (p < .001) so that factor 2 was interpreted as a “spurious” difficulty factor (see McDonald & Ahlawat, 1974). Factor 1 represents reasoning, the fundamental ability needed to solve matrices.

Item parameters

Table 1 shows the item difficulties and item-total correlations. The first two items can be considered as very easy (p > .80). Seven items demonstrate a strong difficulty p < .20. The mean difficulty was M = .37 (SD = .26, Range .10 to .88). The correlation between item position and item difficulty was r = -.94 (p < .001). The item-total correlations ranged from r_it = .19 to r_it = .50. The mean number of correct responses was M = 7.43 (SD = 3.38).

Table 1

Item difficulty (p) and item-total correlations (r_it)

Objectivity

The HMT is a web-based test. Instruction, administration, scoring, and interpretation of the HMT are computer-based and thereby highly standardized. Accordingly, application, evaluation, and interpretation of the HMT can be deemed highly objective.

Reliability

The internal consistency calculated with the Kuder-Richardson Formula 8 (Kuder & Richardson, 1937) was r_KR8 = .78 and the retest-reliability was r_tt = .75. The mean test-retest interval was M = 78 days (SD = 123) with a range of 5 to 388 days. This indicates that the HMT is an adequate test for group-level analyses.

Validity

To attest convergent, divergent and criterion validity, several analyses were performed. To verify convergent validity, correlations with other intelligence tests were calculated. Divergent validity was assessed by correlating the HMT to measures of non-cognitive personality traits.

The highest correlations with other measures of intelligence (see Table 2) were found for the reasoning ability measures from the I-S-T 2000 R (Liepmann et al., 2007). An overview of the correlations between the HMT and other intelligence measures to attest convergent validity are shown in Table 2. According to Cohen (1992) correlations of r = .10 can be considered as small, r = .30 as medium and r = .50 as large effects.

Table 2

Correlations between the HMT and intelligence measures

Correlations with personality traits appear in Table 3. Positive and negative affectivity (measured with the Positive and Negative Affect Schedule; PANAS; Krohne et al., 1996; based on Watson et al., 1988), all Big Five dimensions (measured with the Big Five Inventory; BFI; Lang et al., 2001; based on John & Srivastava, 1999), and narcissism (measured with the Narcissistic Personality Inventory; NPI; Schütz et al., 2004; based on Raskin & Hall, 1979) were not correlated with the HMT. There were some significant but small correlations with certain facets of the HEXACO Personality Inventory-Revised (HEXACO-PI-R 100; Lee & Ashton, 2018) and the Personality-Adjective Scales (PASK5; Brandstätter, 2010, 2014). All in all, these results indicate discriminant validity.

Table 3

Correlations between the HMT and personality traits)

Besides convergent and divergent validity, criterion validity was investigated by calculating correlations with high school grades and the grade point average. Additionally, the correlation with the statistics grade in the B. Sc. Psychology studies was computed. Results are shown in Table 4. The HMT correlated with study grades, high school school-leaving qualification, and high-school grades in mathematics and biology. This indicated criterion validity.

Table 4

Correlations between the HMT and indicators of academic achievement

Descriptive statistics

N = 1,339 participants took the test, the mean number of correctly answered matrices was M = 7.43 with a standard deviation of SD = 3.38.

Further quality criteria

The HMT is an economic test because it is relatively short with a mean duration of less than half an hour. It is web-based and free of cost so that it is a useful tool for group analyses especially in the context of psychological research. Most items are quite difficult, so that it is recommended to use the HMT only with samples of at least average, or even better, above-average intelligence to avoid frustration.

Men (M = 8.37, SD = 4.26, N = 347) solved approximately one more matrix than women (M = 7.11, SD = 3.38) with a difference of M_Δ = 1.26 (d = 0.34, t = 4.98, df = 507.45, p < .001, N = 987). Other studies on sex differences in intelligence tests measuring induction with figural matrices tests found similar results (Irwing & Lynn, 2005).

In addition to gender effects, age effects were detected as well. The association was r = -.12 (p < .001, N = 1,333) indicating that younger participants solved more items correctly. Both, gender and age effects can be considered substantial but not unusual for that kind of test.

Further literature

Heydasch, T., Haubrich, J. & Renner, K.-H. (2020). The Short Version of the Hagen Matrices Test (HMT-S). A 6-Item Induction Intelligence Test (T. Heydasch, Trans.). methods, data, analyses, 7, 183e-205e. https://doi.org/10.12758/mda.2013.021