Equivalence of Montreal Cognitive Assessment alternate forms

Overview

Aim: The purpose of this study was to describe the psychometric properties of the Montreal Cognitive Assessment Czech version Standard form (MoCA-SF) and two Alternate Forms (MoCA-AF). There is limited information regarding the test-retest effect and the reliability of the MoCA.

Methods: Seventy cognitively healthy subjects (mean age 50.33 ± 26.47) were assessed in one session with MoCA-SF and MoCA-AF (7.2 and 7.3) in a counterbalanced fashion.

Results: There was no significant difference between the two MoCA forms (Standard vs. 7.3), MoCA-AF 7.2 being slightly more difficult than the latter. Furthermore, in depth, the exploratory analysis revealed differences between the subtests of the MoCA. However, based on the total score all versions showed sound convergent and discriminative validity.

Conclusion: Our data suggest that the two MoCA-SF and MoCA-AF 7.3 forms are equivalent and useful for repeated administration to minimize the test-retest effect.

---

等价蒙特利尔认知评估的替代形式

目的:

本研究的目的是描述蒙特利尔认知评估捷克版标准形式(MoCA-SF)和两种替代形式(MoCA-AF)的心理测量特性。关于MoCA的重测效果和信度的信息是有限的。

方法:

70名认知健康的受试者(平均年龄50.33±26.47岁)在一次MoCA-SF和MoCA-AF(7.2和7.3)平衡的情况下接受评估。

结果:

MoCA- af 7.2的难度略高于MoCA- af 7.2。进一步，探索性分析揭示了MoCA各子检验之间的差异。然而，从总分来看，所有版本的效度都表现出良好的收敛性和区分性。

结论:

我们的数据表明，MoCA-SF和MoCA-AF 7.3两种剂型是等效的，可重复给药，最大限度地降低复测效果。

关键词:

蒙特利尔认知评估。可靠性。等价性。内部一致性。替代形式

Keywords:

Montreal Cognitive Assessment – Reliability – equivalence – internal consistency – alternate forms

Introduction

There is an emerging need for brief and psychometrically sound screening instruments to measure and document global cognitive abilities in symptomatic predementia phases of neurodegenerative diseases [1–5]. The importance of measuring cognitive performance over short periods necessitates the development of psychometrically sound cognitive instruments that are suitable for test-retest purposes. Consequently, the accuracy of measurement is compromised by measurement error and practice effect, which may alter a true score. Measurement error is one particular source of chance variance that can decrease test-retest reliability [6] and the practice effect is inflation in a true score that suggests artificially improved performance. Generally, there may be a higher propensity for practice effects on repeated testing due to intact cognitive abilities [7–9]. The attenuation of practice effect may be reduced by using a different form, an alternate form, of the test for each assessment session [10]. The alternate form reliability coefficient was originally developed to minimize to the greatest extent possible influence of learning and memory processes on the test-retest assessment of cognitive functioning [11]. To evaluate consistency between form versions, a coefficient of equivalence is computed and reflects the extent to which test forms are interchangeable [12]. The reliability of alternate forms is affected primarily by mea­surement error related to the test forms [13]. The Montreal Cognitive Assessment (MoCA) is a brief screening test that assesses global cognitive function. Importantly, it has been found useful for the detection of mild cognitive impairment (MCI) and dementia [14] and as such is widely used, available in many languages [15] and recommended as a standard tool for identifying cognitive deficits in patients with neurodegenerative diseases [14,16,17]. The MoCA covers eight cognitive domains in seven subtests including executive functions, visuospatial abilities, short-term memory, language, attention, concentration, working memory and orientation [14]. The MoCA takes approximately 10 to 12 min to complete in healthy subjects [5].

In addition to the original Standard version (7.1), the MoCA has two alternate forms (versions 7.2 and 7.3) [15]. Prior research has showed that both alternate forms yielded equivalent total scores in healthy subjects and patients with MCI or Alzheimer’s disease in the original English version as well as in the German, Italian or French and other language versions [18–23], however, there were also findings regarding the differential difficulty level of the alternate versions in comparison with the corresponding original items [24]. Furthermore, there is limited evidence concerning comprehensive psychometric properties (equivalence, internal consistency) for the MoCA Alternate forms (MoCA-AF) in other languages [18,20,21,25]. Other studies reported only one or two types of reliability for the MoCA Standard forms (MoCA-SF) at a time: test-retest [26–32]; internal consistency [26–28,31,33–35]; reliable change index [9,32,36] and inter-rater reliability for the MoCA clock drawing task only [37].

The reliability of the instrument is a necessary condition for the validity of measurement [13]. As there are limited psychometrically sound cognitive screening instruments in the Czech language, we sought to study the Czech language version of the MoCA-SF and AF. Thus, the aims of the present study were to assess the reliability of MoCA-SF and AF in the Czech version to determine if they were equivalent in healthy subjects; and examine the comprehensive psychometric properties (test-retest reliability, internal consistency and split-half reliability) of the MoCA-SF and AF.

Patients and Methods

Participants

Seventy cognitively healthy Caucasian subjects (46 females) participated in the study (Tab. 1). We recruited all the subjects from university student volunteers or the National Normative Study of Cognitive Determinants of Healthy Ageing study described in detail elsewhere [5]. The exclusion criteria were as follows: no history of brain damage, psychiatric illness, substance or alcohol abuse or any medical illness that affects the brain. None of the participants reported subjective memory complaints or had impaired activities of daily living [38]. The institutional ethics committee and review board approved the study and all the subjects provided their written consent.

Materials

The MoCA-SF: The Czech version of the MoCA was approved and published by the test authors in 2006 (available online) [15]. Since that time, the Czech version has been used in three studies; two focused on Parkinson’s disease [26,39] and the other on Huntington’s disease [34]. The studies reported that the MoCA had high discriminative validity. Additional studies provided normative data in the Czech population and reliable change indices for the MoCA [5,32].

The MoCA-AF (versions 7.2 and 7.3) are available in the English language. We constructed Czech language versions of both MoCA-AF according to the original AF [15]. Our translation resulted in items identical to the English language version, except for certain items in the Language and Memory subtests (see Appendices 1–3). Specifically, two sentences in the language repetition task and five words in the memory task were adapted through the process of translation and back translation to ensure semantic equivalence to the original and the Czech Standard form used. A detailed explanation of the psychometric differences between MoCA-SF English and the Czech version can be found in Kopecek et al [5]. Alterations were created for the phonemic fluency portion of the MoCA between the English and Czech language versions. For the MoCA-AF 7.3 version, the letter ‘B’ for phonemic fluency was replaced with the letter ‘P’ because Czech normative data exist for the letters K and P, which are more suitable regarding the phonological and graphemic system of Czech [40]. The equivalency of ‘K’ and ‘P’ in Czech was confirmed by a previous study [41]. For the English language version of the MoCA-SF, phonemic fluency is assessed with the letter ‘F’ but the letter ‘K’ was used in the Czech MoCA-SF form. In the MoCA-AF 7.2 version, the letter ‘S’ remained the same as in the original because two Czech studies showed similar difficulty for the letter ‘S’ as for the letters ‘K’ and ‘P’ [40,41]. Finally, both MoCA-AF Czech versions were approved by the originators of the MoCA [15].

Procedure

The data collection design for testing the reliability of the MoCA-AF was based on counterbalancing testing order in the single session to eliminate ‘test sophistication effect’ [42]. Ran­domization was based on permutations of the forms (see Tab. 2).

Three MoCA administrations followed one immediately after the other in one session. We did not repeat the Orientation domain (data collected on the same day). Inter-rater reliability was estimated independently by two clinical neuropsychologists (O. B., T. N.). The test-retest reliability was performed only for the MoCA-SF with data collected on 30 young subjects. The time interval between testing time points was on average 42 days (range: 21–58 days). Three subjects did not complete testing at the follow-up time point due to loss of motivation.

Statistical analyses

We assessed for normality of data with the Kolmogorov-Smirnov test and by visual inspection of the Q-Q plot. Correlational analyses for continuous and normally distributed data were carried out with Pearson Product-Moment (Pearson) correlation, while the Spearman Rank correlation method was used for ranked or non-normally distributed data (with the median and interquartile range as descriptive values) and Phi correlation for dichotomous variables. One-way repeated measure analysis of variance (ANOVA) design was applied to evaluate alternate forms equivalence. For between-subtests analysis, the Wilcoxon signed-rank test was used. Significant differences (p < 0.05) were further corrected using the Bonferroni correction for post hoc comparisons to keep the Type I error at 5% overall.

The alternate forms reliability was estimated with Pearson correlation coefficients across the three test administrations. The internal consistency of the MoCA-SF and AF was estimated by Cronbach’s alpha coefficient. Split-half reliability (the correlation of scores between odd/ even subtests) was assessed using the Pearson correlation. Inter-rater reliability was estimated using intra-class correlation coefficients (ICC) using a two-way mixed model consistency definition. To assess the test-retest reliability between scores at baseline and follow-up after one month, we calculated the Pearson correlation coefficient. Finally, the relationships between age, education and practice effect between different MoCA versions were estimated using Pearson correlation coefficients and repeated measures ANOVA. All statistical procedures were performed using SPSS 20.0 for Windows (IBM, Armonk, NY, USA). The significance for all correlations was set at p < 0.05 alpha level.

Results

Alternate forms equivalence

A one-way repeated measures ANOVA was conducted to compare the total scores of MoCA-SF [1] and MoCA-AF (version 7.2 and version 7.3) administered in a counterbalanced order in the single session (see Tab. 2). The means and SD are presented in Tab. 3. There was a non-significant effect for form (Wilks’ Lambda = 0.082, F (2, 68) = 2.59, p = 0.082, multivariate partial eta squared = 0.074). Pairwise comparisons were significant in MoCA-SF vs. 7.2, p = 0.017, however, non-significant with Bonferroni adjustment for multiple comparisons and non-significant in all other comparisons (MoCA-SF vs. MoCA-AF 7.3 and 7.2 vs. 7.3; Tab. 3). The results of ANOVA were replicated by one-way repeated measures ANOVA with age and education as covariates (p = 0.012 for MoCA-SF vs. 7.2), other comparisons being non-significant (after Bonferroni correction as well). A detailed analysis of all seven subtests between the MoCA-SF and AF is presented in Tab. 4. SF and AF 7.2 and 7.3 correlation estimated with Pearson correlation coefficients was 0.77; 0.75; and 0.78, all p΄s = 0.001 (two-tailed).

Other psychometric properties

SF and AF had Cronbach’s alpha coefficient on standardized items of 0.63; 0.64; and 0.54, respectively. Split-half reliability based on the Spearman-Brown prophecy formula for un­equal length was 0.46 for SF, 0.54 for AF 2 and 0.58 for AF 3, respectively. The test-retest reliability of the MoCA-SF (range 21–58 days, M = 42, SD = 11.4) using the Pearson correlation coefficient was 0.45 (p = 0.030); we did not find any significant relation between the 21–58 day range and the test or retest MoCA-SF performance (r_test = 0.14, p = 0.555 a r_retest= 0.22, p = 0.345). The test-retest ICC of the MoCA was 0.61 (95% CI 0.12–0.83).

Discriminative validity of MoCA Standard and Alternate versions

We split the sample into young (N = 33 subjects) and old (N = 37) age groups (Tab. 1). Comparisons between groups revealed that all MoCA versions showed significant differences (all p΄s < 0.001).

Discussion

The results of the current study show that the MoCA-AF (7.2 and 7.3) in the Czech version are only partially equivalent and interchangeable with the MoCA-SF when applied in healthy subjects. The Alternate forms (especially 7.3) may be used for test--retest purposes in cases of repeated brief assessments. The MoCA-AF 7.2 seems to be more difficult than the MoCA-SF and 7.3 (e. g., the mean difference between the MoCA-SF and MoCA-AF 7.2 total score is 0.67 points and 0.46 points between MoCA--AF 7.2 and 7.3). These results replicate similar findings by Lebedeva et al [24] who selected five items from the MoCA-AF that were included with items from MoCA-SF and used the Rasch model to estimate the difficulty level of the items. However, none of the alternate version items matched the difficulty of their corresponding standard items. Hence for a deeper analysis of MoCA--SF and AF differences between subtests in the present study, we compared all the versions and showed conclusively that there were further differences between MoCA--SF and MoCA-AF 7.2 and even MoCA-AF 7.2 and 7.3 in the Language and Abstraction subtests. However, this discrepancy could not be influenced by changes in the content validity of the MoCA Czech version because all the items in Language use the same syntactic structures as the original. Surprisingly, we surmise that this finding reflects differences in English and Czech syntactical (repeats) and phonological (fluency) systems in which MoCA-AF 7.2 is more difficult than the MoCA-SF and MoCA-AF 7.3 Czech versions. These findings are in accordance with the latest and largest studies on the MoCA-AF differential item difficulty and in general with studies showing a significant language and culture bias in test instruments [21,24,43–45]. Of note is the fact that previous studies on the MoCA-AF did not report on differences between subtests [19,20,23]. All MoCA versions showed medium correlations (> 0.7) and may be considered as highly convergent. The internal consistency of all MoCA forms did not reach a minimum level of 0.7, which is a recommended threshold for Chronbach alpha [46]. The low Cronbach alpha in our study may be due to the small sample size with only healthy adults (and the presence of a ceiling effect) and the number of items on the MoCA. Indeed, psychometric research has established that Cronbach alpha values are dependent on the sample size and the number of items in a scale. For instance, fewer than ten items in a scale (MoCA΄s consistency in the current study was analyzed based on seven subtest scores) and a ceiling effect causing less variance in the data can result in small Cronbach alpha values [46]. As the authors of the original MoCA did not report the alpha values for the alternate versions [18], we were unable to compare and adequately interpret our findings. Comparable studies that used the MoCA-SF reported considerably higher alpha values [14,27,28,31,33,35]. Furthermore, ICC, which can be interpreted as another measure of test-retest reliability was good. Importantly, all MoCA versions were able to differentiate between the young and the old participants in our sample and showed similar levels of high discriminative validity in relation to the expected age effect on the MoCA.

Several important limitations of the present study must be addressed. The current study did not present the accuracy of measurement of the MoCA-AF in clinical samples in comparison with a standard neuropsychological battery. In addition, there is a lack of a medical assessment (imaging and a neurological examination) in the sample. Furthermore, separate normative values for MoCA--AF 7.2 are necessary; hence the normative data for MoCA-SF cannot be applied [5].

In conclusion, the standard MoCA Czech version and the newly constructed MoCA--AF 7.3 are psychometrically sound and were found to be interchangeable and equivalent (irrespective of form) when applied in healthy young and old subjects. However, MoCA-AF 7.2 was significantly more difficult than the latter versions. All MoCA versions have shown similar levels of convergent and discriminative validity. This has important implications for measurement-based care and repeated assessments for neurocognitive screening. Further research is warranted to determine the psychometric properties of the alternate forms in clinical populations with the comprehensive neuropsychological assessment [18–20,22].

The authors would like to express many thanks to the Prague School of Psychosocial Studies for their support. The research was supported with the following grants: Charles University, Progres Q27; Grant Agency of the Czech Republic (‘Cognitive Predictors of Neurodegeneration’, Grant No. 16-01781S); and by LO1611 from the Ministry of Education, Youth and Sport in NPU I.

The authors of the present study signed an official agreement regarding the translation of the MoCA Alternate versions into Czech (7.2 and 7.3) with copyright owner Ziad Nasreddine, M.D.

The authors declare they have no potential conflicts of interest concerning drugs, products, or services used in the study.

The Editorial Board declares that the manuscript met the ICMJE “uniform requirements” for biomedical papers.

Accepted for review: 15. 3. 2019

Accepted for print: 14. 5. 2019

Mgr. Ondřej Bezdíček, Ph.D.

Laboratoř neuropsychologie

J. Diamanta

Neurologická klinika a Centrum klinických neurověd

1. LF UK a VFN

Kateřinská 30, 128 21 Praha 2

Czech Republic

e-mail: ondrej.bezdicek@gmail.com