Alternative forms parallel to the Czech versions of Rey Auditory Verbal Learning Test, Complex Figure Test and Verbal Fluency

Overview

Objective:
The study evaluates the equivalence of alternative forms of three frequently used neuropsychological testing methods in their Czech versions, i.e. the Rey Auditory Verbal Learning Test (RAVLT), the Complex Figure Test (CFT) and letter and categorical Verbal Fluency (VF).

Methods:
In a within-subject design, 59 healthy volunteers were assessed using the original test forms and their proposed alternative forms, first at baseline and then after three months. The performance of the original and alternative forms of the tests was subjected to the t-test and U-test for paired samples based on the normality of distribution.

Results:
There was no significant mean difference between the original and alternative forms of the RAVLT. In the CFT, the Taylor figure scores were higher than the original Rey-Osterrieth figure scores, suggesting that the figures are not equal in difficulty. In the letter VF, the newly proposed “BTL” set was found to be more difficult than the original “NKP”. For the categorical VF, the category of “given names” can be considered as an alternative form to the “animals” category.

Conclusions:
The RAVLT alternative forms 1 and 2 introduced in the Neuropsychological Battery of Psychiatric Center Prague are equivalent to the original RAVLT. The “BTL” VF set and Taylor Figure were found not to be equivalent to their original test forms “NKP” and Rey-Osterrieth Figure. “Given names” may be a good alternative form to the “animals” category in VF. The study contributes with normative data for the original test forms for a specific demographic sample.

Key words:
cognition – neuropsychological tests – task performance and analysis – visual perception – verbal learning

---

Chinese summary - 摘要

与捷克版Rey听觉口头学习测试，复杂图形测试和言语流利相似的替代形式

目的：

该研究评估了捷克版本中三种常用神经心理测试方法的替代形式的等价性，即Rey听觉言语学习测试（RAVLT），复杂图形测试（CFT）和字母和分类言语流利（VF）。

方法：

在受试者内设计中，59名健康志愿者使用原始测试表格及其建议的替代形式进行评估，首先在基线，然后在三个月后。 基于分布的正态性，对原始和替代形式的测试的性能进行配对样本的t检验和U检验。

结果：

RAVLT的原始形式和替代形式之间没有显着的平均差异。 在CFT中，泰勒数字得分高于原来的雷伊奥斯特里数字分数，表明这些数字在同一难度上并不相同。 在VF的字符中，发现新提出的“BTL”组比原来的“NKP”更难。 对于分类VF，“给定名称”的类别可以被视为“动物”类别的替代形式。

结论：

在布拉格精神病中心的神经心理测试中引入的RAVLT替代形式1和2等同于原始RAVLT。 发现“BTL”VF组和泰勒数字并不等同于其原始测试形式“NKP”和Rey-Osterrieth数字。“给定名称”可能是VF中“动物”类别的一种很好的替代形式。 该研究为特定人口样本的原始测试表格提供了规范性数据。

关键词：

认知 - 神经心理测试 - 任务表现和分析 - 视觉感知 - 言语学习

Introduction

Cognitive impairment is present in a broad range of neurological and psychiatric conditions. The heterogeneity of cognitive impairment in terms of the affected cognitive domains and its severity is manifested not only across various disorders [1,2], but also between patients with the same disorder [3,4]. Even in the same patient, a cognitive deficit can vary in its manifestation during the course of the disorder [5,6]. Hence, for a systematic neuropsychological assessment, repeated measurement is often required, mostly in order to delineate changes in the cognitive performance as a result of the treatment, experimental intervention, or pathological process. Major limitations to many neuropsychological methods are insufficient parallel forms of the original tests, and a lack of normative data for these alternative forms. This is a major methodological issue, as repeated measurement with the same test material poses the risk of the practice effect [7].

In this study, we focus on the Czech versions of three extensively used neuropsychological tests, i.e. the Rey Auditory Verbal Learning test (RAVLT) [8–10] (Czech version: Preiss [10]), the Rey-Osterrieth Complex Figure Test (ROCFT) [11,12] (Czech version: Košč and Novák [13]) and verbal fluency (VF) [14] (Czech version: Preiss [15]); and their parallel forms (AVLT: Preiss, Rodriguez and Laing [16], CFT: Taylor [17], and VF: based on Kopeček and Kuncová [18]).

RAVLT is used to evaluate verbal learning and memory and examines the ability to learn a list of words. The test consists of a repeated presentation and free recall of a 15-word list (list A), an interference list (list B), two post-interference free recall trials (immediate, delayed) and recognition of the target words presented with distractors. The test enables the assessment of immediate word span, total acquisition, proactive and retroactive interference, delayed recall and recognition [19]. Considering the practice effect, significant improvement to almost all measures of RAVLT appeared upon retesting after one month [20,21] and six months [22]. On the other hand, the results of studies conducted so far show that the use of alternative forms parallel to the original forms in retesting showed no improvement [20,21,23]. Currently, there are two existing alternative forms parallel to the original RAVLT form in the Czech Republic [24]. However, the equivalence characteristics of these forms have not been evaluated so far and no normative data are available yet.

ROCFT is used to examine visual perception, construction, executive function and visual memory [19]. The test stimulus is a figure composed of 18 geometric segments and uses a 36-point scoring system that reflects construction and memory funct­ions [25]. In this study, we examine the ability to copy the figure, to recall the figure after 3 and 30 min. Considering the practice effect, a different figure is recommended for retest purposes [19] as studies show that participants are able to recall the basic features of the figure even after one year [26]. Moreover, the practice effect in planning and using a specific strategy cannot be easily attenuated in a retest situation, because the assessed person is already familiar with the task demands after the first assessment. This complicates the process of issuing a relevant retest form for the ROCFT. One alternative version for the ROCFT, the Taylor figure (TCFT), has been found to be equivalently difficult in the copy trial but easier in both recall trials than the ROCFT [27,28]. Despite the easier characteristics of the TCFT, this version is often used as an alternative to the ROCFT in clinical practice in the Czech Republic. Therefore, its characteristics as an alternative form need to be examined.

The VF test examines the ability to generate a list of words and is a highly sensitive indicator for frontal brain dysfunction in terms of word recall and flexibility of the word searching process [19]. In the letter (or phonemic) VF variation, the participant has a limited time to produce as many words as possible beginning with a designated letter. In the category (or semantic) VF variation, the participant has a limited time to produce words of a given category (e. g. animals, clothing). Functional magnetic resonance imaging techniques as well as lesion studies distinguish different brain activation patterns between the two types of VF tasks. While both tasks are sensitive to frontal brain dysfunction, the letter VF is more associated with left prefrontal cortex activation [29,30] and reflects word searching strategy, psychomotor speed and speech abilities [31]. The category VF on the other hand is associated more with the temporal cortex and hippocampi activation, reflecting mainly memory for word generation and is independent of the strategy used for word generation [29,30,32]. Hence, there is a degree of specificity in distinguishing between frontal and temporal cortices and their interconnected subcortical circuit dysfunction using letters and category VF. 

The current version of the letters VF used the most frequently in the Czech Republic uses the letters N, K and P based on the original characteristics of “FAS” form of VF [15]. However, the “NKP” set is considered to be problematic, with one of the main issues being the letter “N”. The letter “N” enables using a strategy of creating negations of verbs using the Czech prefix “ne-” which is a different process of word generation compared to the letters “K” and “P” [33]. Another major problem is the difficulty in establishing a matching alternative form to “NKP”. In relation to this, several studies suggest using different letter sets, e. g. use S instead of N in the “NKP” set (for the new set: “KPS”) [33], arguing that the S letter has more similar characteristics to K and P than N; or use the “NKT” set with a good match with the “BPL” set as the alternative form [18]. While the characteristics of these new sets are promising, “NKP” is still the most frequently used letter set in clinical practice. Moreover, the practice effect in the letters VF has been found to be considerable in both foreign and Czech literature, and studies addressing this issue recommend using alternative letter sets for retest purposes [18,34,35]. Hence, a parallel letter set to “NKP” is still required.

In category VF, the most used test category in the Czech Republic is “animals” [33,36]. While normative data for categories other than animals do exist for the elderly population, equal alternative forms have not been established yet [18].

The aim of this study was to examine the equivalency of the already existing or newly proposed parallel forms and to validate their use. We hypothesized that there would be no significant difference in task performance between the original and alternative test forms in a within-subject design.

Methods

Participants

We recruited participants among university students. All of the participants signed an informed consent form. The exclusion criteria were: 1. a history of neurological and psychiatric disorders; 2. substance or alcohol abuse; 3. any medical condition affecting neurocognition; 4. age below 20 or above 30 years; 5. psychology or psychology-related field of study; 6. exposure to any psychological tests prior to the study.

Materials

Rey Auditory Verbal Leraning Test (RAVLT)

The Czech RAVLT consists of list A (target), list B (interference) and a 50-word list (target words presented with distractors) used for the recognition task. The alternative Form 1 and Form 2 with corresponding recognition task versions for the original A and B lists used in this study were introduced in “Neuropsychologická baterie Psychiatrického centra Praha (Neuro­psy-chological Battery of Psychiatric Center Prague)” (PCP) [24] and were developed based on the translation of the validated alternative forms of the RAVLT introduced by Crawford et al [20]. The alternative word lists were compared with the original lists for consistency in terms of word length and semantic characteristics.

Verbal Fluency (VF)

The Czech version of the letters VF uses the letter set “NKP” [15]. No previous Czech study has brought a matching parallel form to “NKP” and therefore we attempted to develop one. Firstly, we reviewed the frequency dictionary from the Czech corpus (http:/ / www.ujc.cas.cz/ phword) for letters with a similar frequency and studies addressing the issue so far [18,33,34]. The considered letters were B, D, L, M, T, R, S, and V. From these letters, we excluded letters and letter combinations found unsuitable in former studies (“VRS” according to Štorková et al [34]), letters D and M according to Kopeček and Kuncová leaving B, T and L as potential candidates [18]. Due to the fact that one study [18] showed a good match between the sets NKT and BPL, and our need was to propose an alternative form of “NKP”, we switched the letters T and P. Thus, we selected the letters B, T and L as potential parallel forms for “NKP”. We conducted a pilot study where 10 participants (6 females, age range 24–30 years) were asked to generate as many words as possible within 60 seconds for each letter using the standard VF instructions. The participants generated a similar sum of words for the “NKP” and “BTL” sets.

For the semantic part of VF, we utilized the most commonly used categories for categorical VF “animals” and “vegetables” (e. g. [33,36]). For alternative categories, we selected Czech “given names” and “professions”. To ensure the relevance of testing of newly proposed categories, we conducted a pilot study where 10 participants (6 females, age range 24–30 years) generated a similar sum of words for both sets of categories (“animals” + “vegetables” and “given names” + “professions”).

Complex Figure Test (CFT)

The CTF figure used in this study [13] is the original figure introduced by Rey [11]. The alternative to the ROCFT used in the present study is the commonly used figure by Taylor with the same number of segments and an identical scoring system [9].

Procedure

The study was approved by the Ethics Committee of the National Institute of Mental Health, Klecany. All of the participants were assessed at baseline randomly by the original or the alternative test form and then after three months with the opposite form. As the RAVLT has two alternative forms, we assigned the alternative Form 1 or Form 2 to the participants randomly. The tests were individually administered by trained pre-graduate psychology students under the supervision of certified clinical psychologists.

The RAVLT was administered according to the standard administration procedure from “Neuropsychologická baterie” PCP [24]. The measures of interest were: sum of recall of the 5× administered list A (RAVLT – A1–A5), recall of list B (RAVLT – B), followed by the recall of list A without prior presentation (RAVLT – A6) and delayed recall after 30 min (RAVLT – A7).

The ROCFT and TCFT were administered according the Meyers and Meyers administration procedure (copy, 3-min and 30-min delayed recall, no recognition) [25]. Both figures consisted of 18 segments and each correctly drawn and placed segment received 2 points. Participants were asked to construct a copy of the presented figure with no time restriction on a blank sheet of paper. After 3 min, the participants were asked to recall and draw the figure (immediate recall) and after 30 min the participants were asked to recall the figure once (delayed recall). For the reliability of scoring, the copy and both delayed recalls were evaluated independently by three certified psychologists. The measures of interest were: copy, recall after 3 min and recall after 30 min.

The VF test was administered according to Thurstone [14] and Preiss [15] for the letters VF and the recommended instructions from Nikolai et al. [28] for the categorical VF. In the letters VF, the participants were asked to generate as many words as quickly as possible in a given time of 60 seconds beginning with the letters N, K and P (and B, T, L for the alternative form). In the category VF, the participants had 60 seconds to produce as many words as possible from the category of “animals” and afterwards “vegetables” (“clothes” and “given names” respectively as alternatives), with all the words and names belonging to the Czech vocabulary.

Statistical analyses

All of the statistical analyses were performed using IBM SPSS Statistical software (version 23, SPSS Inc., Chicago, IL, USA). The normal distribution of the data was inspected using the Shapiro-Wilk test of normality. The t-test for paired samples was used in cases of normative data distribution and the Wilcoxon matched paired test for repeated measures was used for abnormal data distribution to compare test-retest differences, as well as group performance differences between groups defined by the order of the test-retest administration and gender. The a-level was set at 0.05.

Results

Demographic results

Fifty-nine healthy volunteers aged 20 to 30 years were recruited for the study (30 males, 29 females; average age 23,97 years; SD 2) (Tab. 1). All of the participants were university students or university graduates with non-psychology or psychology-related fields of study.

Original and alternative form comparison

Rey Auditory Verbal Leraning Test (RAVLT)

Two groups were defined by receiving the alternative form 1 (N = 33; 16 females and 17 males; average age 24.36 years; SD 1.97) and the alternative form 2 (N = 23; 13 females and 10 males; average age 23.48 years; SD 2.04). At baseline, alternative form 1 was administered 20× and alternative form 2 14×. Three participants were excluded due to their unavailability for the second measurement. No significant mean differences were found in the performance between male and female groups in the original and alternative forms (1, 2), both at baseline and after three months. There was no significant mean difference between groups defined by the order of administration of the original and alternative forms (original – alternative vs. alternative – original). Due to the fact that the effect of gender and the order of the test-retest administration was non-significant, we compared the test-retest differences of the mean performance of all of the participants in RAVLT – A1–A5, RAVLT – B, RAVLT – 6 and RAVLT – 7 and found no significant difference between the original and alternative forms 1 or 2 (Tab. 2).

Complex Figure Test (CFT)

At baseline, the ROCFT was administered 23× and TCFT was administered 34×. Considering the order of test administration, there were no significant mean differences in the copy and both recall trials between the baseline and after three months scores in both ROCFT and TCFT (e. g. TCFT administered at baseline compared to TCFT administered after three months). There were no significant differences between the groups defined by gender. The correlations of scoring of the three raters showed a strong inter-rater correlation in all three measurements (Tab. 3). Analysis of the whole group of participants revealed mean differences between the original and alternative test forms. The mean difference for the copy trial was significant in two raters; however, the differences of the raw scores were minimal. The scores of the third rater showed no significant mean difference between the copies. For recall after 3 min and recall after 30 min, the mean differences between the original ROCFT and the alternative TCFT test form were significant in all three raters (Tab. 4).

Verbal Fluency (VF)

The original “NKP” form was administered 23× and “BTL” 34× at baseline. Two participants were excluded from the sample due to their unavailability for the second measurement. The comparison of mean differences of groups defined by the order of the original-alternative form administration and by gender revealed no significant differences. The comparison of mean sum of word generation of “NKP” and of “BTL” showed a significant difference. When we compared individual letter VFs, only the “N/ B” letters were equivalent to each other as they showed no significant mean difference (Tab. 5). In category VF, the mean sum of “animals/ vegetables” and the mean sum of “given names/ professions” also showed a significant difference. The comparison of separate categories revealed that “animals” and “given names” showed no significant difference in their means.

Discussion

The main aim of this study was to evaluate whether alternative forms parallel to RAVLT, ROCFT and VF produce scores equivalent to their original versions. For the RAVLT and ROCFT, we used the already existing alternative test forms and examined their characteristics as a retest option. For the “NKP” set we examined the “BTL” set as a potential parallel retest form. A secondary output of the study is an extension of the already existing population standards for the three original tests with the collected data for a 20–30-years-old university degree sample with the perspective of expanding the standards for other demographics in the future. To confirm the equivalent difficulty of the alternative forms, we used a design in which the participants were assessed twice, once with the original form and once with the alternative form, with a 3-month delay. We compared the performance of the first and the second measurement and based on the significance of their mean differences concluded whether the forms were equivalently difficult or not.

The RAVLT parallel forms 1 and 2 adopted from Crawford et al [20] by Preiss et al [24] showed a good match as an alternative task as no significant difference between the original and both the first and the second form was found. This suggests that the alternative forms 1 and 2 can be used as a valid retest option. The limitation of these results is the relatively small sample size for both separate forms, since we had to divide the whole group approximately in half. Another issue is the specificity of the relatively homogenous experimental sample as good and stable memory capacity is a common prerequisite for obtaining or wanting a university degree, which may have influenced the overall performance. This issue should be remedied in a future study on a larger and more heterogeneous sample.

The ROCFT alternative form, the TCFT, has the same scoring system and is composed of the same number of components as the original ROCFT. However, the TCFT has been considered to be easier than the ROCFT in previous research in terms of better overall performance in recall trials [24,28]. According to Hamby et al, the TCFT replicates the visuoperceptual properties of the ROCFT but not its organizational qualities [37]. Thus, the simpler structure of the TCFT yields similar copy scores but has better delayed recall performance scores [19]. Our results are consistent with these results as participants performed significantly better in the TCFT free recall after 3 and 30 min regardless of the order of administration. As for the copy, the mean difference was also statistically significant in the scores of the two raters (Tab. 4), but also marginal in terms of raw scores [19]. In clinical practice, a raw score difference of 1 point is usually considered as non-significant. One explanation for the significant mean difference in the copy trials may be a larger sample of participants (N = 57) as larger samples more likely tend to generate statistical significance even in minimal raw score differences. Our results suggest that the TCFT can be used as a parallel form for the ROCFT in the copy trial. However, performance differences in recall trials between the TCFT and ROCFT should be taken into consideration during the interpretation of the results in a retest situation. The inter-rater reliability in the present study showed a good correlation between raters, thereby confirming the usefulness of Meyers and Meyers administration and scoring [25].

In conclusion, the results suggest that caution should be applied when using the TCFT as an alternative to the ROCFT in a research context. Other studies showed a good match between the ROCFT and its parallel form – the modified Taylor figure (mTCFT) [28,38]. In this study, we did not use the mTCFT as the Czech version has not been adopted yet. The mTCFT, already evaluated abroad, may be a good alternative form for the ROCFT, and should be investigated in further research.

The commonly used VF letter set “NKP” has repeatedly shown to be problematic in terms of developing a suitable alternative form [18,29]. In the present study, our parallel set “BTL” has shown to be more difficult than “NKP” and is therefore unsuitable as an equivalent alternative form. This finding supports the claim that “NKP” is not a good baseline option for repeated measurement. We also compared individual letter pairs of “NKP” and “BTL” in terms of word generation. Only the “N/ B” pair was found to be equivalent, which is congruent with the findings in the study [18]. This suggests that when obtaining the letters VF data in clinical practice and research, “NKP” is not suitable for repeated measurement designs as it still lacks a good equivalent letter set. Creating an entirely different set of letters together with matching parallel forms may resolve the issue for the letter VF task [33]. Interestingly, it is yet possible to use “NKP” in repeated administration after longer time periods because some studies suggest the practice effect in the letters VF is non-significant after one year [39].

Considering category VF, dual sets of “animals/ vegetables” and “given names/ professions” have been found not to be equivalent. However, when evaluated separately, the data show that there is no significant difference in means between “animals” and “given names”, suggesting that the “given names” category in the category VF task is a potential alternative to “animals”. While this result is promising, further research should be conducted on larger samples with more demographic heterogeneity.

One limitation of this study is the absence of a direct control of the practice effect. The sole purpose for developing alternative forms is to attenuate the practice effect. However, the use of alternative forms may produce effects of practice as well. As for the RAVLT and both semantic and phonemic VF, studies by Lemay et al [40] and Benedict and Zgaljardic [23] showed that different alternative forms produced practice effects comparable to the original test forms. Other studies showed that even repeated administration of alternative forms produced only marginal practice effects for VF and RAVLT [20,21,23,40,41]. According to Lezak [19,39], tests that have a large speed component require an unfamiliar or infrequent practice mode of response, or have a single solution – particularly if it can be easily conceptualized once it is attained – are more likely to show significant practice effects. As VF and RAVLT do not correspond to these test characteristics, the effect of practice is only marginal in RAVLT and VF when using alternative forms as was also supported by the studies mentioned above. Although the use of alternative forms may attenuate practice effects, they may still occur in novel tests, tests with high cognitive demands or those in which the participant learns to use an effective test-taking strategy or has acquired ”test-wiseness” [19,30]. This may be the case of CFT and is also the main reason why we consider TCFT to have reduced criterion validity as an alternative form to ROCFT. As stated above, the best remedy in order to attenuate the practice effect in CFT-like tests is to use alternative forms and longer time periods between testing [41,26]. In summary, the ROCFT, RAVLT and VF all show a major reduction in the practice effect when using parallel forms [23,26,28,39,41]. Still, it is reasonable to assume that in our study the practice effect did occur to some degree and should be addressed in following studies by including repeated administration of both the original and alternative forms.

Another major limitations of this study are the sample size, its homogeneity and its narrow demographic characteristics. Psychiatric and neurological disorders in the population spread across all education levels and gender and most have a typical age of onset. For example, the mean age of onset of major depressive disorder is 20 years [44], for schizophrenia the peak age of onset is between 20 and 25 years in men and between 25 and 30 in women, for bipolar disorder the average age of onset is 18 years [42], for Parkinson’s disease the average age of onset is 60, and 65 for Alzheimer’s disease [43]. The usefulness of our results in clinical practice is therefore limited. Moreover, the practice effect is likely to vary in different disorders even when using alternative forms. The method to attenuate the effect of practice in a clinical setting is to compare repeated performance measurements of healthy controls and specific patient groups [44]. Following this, patient group performance change differing from the performance gain of controls (the effect of practice) after repeated administration of a neurocognitive test is considered to be related to the disorder or intervention [7,44]. For example, the practice effect in memory tests and category VF in diagnosed Alzheimer’s disease is non-significant regardless of using alternative or original forms [45]. In schizophrenia on the other hand, most neurocognitive tests show the effect of practice [7]. This implies the need to examine alternative forms not just for a much broader and larger demographic sample but also for different patient groups in order to provide normative standards and the necessary diagnostic sensitivity and specificity.

While the demographic characteristics of our sample pose limits for broader conclusions, they also contribute supplement data for the existing normative standards of the original forms. Our extension is beneficial especially for the ROCFT, as normative data for this demographic group are missing. For the original letter and categorical VF and the RAVLT, the data provide extension of normative standards with the perspective of further extensions of the standards with broader demographic samples in subsequent studies.

Conclusion

In this study, we evaluated the original and alternative forms of three frequently used neuropsychological tests – the RAVLT, CFT and letter/ categorical VF. The results of the study imply that the RAVLT alternative forms 1 and 2 introduced in ”Neuropsychologická baterie” PCP are equivalent to the original RAVLT [24]. The study did not prove that “BTL” was a good alternative to “NKP”. However, we confirmed that the letters “N” and “B” are equivalently difficult and, considering the category VF, the category of “given names” might be a good alternative form to the “animals” category. As for the ROCFT, the TCFT was found to be easier than the original form and should be used with caution as an alternative test in research. The study contributes normative data for the original test forms for a specific demographic sample.

 This study is a result of research funded by the Czech Science Foundation (GAČR) (grant number 16-13093S) and project Nr. LO1611 with financial support from MEYS under the NPU I program.

The authors declare they have no potential conflicts of interest concerning drugs, products, or services used in the study.

The Editorial Board declares that the manuscript met the ICMJE “uniform requirements” for biomedical papers.

Accepted for review: 27. 6. 2017

Accepted for print: 3. 1. 2018

Mgr. Martin Paštrnák

National Institute of Mental Health

Topolová 748

250 67 Klecany

Czech Republic

e-mail: martin.pastrnak@nudz.cz