How to measure statistical learning ability: evidence from test reliability

Abstract: Research has considered statistical learning (SL) as a fundamental learning mechanism in cognition, for which individuals rely on the statistical regularities from visual and verbal input during information processing. Learners’ utilization of SL has been shown to impact different aspects of language development, including phonological, lexical, and syntactic development for infants, school-aged children, and adult second language learners. Take the verbal SL task as an example, participants are first exposed to a nonsensical artificial language or visual sequence for 5~10 mins and then asked to finish a 2 alternative forced choice task (2AFC). Accuracy on each trial is coded in a dichotomous manner, with 0 for incorrect and 1 for correct, and aggregated across participants to generate the mean accuracy of the group. If it is higher than chance level, it is assumed that learning has occurred. This research perspective is called the perspective of inter group differences.
Recent studies have utilized the scores of 2AFC task as participants’ SL ability; learners language development and other high cognitive skills have been predicted based on their performance in these tasks. However, this index is the result from the perspective of inter-group differences, which is suitable for judging whether the test group exhibits statistical learning effects, but not measuring the relationship between SL ability and other cognitive ability. Under this perspective of individual difference, some researchers criticized the low reliability of SL tasks and suggested that the task results are not psychometrically satisfactory. In the current study, we aimed to put forward a modified SL task that is relatively more comprehensive. Two aspects of traditional tasks have been modified; one is that we constructed learning materials with mixed-lengths targets, and another is that we employed a familiarity rating task to measure learning outcomes in addition to the 2AFC task. These two actions aimed to obtain test scores with bigger variability so that improving the reliability of task. Finally, some papers argued that visual SL task is free from linguistics experience, thus is with better reliability index; so we also compared the reliability between the visual and verbal modalities.
A total of 143 participants took part in our experiment: 38 in the artificial language A condition, 36 in artificial language B condition, 35 in visual image A condition, and 34 in visual image B condition. Two types of reliability Cronbach’s alpha coefficient and split-half reliability were computed with the reliability function in R. The results of this study are divided into three aspects. Firstly, the index of two types of reliability in the current study are better than previous studies. This indicates that the learning materials we constructed with mixed-length nonsensical words exhibit some advantages in reliability. Secondly, the results revealed that both the Cronbach’s alpha coefficient and split-half reliability of statistical learning tasks in the visual modality were higher than those in the auditory modality, which is consistent with the opinion of Siegelman (2018a). Then, the reliability of forced-choice tasks in the visual modality was higher than that of familiarity rating tasks, suggesting the results obtained from 2AFC task are more stable and consistent across participants. Additionally, scores from 2AFC task and familiarity rating task correlated with each other in verbal modality, but not in visual modality.
The current study explored the task in measuring SL ability. The results underscore the importance of using mixed-length learning materials and suggest employing visual stimuli in assessing statistical learning abilities in addition to the traditional utilization of forced-choice tasks during the testing phase. Future studies should not only focus on designing brief SL tasks for children and language disorder population that align with psychometric standards, but also rethink the cognitive mechanism underlying various SL task.