< Back to previous page

Publication

Academic language proficiency as a predictor of academic achievement. A validity argument of a low-stakes post-entry academic reading and vocabulary screening test for first-year university students.

Book - Dissertation

The transition from secondary to tertiary education is often conceived as difficult, into a new culture with specific rules, conventions and language use (van Kalsbeek & Kuiken, 2014; Wingate, 2015). In Flanders, where, except for Medicine and Dentistry, there are no entry requirements or selection mechanisms, the follow-up of incoming students is important. Success rates of first-year students are low, at the KU Leuven in 2019-2020 for example, just over 40% of the generation students obtained all of their credits at the end of the first year, and in the humanities, this number was even lower with only 35,4% (KU Leuven, 2021). Deygers (2017) states that in this system, the first year can be considered as the de facto selection mechanism. The increasing diversity in student backgrounds, poses new challenges to help students to 'acculturate' to academic education (van Kalsbeek & Kuiken, 2014; Wingate, 2015). An intrinsic part of students' background is their language ability. Not only international students struggle with the language requirements of university education, many domestic students experience difficulties as well. This relates to one of the corollaries of Hulstijn that individual differences in language task performance will be relatively large, also among L1ers, when lexically, syntactically and cognitively more complex language (Higher Language Cognition) is involved (Hulstijn, 2011; Hulstijn, 2015). International research clearly shows that low language proficiency can be an inhibitor for academic achievement (Davies, 2007; Elder, Bright, Bennet, 2007; Read 2015; Van Dyk, 2015). This doctoral study therefore investigates the validity of an academic reading and vocabulary (ARV) screening test developed in the Encouragement Funds project Taalvaardig aan de Start (TaalVaST - 2009-2016), in which a screening test and follow-up support for first-year students was developed. Earlier research on this screening has already found a moderate correlation between the results of the language screening and students' average exam score in January (De Wachter et al., 2013). Despite the fact that the test is very low-stakes, it is necessary that it is validated further, since thousands of students take the test each year at the start of the first year (Winke & Fei, 2008). The validation of this instrument will be investigated with six studies into different aspects of test validity, following the influential argument based approach (Kane, 1992; Kane, 2001; Kane, 2006; Kane, 2013) and will use a framework specifically developed for Post-Entry Language Assessment (PELA) by Knoch and Elder (2013). The first study will investigate the psychometric properties of the test, examining how the items contribute to the underlying attribute by examining their characteristics using both classical item analyses and Rasch analyses. Furthermore, different reliability indicators will be examined as well as the dimensionality of the instrument. The second study will investigate stakeholder perception of the screening test to investigate whether administrations conditions are clear and consistent, and how test-takers and organisers perceive the representativeness and usefulness of the screening test. The third study will look into the linguistic features of the texts used in the screening test and whether they are representative of the key first-year textbooks students have to study in the first year using automated text analysis. A fourth study will look into students' use of reading and vocabulary strategies while making the test through verbal protocol analysis (Cohen, 2016; Green, 1998). The goal is to investigate if students use the theoretically expected cognitive strategies when solving the test items thus gaining insight into the appropriateness of the tests' underlying construct. In a fifth study, both the ARV screening test and the ITNA, an official language proficiency university entrance test for L2 speakers, were administered to the group of starting criminology students. The correlations between both tests, and the ITNA's subtests of reading, listening and language in use, will be the main focus of this study. Lastly, the predictive utility of the screening test will be investigated in a sixth study by examining the relation between the screening test and students' academic achievement. Two measures of achievement will be used: students study progress, measured as their credit completion rate (CCR) and students' persistence, measured as students' time to a bachelor's degree (TTD). This study also uses students demographic and educational background variables to examine how they overlap with the ARV screening test scores. As an overall judgement following the argument, the test can be concluded to be useful in signalling possible at-risk students as lower test scores indicate a higher risk of underachievement, however, its technical limitations and predictive 'noise' restrict it to a low-stakes use, i.e., with no impact on students' university career. Also, more research into its consequences is necessary thereby further examining how the feedback format can be optimised, as well as the test's impact on students' motivation and wellbeing and the effect of the support measures following the screening test. As the instrument is quite practical in nature, further research could focus on the predictive relation of specific subcomponents of academic language proficiency. Overall, the screening test fulfils its purpose and both its interpretation and its consequences, as long as they remain low-stakes, can be considered acceptable within an open access university system.
Publication year:2021
Accessibility:Closed