In May 2022 I’d recently started a Master of Studies in English Language Assessment run by Cambridge University and Cambridge English Assessment. For some time I’d been interested in mobile assessment and as I read the literature it became clear that there was very little research into the use of mobile devices for language testing. So I began to dig deeper and explore whether a test taker’s performance might be affected by their choice of device (mobile v. desktop).
Research from ten or more years ago suggested that mobile scores might be lower in some cases. But this was from a time before good mobile-responsive design, when mobile users had to interact with content that was simply shrunk down from a webpage designed for a PC. Now that applications are routinely designed for both mobile and desktop from scratch, the evidence indicates that the choice of device has no effect on scores, at least for tests where a single item is presented on a screen at a time.
In language tests, however, not all tasks fit onto a single mobile screen. With reading tasks, in particular, where there is a text of 200-plus words followed by several questions, the mobile user has to scroll down and then back up, or swipe between panels, to find information in the text and in the questions. Is it possible that such behaviour might create additional cognitive load, with a consequent effect on reading efficiency and therefore test scores?
To answer this question, I devised an experiment to measure test takers’ performance on extended reading tasks in ClarityEnglish’s Dynamic Placement Test. With the support of Kima Huang, Clarity’s partner in Taiwan, 136 university students at B1 and B2 level were recruited for the experiment. Students were divided into two groups and both groups took two parallel ‘forms’ (A and B) of an online reading test – that is, two equivalent versions of the test, each consisting of six extended reading tasks at the same level of difficulty. The first group took form A on mobile and form B on desktop; the second group took form B on mobile then form A on desktop. This experiment allowed scores to be compared according to choice of device types. And by reversing the order in which forms were taken by each group, it was possible to check that the results were not affected by the order in which students took each form (they weren’t).
The findings showed a very slight difference in scores between desktop (average score 50.4%) and mobile (average score 49.5%). However, the difference was so small (less than 1%) as to be not statistically significant – in other words, it was a level of difference that you would expect when comparing scores between any two groups of this size and couldn’t be attributed to the choice of device. The reason why there is no difference in performance is almost certainly the implementation of good UX design which makes it equally easy for test takers to interact with desktop and mobile versions. For a teacher or administrator using the Dynamic Placement Test, this is a really important finding, confirming that they can feel confident in the reliability of DPT results regardless of the test taker’s use of mobile or desktop device.
There was one finding that suggested a small difference in experience between desktop and mobile users. This related to timing, with completion times for the test being slightly shorter on desktop (13.6 minutes) than on mobile (15.1 minutes). Using statistical tools to measure correlation, I was able to show that the slightly longer completion time on mobile did not lead to lower scores; in other words, although it took test takers slightly longer to complete the test on mobile, they still achieved the same score as on desktop. The key takeaway for test providers is that the overall time limit for a test must be set with mobile users in mind, to ensure that they are not unfairly timed out.
The findings of the research have implications beyond the individual case of DPT. In some parts of the world, people have limited ownership of desktops or laptops, with as many as 75% of users in Africa relying on mobile devices to get online. Although language testing has begun to move online, to date most high-stakes tests such as IELTS and TOEFL have been restricted to desktops or laptops. However, as online testing grows in importance, we need to be aware of the issue of equality – is it fair to exclude large numbers of learners for whom an English language qualification could be a life-changing event when their only means of access is via a mobile device?
