Written by Sha Liu (PhD student member)
“Do you think that we can use eye-tracking to find out how students respond to automated feedback generated by computer software?”
“I don’t think so…”
This conversation occurred on 8 March 2017 when I attended a research seminar at University of Bedfordshire. At the one-day seminar, all CRELLA researchers reported and reflected on their use of eye-tracking in readi
ng research. Being deeply inspired, I asked this question and was expecting others to applaud my idea of using eye-tracking in research on automated writing evaluation (AWE) systems. Surprisingly, I received the negative response which triggered a heated discussion regarding the potential use of eye-tracking in various areas of assessment research. The take-home m
essage is that researchers need to be clear about their actual goal and motive of using eye-tracking and what they can get from using such technology.
After attending the seminar, I have been questioning myself whether my proposed use of eye-tracking method is necessary. I don’t use this method because of its increasing popularity in my field of research but instead, I plan to use this scientific method because it can tell me about people’s thinking processes which may not be effectively depicted by other data elicitation methods. By tracking the participating
students’ eye-movement, I will get detailed evidence of their physical behaviour and be able to infer the cognitive writing processes underlying such behaviour. This is probably why eye-tracking has been claimed to be by far one of the best ways to understand a participant’s cognitive processes in assessment research.
However, another important message I learned from the seminar is that the use of eye-tracking is not without challenges. First, data generated by eye-tracking method is overwhelmingly large and complex. For example, a change of single word in the revision process can be tracked, which may lead to the overload of data. It may turn out to be highly demanding for me to deal with the very richness of the data generated. Second, eye-tracking evidence appear in visual form and it is challenging for me to map the visual evidence onto my participants’ cognitive processes. One possible solution, as demonstrated in previous studies, is to combine eye-tracking with stimulated recalls and interviews that target certain aspects of visual evidence. For example, what does it indicate when a student fixates more on the feedback on his use of synonyms that on other feedback? To gain insights into the cognitive processes, I can show the eye-tracking data to the participant and ask for the explanation of his physical behaviour.
Before embarking on my doctoral study, I attended the 3rd annual conference of Asian Association of Language Assessment (AALA) at Bali, Indonesia. A very-well established professor I met there suggested that I become a lumberjack who uses both axes and saws when I asked for his advice regarding my professional development. At that time, my take of the suggestion is that I should learn as many research methods (both quantitative and qualitative) as possible. After my several months of being a social science inquirer, I would add another important point into my original interpretation: one important aspect of doctoral study is about learning to use proper methods in an appropriate manner.