When learning data travels, trust is tested

A student rarely fails a difficult course in a single moment. More often, difficulties build up gradually: a missed exercise here, a late submission there, less time spent with practice materials, fewer attempts to test one’s own understanding. By the time the final exam reveals the problem, meaningful support may already have come too late.
This is one of the central promises of learning analytics: digital learning environments can make some of these patterns visible earlier. But visibility alone does not make a system trustworthy. A prediction model may work well in one course and fail in another. It may identify patterns that seem precise, but depend heavily on local teaching structures. It may produce risk scores that look objective, while being poorly calibrated for the students and courses to which they are applied.

A new paper in Computers & Education addresses precisely this tension. In Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces, Dr. Jakob Schwerter and his co-authors investigate how well predictive models based on digital learning traces can identify at-risk students across different theory-intensive computer science courses – and where such models reach their limits.

The publication is also an example of how seed funding can grow into visible research output. The study emerged from a research context supported by the RC Trust-funded project “Trustworthy Didactics for Theoretical Computer Science”. The project was designed to connect educational psychology, theoretical computer science, higher education didactics, and statistics. Its aim was not simply to apply data-driven methods to education, but to ask under which conditions such methods can support learning in a trustworthy and meaningful way.

At the center of the publication is Dr. Jakob Schwerter, first author of the paper. Today, Schwerter is a postdoctoral researcher at the Hector Research Institute of Education Sciences and Psychology at the University of Tübingen. During the development of the project, he worked at TU Dortmund University, where his research connected educational research, statistics, self-regulated learning, and learning analytics. His work focuses on large-scale data, experiments, and advanced quantitative methods to better understand how educational decisions and digital learning environments can support learners and reduce educational inequalities.

The paper was written by an interdisciplinary team: Jakob Schwerter, Loreen Sabel, Judith Bose, Matthew L. Bernacki, Di Xu, Marko Schmellenkamp, Thomas Zeume, and Philipp Doebler. The authors are affiliated with the University of Tübingen, TU Dortmund University, the University of North Carolina at Chapel Hill, the University of California Irvine, and Ruhr University Bochum. This constellation reflects the larger idea behind the project: questions of trustworthy learning analytics cannot be answered from one discipline alone. They require educational theory, statistical modeling, computer science expertise, and a careful understanding of teaching contexts.

The problem the study addresses is highly relevant for universities. Dropout rates in STEM subjects remain a major concern, particularly in computer science programs with demanding theoretical courses. Students often need to manage complex tasks, abstract concepts, and substantial workloads early in their studies. Digital learning environments can provide practice opportunities, feedback, and traces of learning behavior. These traces may help instructors identify students who need support before exam results make difficulties visible.

However, the authors do not treat prediction as an easy solution. Instead, they ask a more difficult question: Can models trained in one course be trusted when applied to another?

To explore this, the study analyzed digital trace data from three undergraduate theoretical computer science courses at two universities. The models were guided by the theory of self-regulated learning, which describes how learners plan, monitor, regulate, and adapt their learning behavior. The researchers examined indicators such as time management, effort regulation, regular engagement, task submissions, and interaction with learning materials. They then tested different predictive modeling approaches, including Elastic Net, Random Forest, and XGBoost.

The results show both the potential and the limits of learning analytics. Early identification of at-risk students was possible. Behaviors related to time management, sustained engagement, and effort regulation emerged as important indicators. In other words: the way students distribute their work over time, engage with practice tasks, and interact with course materials can provide meaningful signals for later academic risk.

But the study also shows why trustworthy learning analytics requires caution. Models that performed strongly within one course did not automatically generalize across courses or institutions. Random Forest achieved high in-sample performance, but Elastic Net proved more robust when models were transferred across contexts. Prediction quality and probability calibration declined especially when models were applied between institutions with different at-risk rates.

This is more than a technical detail. Calibration matters because risk estimates can influence how support resources are allocated. If a model overestimates or underestimates student risk, interventions may miss those who need help or burden those who do not. For learning analytics to be useful, predicted probabilities must correspond to actual observed risks. For learning analytics to be fair, they must be tested in the specific contexts in which they are used.

The paper therefore offers an important message for the growing field of AI-based and data-driven support systems in education: do not mistake local accuracy for general trustworthiness. A model is not trustworthy simply because it performs well on the data it was trained on. It becomes trustworthy only when its assumptions, limits, calibration, and transferability are examined carefully.

For universities, this has practical implications. Predictive learning analytics should not be deployed as one-size-fits-all tools. They need local validation, transparent modeling decisions, and a close connection to teaching practice. The structure of a course, the timing of assignments, the incentive system, and the available learning resources all shape the meaning of digital traces. A late submission, a repeated attempt, or a missing interaction may mean different things in different learning environments.

This is where the publication speaks directly to the research agenda of the RC Trust. Trustworthy data science is not only about building powerful models. It is about understanding when models are reliable, when they fail, and how they can be used responsibly in real-world settings. In education, that responsibility is especially sensitive: the goal is not to label students, but to support learning pathways.

The RC Trust’s intramural funding helped create the conditions for this line of work. It supported a project that brought together researchers across institutions and disciplines, connecting questions of student learning with statistical modeling and trustworthy data science. The resulting publication shows how seed funding can do more than initiate a project. It can help build research environments in which new collaborations, data infrastructures, and academic careers develop.

In this sense, the paper tells two stories at once. It contributes to an international debate on the future of learning analytics. And it demonstrates how targeted funding at the RC Trust can support research that is methodologically rigorous, socially relevant, and deeply connected to the question of trust.

The lesson of the study is clear: digital traces can help universities see learning difficulties earlier. But seeing earlier is not enough. To support students responsibly, learning analytics must also know where its own predictions stop.