Hello world! DigiErko goes international with this guest writer blog entry! Xiaoshan Huang from the University of Turku has kindly summarised the key findings and insights of a research article on AI-driven automated assessment. Thank you, Xiaoshan! The floor is yours.
In this blog post, I’d like to share insights from our recently published work on AI-driven automated assessment, specifically, how teachers and researchers can collaborate across disciplines to refine these technologies in ways that truly benefit teaching and learning. Our research examines where such collaborations often break and suggests more holistic ways to use automated grading tools responsibly.
Why automated assessment matters and where collaborations break?
You may have come across the idea of using artificial intelligence (AI) to mark student work, with the hope that it can lighten teachers’ marking workloads and speed up feedback so students receive timely guidance. However, it also raises pressing questions:
- Are AI tools reliable enough to grade student work fairly?
- How can we guarantee ethical use—for instance, by guarding against algorithmic biases?
- How do these systems affect teachers and students—and, more importantly, how can they empower teaching and learning?
Our article systematically reviews recent research on automated grading, highlighting where challenges arise and how to address them. First, accuracy alone often defined by how closely machine scores match human grades. But this is only part of the picture. Many systems focus on matching teacher marks but don’t always capture the deeper learning objectives teachers want to assess (like higher-order thinking). Second, few studies explore how algorithms might be biased or ensure that teachers and students understand why a tool arrives at a particular score. For instance, if a model favors certain linguistic patterns, it could unintentionally disadvantage some learners. Third, teachers’ roles are limited to simply providing data to train the AI model. However, a true collaboration, where educators help shape the AI remains rare.
The need for interdisciplinary collaboration
A key message from our study is that developers, educators, and students must work together throughout the design and implementation process. Before coding the AI, the models should reflect the specific competencies or skills that teachers aim to cultivate. Educators can shape how data is labeled and even refine the AI’s scoring guidelines, ensuring that the results are not only “accurate” in a statistical sense but also pedagogically meaningful. AI systems can feel like black boxes, so teachers and students should be able to see how scoring decisions are made. This openness fosters trust and provides a more informative diagnosis of students’ learning, encouraging them to reflect on their own progress.
What this means for teachers and students?
As teachers, even if you’re not directly coding AI algorithms yourself, being aware of these issues can help you make better decisions when adopting new digital solutions. One often overlooked aspect of AI-driven assessment is how teachers and students can actively shape and interpret automated feedback. Rather than viewing themselves as passive recipients of AI-generated scores, both teachers and learners can become constructive, agentic partners
For Teachers
You know best what competencies or skills they want their students to master, be it creativity, problem-solving, or critical thinking. By working actively to define rubrics and features for the AI model, teachers can evaluate if it aligns with learning objectives rather than just reflecting a machine’s abstract notion of correctness.
When the AI flags areas needing improvement for students, You can offer personalized follow-up activities or “second-chance” responses. This human-in-the-loop approach refines algorithmic grading in real time, ensuring that students receive not only a numerical score or machine-generated feedback but also valuable professional insights.
Even if automated grading speeds up routine marking, teachers should take the final decision of scoring. By integrating your professional judgment, especially for tasks involving higher-order thinking, teachers can moderate the AI’s outputs and ensure fairness and validity.
For Students
Students can be taught to interpret automated feedback, ask questions about how their work was evaluated, and decide what action to take next. This approach fosters metacognition, helping them see assessment not as a “gotcha” moment but as useful data for improvement.
When students understand the basics of how these systems work and recognize their limitations, they become better equipped to question or refine the feedback they receive. When either the AI or the teacher provides formative feedback, students can be encouraged to reflect on its accuracy by asking themselves, “Does this comment accurately capture my writing challenges?” or “What do I feel is missing?”
In short, teachers and students each bring unique perspectives and expertise,whether that’s pedagogical insight or lived learning experiences. Teachers and students act as active, constructive users throughout the lifecycle of AI-based assessments can lead to more trustworthy results, better alignment with local teaching goals, and deeper student engagement in the learning process.
Our research underscores that AI-driven assessment can be an incredible tool, but only if it’s grounded in sound pedagogy, transparency, and ongoing dialogue among teachers, developers, and learners. As AI continues to evolve, we hope more researchers and teachers will co-design solutions that genuinely enhance teaching and learning experiences.
Xiaoshan Huang, University of Turku
Reference
X. Huang, L. -H. Chang, K. Veermans and F. Ginter, ”Breakpoints in Iterative Development and Interdisciplinary Collaboration of AI-Driven Automated Assessment,” 2024 21st International Conference on Information Technology Based Higher Education and Training (ITHET), Paris, France, 2024, pp. 1-10, doi: 10.1109/ITHET61869.2024.10837673.
Valtakunnallinen DigiErko-verkosto 