LLM Psychometrics

Large Language Models (LLMs) are increasingly used for tasks traditionally performed by human learners, such as reading, writing, problem solving, programming, and data analysis. While current evaluations rely heavily on benchmarks, mature frameworks from educational measurement — such as Item Response Theory (IRT), cognitive diagnostic models (e.g., DINA), and learning taxonomies — offer principled approaches for understanding their capabilities and limitations. This workshop explores how these theories can inform the evaluation of LLMs and human–AI collaboration, highlights divergences and alignments with human learning processes, and addresses concerns around responsible AI use in education.

Following the successful first workshop at LAK’26, the second edition will again bring together researchers from learning analytics, educational measurement, cognitive modeling, and AI. Our goal is to advance theory-grounded evaluation of LLMs and LLM-supported learners, connect researchers across disciplines, and translate evidence into responsible assessment practices.