Advancing AI in Higher Education: A Comparative Study of Large Language Model-Based Agents for Exam Question Generation, Improvement, and Evaluation DOI Creative Commons
Vlatko Nikolovski, Dimitar Trajanov, Ivan Chorbev

и другие.

Algorithms, Год журнала: 2025, Номер 18(3), С. 144 - 144

Опубликована: Март 4, 2025

The transformative capabilities of large language models (LLMs) are reshaping educational assessment and question design in higher education. This study proposes a systematic framework for leveraging LLMs to enhance question-centric tasks: aligning exam questions with course objectives, improving clarity difficulty, generating new items guided by learning goals. research spans four university courses—two theory-focused two application-focused—covering diverse cognitive levels according Bloom’s taxonomy. A balanced dataset ensures representation categories structures. Three LLM-based agents—VectorRAG, VectorGraphRAG, fine-tuned LLM—are developed evaluated against meta-evaluator, supervised human experts, assess alignment accuracy explanation quality. Robust analytical methods, including mixed-effects modeling, yield actionable insights integrating generative AI into processes. Beyond exam-specific applications, this methodology provides foundational approach the broader adoption post-secondary education, emphasizing fairness, contextual relevance, collaboration. findings offer comprehensive AI-generated content detailing effective integration strategies, addressing challenges such as bias limitations. Overall, work underscores potential while identifying pathways responsible implementation.

Язык: Английский

The Impact of Prompt Engineering and a Generative AI-Driven Tool on Autonomous Learning: A Case Study DOI Creative Commons
Kovan Mzwri, Márta Turcsányi-Szabó

Education Sciences, Год журнала: 2025, Номер 15(2), С. 199 - 199

Опубликована: Фев. 7, 2025

This study evaluates “I Learn with Prompt Engineering”, a self-paced, self-regulated elective course designed to equip university students skills in prompt engineering effectively utilize large language models (LLMs), foster self-directed learning, and enhance academic English proficiency through generative AI applications. By integrating concepts tools, the supports autonomous learning addresses critical skill gaps market-ready capabilities. The also examines EnSmart, an AI-driven tool powered by GPT-4 integrated into Canvas LMS, which automates test content generation grading delivers real-time, human-like feedback. Performance evaluation, structured questionnaires, surveys were used evaluate course’s impact on prompting skills, proficiency, overall experiences. Results demonstrated significant improvements accessible patterns like “Persona” proving highly effective, while advanced such as “Flipped Interaction” posed challenges. Gains most notable among lower initial though engagement practice time varied. Students valued EnSmart’s intuitive integration accuracy but identified limitations question diversity adaptability. high final success rate that proper design (taking consideration Panadero’s four dimensions of learning) can facilitate successful learning. findings highlight AI’s potential task automation, emphasizing necessity human oversight for ethical effective implementation education.

Язык: Английский

Процитировано

2
Bridging LMS and Generative AI: Dynamic Course Content Integration (DCCI) for Connecting LLMs to Course Content – The Ask ME Assistant DOI Creative Commons
Kovan Mzwri, Márta Turcsányi-Szabó

Research Square (Research Square), Год журнала: 2025, Номер unknown

Опубликована: Март 31, 2025

Abstract The integration of Large Language Models (LLMs) with Learning Management Systems (LMSs) has the potential to enhance task automation and accessibility in education. However, hallucination where LLMs generate inaccurate or misleading information remains a significant challenge. This study introduces Dynamic Course Content Integration (DCCI) mechanism, which dynamically retrieves integrates course content curriculum from Canvas LMS into LLM-powered assistant, Ask ME. By employing prompt engineering structure retrieved within LLM’s context window, DCCI ensures accuracy, relevance, contextual alignment, mitigating hallucination. To evaluate DCCI’s effectiveness, ME’s usability, broader student perceptions AI education, mixed-methods approach was employed, incorporating user satisfaction ratings structured survey. Results pilot indicate high (4.614/5), students recognizing ability provide timely contextually relevant responses for both administrative course-related inquiries. Additionally, majority agreed that reduced platform-switching, improving engagement, comprehension. AI’s role reducing classroom hesitation fostering self-directed learning intellectual curiosity also highlighted. Despite these benefits positive perception tools, concerns emerged regarding over-reliance on AI, accuracy limitations, ethical issues such as plagiarism student-teacher interaction. These findings emphasize need strategic implementation, safeguards, pedagogical framework prioritizes human-AI collaboration over substitution. contributes AI-enhanced education by demonstrating how context-aware retrieval mechanisms like improve LLM reliability educational engagement while ensuring responsible integration.

Язык: Английский

Процитировано

0
Advancing AI in Higher Education: A Comparative Study of Large Language Model-Based Agents for Exam Question Generation, Improvement, and Evaluation DOI Creative Commons
Vlatko Nikolovski, Dimitar Trajanov, Ivan Chorbev

и другие.

Algorithms, Год журнала: 2025, Номер 18(3), С. 144 - 144

Опубликована: Март 4, 2025

The transformative capabilities of large language models (LLMs) are reshaping educational assessment and question design in higher education. This study proposes a systematic framework for leveraging LLMs to enhance question-centric tasks: aligning exam questions with course objectives, improving clarity difficulty, generating new items guided by learning goals. research spans four university courses—two theory-focused two application-focused—covering diverse cognitive levels according Bloom’s taxonomy. A balanced dataset ensures representation categories structures. Three LLM-based agents—VectorRAG, VectorGraphRAG, fine-tuned LLM—are developed evaluated against meta-evaluator, supervised human experts, assess alignment accuracy explanation quality. Robust analytical methods, including mixed-effects modeling, yield actionable insights integrating generative AI into processes. Beyond exam-specific applications, this methodology provides foundational approach the broader adoption post-secondary education, emphasizing fairness, contextual relevance, collaboration. findings offer comprehensive AI-generated content detailing effective integration strategies, addressing challenges such as bias limitations. Overall, work underscores potential while identifying pathways responsible implementation.

Язык: Английский

Процитировано

0