Summary of Cogniti Empirical Studies

Summary of Cogniti Empirical studies

Educator-designed generative AI agents in education: A synthesis of empirical Cogniti AI evidence

Overview The emerging literature on educator-designed generative AI agents suggests considerable potential to extend learning support, facilitate active learning, and provide personalised feedback at scale. However, findings across the reviewed studies indicate that educational effectiveness depends less on the availability or technical sophistication of AI agents than on their pedagogical design and integration into learning activities. Although several studies report promising learning outcomes, the evidence remains uneven, with important tensions between pedagogical intentions and student behaviour.

1. Extending access to learning support

One of the most consistently demonstrated benefits of educational AI agents is their capacity to provide immediate, discipline-specific support beyond conventional teaching hours. Studies by Camacho et al. (2026) and Mathews et al. (2025) found substantial out-of-hours usage, particularly around assessment deadlines. In nursing education, Mathews et al. reported that approximately 81% of chatbot interactions occurred outside standard daytime teaching hours. Similarly, Elliott et al. (2026) found that students valued the accessibility and non-judgemental nature of AI support, particularly when hesitant to approach teaching staff.

These findings suggest that AI agents can address temporal and interpersonal barriers to help-seeking, potentially improving access for students with competing commitments or limited opportunities for face-to-face support. Nevertheless, increased access should not be conflated with improved learning. Elliott et al. reported statistically significant improvements in perceived learning and success (both p = .04), but no significant changes in overall engagement or wellbeing. Furthermore, the out-of-hours support model primarily demonstrates the availability and uptake of assistance, rather than its independent educational effectiveness.

2. Pedagogical scaffolding: The tension between intended and actual use

Socratic questioning and scaffolded dialogue feature prominently across the studies, reflecting a deliberate attempt to position AI as a facilitator of learning rather than an answer-generation tool. Positive outcomes were reported in reflective writing (Chanane & Kuo, 2025), disciplinary reasoning (Liang, 2025), and mathematics support (Camacho et al., 2026). These studies suggest that appropriately configured agents can encourage explanation, reflection, and iterative reasoning.

However, evidence concerning students’ willingness to engage with such scaffolding is contradictory. In an analysis of 12,650 messages, Kobler et al. (2026) found that solution-seeking dominated 66.5% of conversations with pedagogically constrained agents, increasing to 92% with unrestricted AI. Students frequently ignored or attempted to circumvent Socratic guidance. Similarly, MacKay et al. (2025) documented students attempting to extract direct answers and becoming frustrated with repetitive questioning.

In contrast, Camacho et al. (2026) reported that students predominantly used mathematics agents for active learning rather than cognitive offloading. These divergent findings suggest that Socratic prompting alone does not guarantee cognitively productive engagement. Context of use, assessment pressures, task design, and the flexibility of agent responses appear consequential. The evidence appears to favour adaptive scaffolding that responds to learners’ immediate needs over rigid adherence to a prescribed conversational approach.

3. Learning gains: Promising but conditional evidence

A smaller subset of studies provides evidence beyond perceived usefulness or satisfaction. Naeyaert et al. (2025) found that students completing an AI-supported Think–Pair–Chatbot–Share activity produced more accurate final responses than those using traditional Think–Pair–Share in three of four chemistry tasks. However, participation in the AI-supported condition declined from 100% to 70.9% across tasks, revealing a potential trade-off between the benefits of additional scaffolding and the interactional effort required.

Mathews et al. (2025) similarly reported improved nursing assessment performance relative to historical cohorts, while Otemis and Caiko (2025) identified substantial gains in higher-order programming skills using an AI-enhanced learning environment. These findings are encouraging but require qualification: historical comparisons cannot isolate the effects of AI.

Other studies demonstrate the importance of task matching. Katsanos (2026) found that AI enhanced performance on complex brainstorming tasks but produced less favourable outcomes for certain divergent-thinking activities suggesting more research on suitable tasks.

4. Feedback and assessment: Scaling support without displacing judgement

Research on AI-assisted feedback identifies particular value in generating timely, structured, and discipline-specific formative guidance. Huynh et al. (2024) and Saravana et al. (2025) describe how educator-configured agents can support feedback consistency across large cohorts, while Macnaught (2025) demonstrates the potential for AI-generated feedback to reinforce disciplinary language and concepts taught in class.

Nevertheless, limitations become apparent when feedback requires nuanced interpretation. Macnaught found that rigidly designed agents struggled to interpret poorly structured texts that departed from their embedded exemplars. More significantly, Goundar and Li (2026) reported only 13% exact agreement between AI-generated and human-assigned academic writing proficiency levels, despite reasonable convergence on grammatical and mechanical errors. This distinction suggests that AI may be more reliable for identifying and explaining relatively constrained textual features than for making final academic judgements.

Vallis et al. (2026) extend this argument by demonstrating that effective feedback systems are relational rather than transactional. Students selectively combined AI and human feedback, relying on educators for contextual interpretation, standards, and judgement. Collectively, these studies support a differentiated model of feedback, in which AI facilitates initial diagnosis, formative feedback, iterative practice, and routine guidance, while educators retain responsibility for complex interpretation and consequential decisions.

5. Simulations, reflection, and the continuing role of human interaction

Beyond tutoring and feedback, educator-designed agents support pedagogical approaches that may otherwise be difficult to scale. Freeman (2025) illustrates the use of interactive role-play to explore social engineering, while Or et al. (2025) report improved perceived competence following virtual-patient history-taking practice. McHugh and Şahin (2026) further suggest that AI-mediated dialogue can create opportunities for exploring emotionally sensitive topics, with 76% of participants reporting moderate to high comfort discussing racism with an AI interlocutor.

These applications suggest that agents can create safe or low-stakes opportunities for rehearsal, experimentation, and reflection.

Importantly, the reviewed studies consistently position AI as complementary to human teaching. Despite positive evaluations of an AI laboratory assistant, 60% of surveyed students in MacKay et al. (2025) preferred interacting with human demonstrators. Similarly, Vallis et al. (2026) highlight the importance of relational feedback, while Tibbs (2025) identifies learning designer support and educator capability as critical to successful implementation.

6. Methodological limitations and research gaps

Despite their breadth of applications, the studies provide a stronger evidence base for feasibility, accessibility, and perceived usefulness than for sustained learning effectiveness. Much of the literature comprises exploratory case studies, small-scale pilots, conference reports, self-reported outcomes, and retrospective conversation-log analyses. Relatively few studies employ controlled comparisons, independently assessed learning outcomes, or longitudinal designs.

Three methodological issues are particularly important. First, engagement metrics, conversation length, and satisfaction are frequently treated as indicators of success, despite Kobler et al.’s (2026) demonstration that such measures may be poorly aligned with educational objectives. Second, interventions often combine multiple design changes, complicating attribution of learning gains specifically to AI. Third, evaluation frequently privileges students who engage with or complete AI-supported activities, potentially overlooking barriers experienced by non-users or those who disengage.

Future research should therefore examine not simply whether students use educational AI agents, but how particular design decisions and interaction patterns contribute to independently assessed learning outcomes, under what pedagogical conditions, and for which learners. Comparative investigations of scaffolding flexibility, assessment integration, student agency, and human–AI feedback arrangements would be particularly valuable.

Overall synthesis

Taken together, the studies indicate a progression from using AI agents primarily to extend access to information towards embedding them in more sophisticated pedagogical activities, including guided inquiry, formative feedback, simulation, reflection, and collaborative learning. The most persuasive evidence emerges where AI is designed to support a clearly defined learning process rather than function as a general-purpose conversational assistant.

However, the literature also reveals a fundamental tension: the pedagogical behaviours educators seek to promote through AI do not necessarily correspond to the behaviours students adopt. Addressing this mismatch requires attention not only to agent instructions and safeguards, but also to the broader learning environment, including task authenticity, assessment incentives, student motivation, and opportunities for human support.

The central conclusion is therefore not that educational AI agents inherently improve learning, but that they can expand the pedagogical possibilities available to educators when carefully designed, purposefully embedded, and critically evaluated. Their educational value should ultimately be judged by the quality of learning they enable, rather than by their availability, sophistication, or frequency of use.

Admin and Pedagogical Use

Espinosa, H. G., Whale, S., & Selvan, K. T. (2025). Work in progress: Integrating active learning and emerging AI agent support in a blended first-year engineering course. School of Engineering and Built Environment, Griffith University

Note: An example of a common ‘starting point’ for institutions – using agents to answer admin questions

Abstract: This work-in-progress paper reports on the early implementation of a blended and flipped learning approach in a first-year Electronic Engineering course, introduced to modernize the learning experience and increase student engagement. Core concepts are delivered through pre-recorded videos, while workshop sessions incorporate active learning through multi-level problem sets and real-time interactive activities that promote discussion, participation, and peer engagement. These activities aim to make problem solving more dynamic by encouraging students to share their reasoning. The course uses a continuous assessment model comprising quizzes and laboratory activities, removing the traditional final exam. Laboratories provide hands-on opportunities to reinforce weekly theoretical content. An emerging AI agent was incorporated to offer content support and respond to course-related queries. Preliminary survey responses indicate that students find the developing structure more engaging than previous formats. Ongoing work aims to refine the intervention, examine patterns of AI adoption, and evaluate its educational impact as implementation continues.

AI Use/Findings: From the perspective of AI integration and usage, the paper details the early implementation of artificial intelligence as a structured, course-bounded learning support system in a first-year undergraduate “Electric Circuits” course.

Pedagogical Strategy and Design Principles

Complementary Support: The AI tools were built on the Cogniti platform and purposefully designed as complementary learning aids aligned strictly with official course materials, rather than serving as generic or autonomous tutors.
Instructor-Centered Framework: The AI was positioned to supplement classroom instruction and lower barriers to seeking help, while keeping the human teaching staff central to the educational process
Promoting AI Literacy: Students were encouraged to critically evaluate AI outputs and seek instructor clarification when necessary, aiming to develop appropriate reliance, trust, and early AI literacy.

Implementation of the Two AI Agents The course introduced two specialized AI agents to serve distinct functions:

Administrative / FAQ Agent: Handled logistical and procedural queries, such as assessment due dates, laboratory instructions, and resource locations Used primarily by students seeking quick clarification on schedules and course requirements. Strictly constrained to access only information uploaded to the course site, preventing it from drawing on external or unverified information
Course Content / Tutor Agent: System-prompted to adopt a tutoring persona—assisting students with learning and concept explanations without simply handing out answers.

Recorded higher engagement than the FAQ agent, supporting activities like generating practice quizzes and summarizing topics