Observing how a student tries to solve a problem and deciding what to do next to advance their learning is central to teaching. But evaluating that skill — known as “professional noticing” — is challenging, and teachers rarely get feedback soon enough to use it in the moment. 

A team of researchers wants to know whether artificial intelligence (AI) may hold the key to faster, more useful feedback that can improve teaching and learning.

With support from the National Science Foundation, researchers are studying whether commercially available generative AI tools can accurately assess professional noticing in mathematics instruction and provide helpful guidance more quickly than humans.

Molly Fisher, a professor in Rowan University’s College of Education, and a team from the University of Kentucky College of Education will use Claude, ChatGPT and a third tool, to be selected, to assess noticing among 150 in-service and preservice math teachers. 

All teacher responses evaluated by AI will be de-identified, Fisher says, and no videos of students will be entered into any of the AI tools tested. The two-year, $67,875 grant ends in July 2028.

Professional noticing is the ability to attend to, interpret and make instructional decisions based on students’ thinking. Recent research has highlighted it as “a critical component of teacher expertise,” particularly in math.

The team will use data from previous studies to train AI models to recognize examples of effective professional noticing. The goal is to develop a tool that can help teachers evaluate their noticing abilities, improve instructional decision-making and better support student learning, Fisher says.

Effective professional noticing can reveal a student’s mathematical development, including their early arithmetic strategies, how they construct units, combine sets, adhere to a developmental progression and build confidence, says Jonathan N. Thomas, a principal investigator for the project and chair of the Department of Teaching and Learning at the University of Kentucky. 

Why Noticing Is Hard to Measure

Fisher has studied professional noticing for years, in one-on-one interactions between a teacher and a student and in group settings. Across that work, she says, her team has struggled to find reliable ways to evaluate a teacher’s abilities and the scoring is slow.

The typical approach is to “sit in their classroom, observe their teaching, take notes about how they respond to students and then sit down and discuss those observations after the lesson,” Fisher says. “We can get a sense of how teachers are noticing, but we can’t get good feedback to teachers quickly.”

The study aims to explore the degree to which generative AI tools can validly and reliably measure teachers’ professional noticing of children’s mathematical thinking and generate feedback that supports teacher learning. 

“It’s a deliberate choice for us to use commercially available AI tools because we want to see how these tools work and the way that educators would use them, not just in a lab scenario,” Fisher says. “We want this to have the greatest utility potentially, or at least provide information about whether this is viable as a practice.”

Fisher and her team will build the scoring materials the AI tools draw on — rubrics, decision trees and background on each noticing measure — and bundle them into standardized packages that other educators could eventually use.

The team plans to recruit about 100 in-service elementary teachers and 50 preservice elementary teachers. Each teacher will follow the three-step framework of noticing — attending, interpreting and deciding — while completing three established noticing assessments with students: two built around classroom video and one built around an in-class writing assignment.

Expert raters will score the teachers’ responses using validated protocols. AI tools will then score the same responses, using prompts the team refines through repeated testing.

The Potential for AI to Evaluate Performance

Comparing the two sets of scores will show whether the AI tools match the experts, and whether they hold up across different types of assessments. The team will also survey and interview teachers about how they interpret and act on the AI-generated feedback.

The project is still in an early phase, says Robin Jocius, a co-investigator for the project and an associate professor in the Department of Teaching and Learning at the University of Kentucky. To test the scoring packages before involving teachers, the team will use written responses mimicking real responses. Data collection with teachers will begin in spring 2027.

With research just underway, Fisher has not yet developed feedback reports. Ideally, the reports will note both strengths and growth opportunities. For example, she says, the feedback may say, “You accurately noticed that the child used his fingers to count to seven,” and add, “Be sure to note the order in which the child’s actions occur.”

Fisher’s goal is to send feedback reports to teachers within 24 hours, but the technology is not yet available to do that.

Court Shuller, a middle school English teacher in Sicklerville, New Jersey, who has written of their skepticism about AI in the classroom, is open-minded about the project but wonders how much AI would help teachers improve their noticing skills. 

“The only way I can imagine it working is ... taking a video [of the teacher noticing], creating the script of it, then analyzing specific sentence stems, etc.,” says Shuller, who adds that teaching feedback typically takes 48-72 hours. “That’s not super plausible.”

Shuller thinks the AI tool could help administrators more than teachers. “It sounds almost better as an evaluation of a school or district’s success with these skills, much in the same way that we use standardized test scores, or student growth objectives to evaluate teacher effectiveness and prepare professional development for the upcoming year,” Shuller explains.

Where the Human Element Comes In

Fisher acknowledges that some educators may resist the idea of using AI tools for evaluating professional noticing or other skills. “There are also the traditionalists who don’t think that should be possible because you’re losing that human element, like the feelings or picking up on body language,” she says.

However, she points out that professional noticing is qualitative in nature, and many researchers are already using AI to code qualitative data.

Riya Nathrani, an instructional technology coach with the Commonwealth of the Northern Mariana Islands Public School System, says she believes using AI to evaluate professional noticing could surface learning patterns more quickly and help teachers respond to student thinking sooner.

“Imagine a teacher reviewing student answers after teaching a lesson — AI could quickly flag that several students chose the right answer but gave explanations that contradicted their answers,” she says. “That gives the teacher something specific to follow up on. The teacher can ask students to talk through their reasoning and decide whether they need help with the concept.” 

Understanding the patterns uncovered by AI, she adds, requires a conversation between the student and the teacher — and that is ultimately what educators want when using AI.