Conference: Advancing Evidence for EdTech and AI: Expanding the Education R&D Framework

Educational technology (Edtech) and AI-enabled education tools are evolving rapidly, while state and local education agencies are under growing pressure to make adoption decisions on short timelines. Although many tools claim to be evidence-based, it is often unclear what that evidence supports, how studies should be interpreted, or what claims they can legitimately justify. Interpreting evidence requires clarity about a study’s purpose, research design, counterfactual (when applicable), outcome measures and their validity, and intended claims. Existing frameworks, such as the Common Guidelines for Education Research & Development (2013), help establish shared expectations of what different studies are intended to accomplish and what can be learned from them. However, studies of edtech and AI-enabled tools increasingly stretch the limits of these frameworks. To promote transparent expectations and a common understanding of what claims evidence from studies of edtech and AI-enabled tools can support, these convenings focus on re-examining and extending existing research and development (R&D) frameworks. This project will host two convenings to bring together methodologists, edtech and AI researchers, and developers of edtech and AI-enabled tools. The primary purposes across the set of convenings are: 1) identify the specific tensions between existing R&D frameworks and research on edtech and AI-enabled tools, 2) develop an extended R&D framework that accommodates research on edtech and AI-enabled tools, and 3) develop a Field Guide for Evidence Building in Edtech and AI that provides a framework for research on edtech and AI-enabled tools. This resource is intended to include the purposes, designs, counterfactuals, outcomes, and claims associated with different study types. The Field Guide for Evidence Building in Edtech and AI will make expectations transparent and support a more consistent understanding of how studies in this area should be interpreted and what claims their evidence can support. The goal is to improve the quality, coherence, and usability of science in this domain. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2622121 | Program: 04002526DB NSF STEM Education | Principal Investigator: Jessaca Spybrook | Institution: University of South Carolina at Columbia, COLUMBIA, SC | Award Amount: $199,695 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2622121 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2622121.html