ERI: Gesture2CAD: Multimodal AI for Gesture-Driven Parametric CAD Modeling

This Engineering Research Initiation (ERI) award supports research that focuses on the development of a novel gesture-driven multimodal shape description system, Gesture2CAD, to enable immersive, explainable, and controllable human-AI collaborative design for generating high-quality parametric CAD models from natural hand gesture input. As manufacturing transitions toward a human-centric Industry 5.0, new forms of interaction are required to empower workforce creativity and relieve them from repetitive tasks. While prior work has explored hand gestures for sketching or specifying basic geometric forms, its potential as a continuous, structured input modality for conveying complex design intent remains largely unexplored. This project advances fundamental knowledge across gesture-based design representation, AI-driven CAD modeling, immersive design in extended reality (XR), and explainable models to support human-AI collaborations. This ERI award advances theory and methods for intelligent CAD modeling through three research activities: (1) developing the first multimodal CAD dataset integrating gestures and text representations; (2) establishing a gesture-driven generative approach capable of translating natural hand movements into parametric CAD commands and expert-level text prompts; and (3) enabling immersive and explainable human-AI collaborative CAD modeling using hand gestures in XR. Feedback from expert designers and rigorous empirical studies with participants of varying CAD design experience will guide system design and assess usability. This research includes advancing the design and development of the first AI-aided parametric CAD generation system from gesture input, supporting a new paradigm in intelligent CAD modeling. The project aims to produce high-quality, editable 3D outputs compatible with standard CAD tools, enabling direct integration with existing design workflows and lowering the entry barriers to parametric CAD. The project will also release open-source gesture-CAD data and methods to advance human-centered AI, manufacturing, and healthcare applications. In parallel, the project will enhance STEM education and workforce development by integrating Gesture2CAD into coursework to support hands-on, interactive CAD learning across STEM and design disciplines. Outreach activities will include interactive workshops to engage middle to high school students in engineering design, AI, and XR, as well as partnerships with a local art school to give secondary students the opportunity to explore AI-assisted CAD modeling, thereby increasing participation and strengthening pathways into the future STEM workforce. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2552514 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Di Qi | Institution: Chapman University, ORANGE, CA | Award Amount: $199,875 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2552514 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2552514.html