NSF TTP-T: Technology Platform for Sign Language Data Collection, Annotation, and Recognition

This project is funded through the NSF Translation to Practice (TTP) program, which supports efforts to translate research discoveries into innovations that benefit communities, industry, and society. For the TTP program, teams advance research results toward real-world deployment and adoption. Large language models are advanced AI systems trained on vast amounts of text to understand and generate human‑like language, enabling tasks such as answering questions, summarizing information, drafting text, and assisting with analysis. Large language models have transformed daily life by making information access, communication, learning, and many routine tasks faster, more personalized, and more accessible. While there are many benefits of large language models, these systems work almost exclusively with spoken and written languages; they are not effective for signed languages. This prevents communication between the many deaf and hard of hearing people who use sign language to communicate and hearing people who need to communicate with sign language users. Sign language translation tools have enormous impact, enabling sign language and hearing users to communicate effectively in education and in the workforce. There is a critical need for systems that efficiently create and label sign language datasets to enable translation between sign language and written or spoken language. The team addresses this problem by building open-source tools that make it easier to collect, organize, and share sign language videos and use large language models to learn from these data. Removing technical barriers and creating shared standards allows researchers, educators, startups, and technology companies to build sign language technologies to meet the needs of people using spoken, written, or signed languages to effectively communicate with one another. The project creates a unified cyberinfrastructure for signed languages. First, a Lexicon and Data Management system assigns permanent identifiers to signs and uses standard data formats. This allows for combination and comparison across datasets from different research teams and sign languages. Second, a Sign Language Machine Learning tool recognizes signs in videos and returns the matching sign entries. Third, a website for collecting videos and human-in-the-loop AI labeling, speeds up data collection. Fourth, the Interactive Visualization Tools let researchers explore how signs are organized across different sign languages. The platform is an open-source software with complete technical guides, for use by researchers, nonprofits, government organizations, and companies. By transforming fragmented research tools into an open, standards-based cyberinfrastructure, the platform enables translation and communication technologies that augment limited and costly human interpreting services, which currently total an estimated $1.2 billion annually in the U.S. alone. The resulting capabilities support improved educational access, workforce productivity, healthcare communication, and public safety, while extending translation into everyday contexts such as digital media and informal interaction that are typically unsupported. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2613133 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Naomi Caselli | Institution: Trustees of Boston University, BOSTON, MA | Award Amount: $1,199,999 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2613133 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2613133.html