Enhancing the International Brain Laboratory's Video Processing Pipeline with Advanced Pose Estimation and Action Segmentation Tools

/abstract The International Brain Laboratory (IBL) U19 research program aims to understand how internal states influence decisions and identify their underlying neural mechanisms in mice. Achieving this goal critically depends on the rigorous and robust characterization of behavioral states. The IBL employs a modular video processing pipeline that extracts pose estimates (estimates of the positions of body parts in space) and motion energy information, and incorporates advanced features like diagnostic plots served back to users via a centralized dashboard, a versioning system to track model updates, and the flexibility to run on heterogeneous compute resources. While this robust and scalable infrastructure is state-of-the-art, the individual modules within the pipeline must be updated to improve accuracy and handle the growing diversity of the video data, which currently limits detailed behavioral analysis across studies. To address this, I will modernize the video processing pipeline by incorporating cutting-edge methods: replacing DeepLabCut with Lightning Pose for more accurate 2D pose estimation, integrating 3D pose estimation for precise tracking of paw movements using Anipose, and implementing state-of-the-art action segmentation on both 2D and 3D poses to provide richer behavioral outputs. Each new output will be integrated into existing pipelines for brainwide encoding and decoding models, ensuring that researchers have access to effective tools for linking these new behavioral descriptions to neural activity. By updating and integrating these tools into the IBL’s scalable and flexible infrastructure, this effort will support the U19 research program and advance behavioral analysis across the IBL network and the broader scientific community. Project Number: 1R50NS145433-01 | Fiscal Year: 2025 | NIH Institute/Center: National Institute of Neurological Disorders and Stroke (NINDS) | Principal Investigator: Matt Whiteway | Institution: COLUMBIA UNIV NEW YORK MORNINGSIDE, NEW YORK, NY | Award Amount: $154,308 | Activity Code: R50 | Study Section: Special Emphasis Panel[ZRG1 MCST-M (53)] View on NIH RePORTER: https://reporter.nih.gov/project-details/11240675