Research
We advance the frontier of how AI and other technologies can be adapted to tackle foundational biological problems that would not be tractable otherwise.
Our approach combines technical expertise in deep learning, software engineering and computational modeling with deep biological domain knowledge and tight integration with experimental research programs. Our core interests are neuroethology (natural behavior and its neural mechanisms), degenerative diseases (especially ALS and Alzheimer's disease) and systems biology (connecting genotype to phenotype).
The lab works in three directions, each with its own workflow, methods and measures of success. Motion capture feeds the other two: better data and analytics for phenoinformatics, and better models and theories for virtual biology.
Biological motion capture
Advance and validate the computational techniques and new technologies that measure how living things move, and build them into tools that scientists can rely on.
We develop deep learning and computer vision methods for motion capture of living things: finding the body parts of animals in ordinary video, following each individual over time, and reconstructing how they move in 3D. This is how we quantify behavior (Pereira et al., Nature Neuroscience, 2020), and it is the foundation for the rest of the lab’s work.
A new algorithm or AI model is like a new measurement instrument: making sure it works on new data is not trivial. Many systems make implicit assumptions about their data, so they underperform on someone else’s or, worse, appear to work while introducing systematic biases that compromise the biology downstream. And a sound method still has to be usable. It takes a tight feedback loop with the people who use a tool to make sure it adds to their throughput rather than handing them the troubleshooting.
So we build our methods as open-source software, with the engineering and FAIR practices that keep it maintainable, reproducible and easy to use. We measure impact by what the tools let other scientists do, as well as by publications.
Pose estimation
Our work on pose estimation began with LEAP (LEAP Estimates Animal Pose), a deep network that learns to find an animal’s body parts in video from a small number of labeled frames (Pereira et al., Nature Methods, 2018). Its successor, SLEAP, is our flagship tool: an open-source framework for the entire multi-animal pose tracking workflow, from labeling a few frames to training models and tracking animals through video (Pereira et al., Nature Methods, 2022). It has been developed and maintained in the lab since.
Making deep learning-based motion capture accessible has created an ecosystem that builds on it. With the behavior quantification community, we set out best practices for using this technology (Luxem et al., eLife, 2023). We have also contributed to methods downstream of pose tracking: unsupervised behavior segmentation with Keypoint-MoSeq (Weinreb et al., Nature Methods, 2024), and imputation of missing movement data with DISK (Rose et al., Nature Methods, 2025).
Identity tracking
Reliably linking each detection to the right individual over time is a hard, unsolved problem. A landmark can be a few pixels off, but an identity switch is an all-or-none error. DREEM (DREEM Relates Every Entity’s Motion) uses a transformer to learn how to combine visual, spatial and temporal cues when deciding who is who. It takes detections as keypoints, bounding boxes or segmentation masks, and comes with pretrained models for both animals and microscopy.
3D pose
Some behavior cannot be resolved from a single view, social behavior most of all. We are extending motion capture to 3D with multiple cameras, which for more than one animal means working out which detection in each view belongs to which individual. LUC3D annotates poses across several cameras at once, in the browser, and triangulates them into 3D (Leonardis et al., bioRxiv, 2026).
Selected papers
- LUC3D: Multi-animal 3D pose tracking in the browser
- A community benchmark enables high-resolution, cross-lab quantification of mouse social behavior
- Deep Imputation for Skeleton data (DISK) for behavioral science
- Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics
- Multi-view triangulation-enabled annotation for multi-animal 3D pose in SLEAP
- Open-source tools for behavioral video analysis: Setup, methods, and best practices
- SLEAP: A deep learning system for multi-animal pose tracking
- Quantifying behavior to understand the brain
- Fast animal pose estimation using deep neural networks
Phenoinformatics
Develop and implement systems for large-scale phenotyping experiments.
Bioinformatics develops and applies computational methods to acquire, store and analyze biological sequence data. Phenoinformatics does the same for phenotypes: what an organism looks like and how it behaves. Unlike most sequence data, our primary data are images, sampled continuously over long periods, and that sets our methods apart from those of bioinformatics. We combine hardware, software and analysis to scale SLEAP and other methods to high-throughput studies of phenotypic traits.
These pipelines also support collaborations in which behavioral analysis is one part of a larger study, on questions from social isolation and alcohol drinking (Patel et al., Nature Neuroscience, 2026) to cooperative behavior (Cheng et al., Nature, 2026). Our goal for the coming years is to build the analytical and informatics frameworks that mine these large phenotypic datasets for biological insight.
Home cage monitoring
Short behavioral assays miss what continuous recording catches. In developing gerbils, monitoring families around the clock revealed structured social behavior well before the animals become independent (Mitelut et al., PLoS Biology, 2025).
With experimental labs at Salk, we have been scaling a home cage monitoring system for longitudinal phenotyping of rodents. It records around the clock and uses high-resolution pose tracking and downstream analysis to extract behavioral biomarkers of aging and disease.
Neurodegenerative disease
We are using home cage monitoring to map how behavior changes over the course of disease in mouse models of ALS and Alzheimer’s disease. Dense, longitudinal recording is what makes it possible to detect the earliest signs of disease and to resolve differences between individuals. Early behavioral biomarkers would support fast and sensitive screening of candidate therapeutics, and what we learn about the mechanisms of disease could carry over to diagnosis and intervention.
Plant roots
With the Busch lab at Salk, we developed sleap-roots, a toolkit for extracting the traits that characterize root system architecture in developing crop plants. It was the first approach to use pose estimation for root phenotyping (Berrigan et al., Plant Phenomics, 2024).
This work is part of the Salk Harnessing Plants Initiative, which is looking for the genetic basis of traits that enhance carbon sequestration in crop plants. With the Michael and Busch labs, we are developing Bloom, a data lakehouse that brings phenotypic and genetic data together so that both can be analyzed and reused.
Behavior in spaceflight
Supported by NASA Space Biology, we developed a system for phenotyping rodent behavior in microgravity, and applied it to video of mice aboard the International Space Station from the Rodent Research-1 mission (Kiffer et al., bioRxiv, 2026). Animal models like these will inform the next generation of human space exploration (Duporge et al., npj Microgravity, 2025).
Human movement
We are extending these systems to human motor phenotyping, with the aim of clinically relevant measures of movement in conditions such as autism, ALS and Alzheimer’s disease.
Software
Selected papers
- Amygdalostriatal transition zone neurons encode sustained cue responses to guide defensive behaviors
- Social isolation recruits amygdala–medial prefrontal cortex projections to escalate alcohol drinking in male mice
- Asymmetric prefrontal representations for leader–follower dynamics
- Automated behavioral segmentation and markerless pose tracking of mice during spaceflight
- Continuous monitoring and machine vision reveals that developing gerbils exhibit structured social behaviors prior to the emergence of autonomy
- Social Exclusion Amplifies Behavioral Responses to Physical Pain via Insular Neuromodulation
- The utility of animal models to inform the next generation of human space exploration
- Predicting Future Development of Stress-Induced Anhedonia From Cortical Dynamics and Facial Expression
- Fast and Efficient Root Phenotyping via Pose Estimation
- Fast estimation of plant growth dynamics using deep neural networks
Virtual biology
Develop computational models that emulate biological systems faithfully enough to permit in silico experiments: ones that generate hypotheses we can test in the lab, and that are refined by the outcomes.
Bioinformatics aims to capture, store and process biological data at scale. Computational biology develops mathematical and statistical models that reflect the principles underlying biological processes. Virtual biology is a frontier field that aims to emulate biology, with the explicit goal of simulating the experiments that could be done in the lab while explaining all of the available data.
This has only become possible with AI and with modern methods for scalable numerical simulation of physical processes. By aligning in silico experiments more closely with in vivo ones, and by building tools that make them accessible to experimental biologists, we believe discovery can be accelerated through hypotheses that are computationally guided and informed by data. In time, this could be coupled with lab automation for efficient high-throughput experimentation.
This is the most ambitious direction of our research program, and there is much left to do to validate the approach.
Embodied neuromechanics
MIMIC-MJX is a framework for building digital twins that learn the neural computations required to imitate the body language of real animals. Deep neural networks are trained to work like the brain: they observe what a biological movement looks like, and are rewarded for producing the control signals that drive a biomechanical body, through physics simulation, to move the way the real animal did.
Building on work from Google DeepMind and our collaborators at Harvard, we generalized the approach to work with any body model, adapted it to GPU-accelerated physics simulators, and validated it across behaviors, species and levels of biomechanical fidelity (Zhang et al., arXiv, 2025). STAC-MJX fits a body model to motion capture data, and track-mjx trains the control policies that imitate it, for rodent, fruit fly, mouse arm, stick insect and worm models.
In a companion study, we trained neural control models to reproduce the movements of mice performing a dexterous reaching task. Models that controlled muscle-based actuators predicted recorded muscle activity better than models with less biologically realistic torque actuators (Leonardis et al., NeurIPS Workshop, 2025): evidence that detailed biomechanics matter when modeling sensorimotor control.
Neural networks as models of brains
Our first step into in silico modeling asked how the visual diet of artificial neural networks affects how well they predict biological ones. Networks trained on images with more naturalistic statistics better predicted neural activity along the ventral visual stream of mice shown the same stimuli (Prasad et al., NeurIPS Workshop, 2022).
Selected papers
- Discrete Actions for Naturalistic Behavior in Embodied Biomechanical Animal Models
- A latent control model for realistic rodent motion
- Massively Parallel Imitation Learning of Mouse Forelimb Musculoskeletal Reaching Dynamics
- MIMIC-MJX: Neuromechanical Emulation of Animal Behavior
- The time is ripe to reverse engineer an entire nervous system: simulating behavior from neural interactions
- Exploring the role of image domain in self-supervised DNN models of rodent brains
See Publications for everything the lab has published, and Tools for the software behind it.