Research

We advance the frontier of how AI and other technologies can be adapted to tackle foundational biological problems that would not be tractable otherwise.

Our approach combines technical expertise in deep learning, software engineering and computational modeling with deep biological domain knowledge and tight integration with experimental research programs. Our core interests are neuroethology (natural behavior and its neural mechanisms), degenerative diseases (especially ALS and Alzheimer's disease) and systems biology (connecting genotype to phenotype).

The lab works in three directions, each with its own workflow, methods and measures of success. Motion capture feeds the other two: better data and analytics for phenoinformatics, and better models and theories for virtual biology.

Biological motion capture

Goal Advance and validate the computational techniques and new technologies that measure how living things move, and build them into tools that scientists can rely on.

SLEAP across species: flies, bees, mice and gerbils.

We develop deep learning and computer vision methods for motion capture of living things: finding the body parts of animals in ordinary video, following each individual over time, and reconstructing how they move in 3D. This is how we quantify behavior (Pereira et al., Nature Neuroscience, 2020), and it is the foundation for the rest of the lab’s work.

A new algorithm or AI model is like a new measurement instrument: making sure it works on new data is not trivial. Many systems make implicit assumptions about their data, so they underperform on someone else’s or, worse, appear to work while introducing systematic biases that compromise the biology downstream. And a sound method still has to be usable. It takes a tight feedback loop with the people who use a tool to make sure it adds to their throughput rather than handing them the troubleshooting.

So we build our methods as open-source software, with the engineering and FAIR practices that keep it maintainable, reproducible and easy to use. We measure impact by what the tools let other scientists do, as well as by publications.

Pose estimation

Our work on pose estimation began with LEAP (LEAP Estimates Animal Pose), a deep network that learns to find an animal’s body parts in video from a small number of labeled frames (Pereira et al., Nature Methods, 2018). Its successor, SLEAP, is our flagship tool: an open-source framework for the entire multi-animal pose tracking workflow, from labeling a few frames to training models and tracking animals through video (Pereira et al., Nature Methods, 2022). It has been developed and maintained in the lab since.

Making deep learning-based motion capture accessible has created an ecosystem that builds on it. With the behavior quantification community, we set out best practices for using this technology (Luxem et al., eLife, 2023). We have also contributed to methods downstream of pose tracking: unsupervised behavior segmentation with Keypoint-MoSeq (Weinreb et al., Nature Methods, 2024), and imputation of missing movement data with DISK (Rose et al., Nature Methods, 2025).

Two fruit flies tracked with SLEAP: the video with predicted poses, and the poses alone.

Identity tracking

Reliably linking each detection to the right individual over time is a hard, unsolved problem. A landmark can be a few pixels off, but an identity switch is an all-or-none error. DREEM (DREEM Relates Every Entity’s Motion) uses a transformer to learn how to combine visual, spatial and temporal cues when deciding who is who. It takes detections as keypoints, bounding boxes or segmentation masks, and comes with pretrained models for both animals and microscopy.

3D pose

Some behavior cannot be resolved from a single view, social behavior most of all. We are extending motion capture to 3D with multiple cameras, which for more than one animal means working out which detection in each view belongs to which individual. LUC3D annotates poses across several cameras at once, in the browser, and triangulates them into 3D (Leonardis et al., bioRxiv, 2026).

Four mice reconstructed in 3D with LUC3D: from the camera rig into one camera's view and back.

Phenoinformatics

Goal Develop and implement systems for large-scale phenotyping experiments.

Three mice tracked in their home cage, their trajectories stacked over time behind the video.

Bioinformatics develops and applies computational methods to acquire, store and analyze biological sequence data. Phenoinformatics does the same for phenotypes: what an organism looks like and how it behaves. Unlike most sequence data, our primary data are images, sampled continuously over long periods, and that sets our methods apart from those of bioinformatics. We combine hardware, software and analysis to scale SLEAP and other methods to high-throughput studies of phenotypic traits.

These pipelines also support collaborations in which behavioral analysis is one part of a larger study, on questions from social isolation and alcohol drinking (Patel et al., Nature Neuroscience, 2026) to cooperative behavior (Cheng et al., Nature, 2026). Our goal for the coming years is to build the analytical and informatics frameworks that mine these large phenotypic datasets for biological insight.

Home cage monitoring

Short behavioral assays miss what continuous recording catches. In developing gerbils, monitoring families around the clock revealed structured social behavior well before the animals become independent (Mitelut et al., PLoS Biology, 2025).

With experimental labs at Salk, we have been scaling a home cage monitoring system for longitudinal phenotyping of rodents. It records around the clock and uses high-resolution pose tracking and downstream analysis to extract behavioral biomarkers of aging and disease.

Neurodegenerative disease

We are using home cage monitoring to map how behavior changes over the course of disease in mouse models of ALS and Alzheimer’s disease. Dense, longitudinal recording is what makes it possible to detect the earliest signs of disease and to resolve differences between individuals. Early behavioral biomarkers would support fast and sensitive screening of candidate therapeutics, and what we learn about the mechanisms of disease could carry over to diagnosis and intervention.

Plant roots

With the Busch lab at Salk, we developed sleap-roots, a toolkit for extracting the traits that characterize root system architecture in developing crop plants. It was the first approach to use pose estimation for root phenotyping (Berrigan et al., Plant Phenomics, 2024).

This work is part of the Salk Harnessing Plants Initiative, which is looking for the genetic basis of traits that enhance carbon sequestration in crop plants. With the Michael and Busch labs, we are developing Bloom, a data lakehouse that brings phenotypic and genetic data together so that both can be analyzed and reused.

Roots of Arabidopsis, soybean, canola and rice, imaged from every side as their cylinders turn. SLEAP's keypoints turn with them: main roots as lines, side roots as points, teal at the base to yellow at the tip.

Behavior in spaceflight

Supported by NASA Space Biology, we developed a system for phenotyping rodent behavior in microgravity, and applied it to video of mice aboard the International Space Station from the Rodent Research-1 mission (Kiffer et al., bioRxiv, 2026). Animal models like these will inform the next generation of human space exploration (Duporge et al., npj Microgravity, 2025).

Human movement

We are extending these systems to human motor phenotyping, with the aim of clinically relevant measures of movement in conditions such as autism, ALS and Alzheimer’s disease.

Virtual biology

Goal Develop computational models that emulate biological systems faithfully enough to permit in silico experiments: ones that generate hypotheses we can test in the lab, and that are refined by the outcomes.

Rendered body models of a hooded rat, a fruit fly, a stick insect, a translucent green C. elegans worm and a mouse forelimb with red muscles, posed on a blue checkered arena against black.
The five MIMIC-MJX body models, each posed from a frame of real motion.

Bioinformatics aims to capture, store and process biological data at scale. Computational biology develops mathematical and statistical models that reflect the principles underlying biological processes. Virtual biology is a frontier field that aims to emulate biology, with the explicit goal of simulating the experiments that could be done in the lab while explaining all of the available data.

This has only become possible with AI and with modern methods for scalable numerical simulation of physical processes. By aligning in silico experiments more closely with in vivo ones, and by building tools that make them accessible to experimental biologists, we believe discovery can be accelerated through hypotheses that are computationally guided and informed by data. In time, this could be coupled with lab automation for efficient high-throughput experimentation.

This is the most ambitious direction of our research program, and there is much left to do to validate the approach.

Embodied neuromechanics

MIMIC-MJX is a framework for building digital twins that learn the neural computations required to imitate the body language of real animals. Deep neural networks are trained to work like the brain: they observe what a biological movement looks like, and are rewarded for producing the control signals that drive a biomechanical body, through physics simulation, to move the way the real animal did.

Building on work from Google DeepMind and our collaborators at Harvard, we generalized the approach to work with any body model, adapted it to GPU-accelerated physics simulators, and validated it across behaviors, species and levels of biomechanical fidelity (Zhang et al., arXiv, 2025). STAC-MJX fits a body model to motion capture data, and track-mjx trains the control policies that imitate it, for rodent, fruit fly, mouse arm, stick insect and worm models.

In a companion study, we trained neural control models to reproduce the movements of mice performing a dexterous reaching task. Models that controlled muscle-based actuators predicted recorded muscle activity better than models with less biologically realistic torque actuators (Leonardis et al., NeurIPS Workshop, 2025): evidence that detailed biomechanics matter when modeling sensorimotor control.

MIMIC-MJX: a rat, a fruit fly, a mouse forelimb, a worm and a stick insect, each beside its biomechanical model.

Neural networks as models of brains

Our first step into in silico modeling asked how the visual diet of artificial neural networks affects how well they predict biological ones. Networks trained on images with more naturalistic statistics better predicted neural activity along the ventral visual stream of mice shown the same stimuli (Prasad et al., NeurIPS Workshop, 2022).

See Publications for everything the lab has published, and Tools for the software behind it.