SLEAP: A deep learning system for multi-animal pose tracking

Nature Methods, Journal Year: 2022, Volume and Issue: 19(4), P. 486 - 495

Published: April 1, 2022

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools quantify natural animal behavior. While advances deep learning computer vision have enabled markerless pose estimation individual animals, extending these multiple animals presents unique challenges for studies of social behaviors or their environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine system multi-animal tracking. This enables versatile workflows data labeling, model training inference on previously unseen data. SLEAP features an accessible graphical user interface, standardized model, reproducible configuration system, over 30 architectures, two approaches part grouping identity We applied seven datasets across flies, bees, mice gerbils systematically evaluate each approach architecture, compare it with other existing approaches. achieves greater accuracy speeds more than 800 frames per second, latencies less 3.5 ms at full 1,024 × image resolution. makes usable real-time applications, which demonstrate by controlling one basis tracking detection interactions another animal.

Language: Английский

---

Perspectives in machine learning for wildlife conservation

Nature Communications, Journal Year: 2022, Volume and Issue: 13(1)

Published: Feb. 9, 2022

Data acquisition in animal ecology is rapidly accelerating due to inexpensive and accessible sensors such as smartphones, drones, satellites, audio recorders bio-logging devices. These new technologies the data they generate hold great potential for large-scale environmental monitoring understanding, but are limited by current processing approaches which inefficient how ingest, digest, distill into relevant information. We argue that machine learning, especially deep learning approaches, can meet this analytic challenge enhance our capacity, conservation of wildlife species. Incorporating ecological workflows could improve inputs population behavior models eventually lead integrated hybrid modeling tools, with acting constraints latter providing data-supported insights. In essence, combining domain knowledge, ecologists capitalize on abundance generated modern sensor order reliably estimate abundances, study mitigate human/wildlife conflicts. To succeed, approach will require close collaboration cross-disciplinary education between computer science communities ensure quality train a generation scientists conservation.

Language: Английский

---

Multi-animal pose estimation, identification and tracking with DeepLabCut

Nature Methods, Journal Year: 2022, Volume and Issue: 19(4), P. 496 - 504

Published: April 1, 2022

Abstract Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate association detected keypoints to correct individuals, as well having highly similar looking that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source estimation toolbox, provide high-performance animal assembly tracking—features required for multi-animal Furthermore, integrate ability predict animal’s identity assist tracking (in case occlusions). We illustrate power framework with four datasets varying complexity, which release serve benchmark future algorithm development.

Language: Английский

---

Quantifying behavior to understand the brain

Nature Neuroscience, Journal Year: 2020, Volume and Issue: 23(12), P. 1537 - 1549

Published: Nov. 9, 2020

Language: Английский

---

Revealing the structure of pharmacobehavioral space through motion sequencing

Nature Neuroscience, Journal Year: 2020, Volume and Issue: 23(11), P. 1433 - 1443

Published: Sept. 21, 2020

Language: Английский

---

Simple Behavioral Analysis (SimBA) – an open source toolkit for computer classification of complex social behaviors in experimental animals

bioRxiv (Cold Spring Harbor Laboratory), Journal Year: 2020, Volume and Issue: unknown

Published: April 20, 2020

Abstract Aberrant social behavior is a core feature of many neuropsychiatric disorders, yet the study complex in freely moving rodents relatively infrequently incorporated into preclinical models. This likely contributes to limited translational impact. A major bottleneck for adoption socially complex, ethology-rich, procedures are technical limitations consistently annotating detailed behavioral repertoires rodent behavior. Manual annotation subjective, prone observer drift, and extremely time-intensive. Commercial approaches expensive inferior manual annotation. Open-source alternatives often require significant investments specialized hardware computational programming knowledge. By combining recent advances convolutional neural networks pose-estimation with further machine learning analysis, primed inclusion under umbrella neuroethology. Here we present an open-source package graphical interface workflow (Simple Behavioral Analysis, SimBA) that uses create supervised predictive classifiers behavior, millisecond resolution accuracies can out-perform human observers. SimBA does not video acquisition nor extensive background. Standard descriptive statistical along region interest annotation, provided addition classifier generation. To increase ease-of-use behavioural neuroscientists, designed accessible menus pre-processing videos, training datasets, selecting advanced options, robust validation functions flexible visualizations tools. allows transparency, explainability tunability prior to, during, experimental use. We demonstrate this approach both mice rats by classifying behaviors commonly central brain function motivation. Finally, provide library poseestimation weights resident-intruder rats. All code data, together tutorials documentation, available on GitHub repository . Graphical abstract (GUI) creating (a) Pre-process videos supports common (e.g., cropping, clipping, sampling, format conversion, etc.) be performed either single or as batch. (b) Managing data classification projects Pose-estimation tracking DeepLabCut DeepPoseKit imported created managed within user interface, results projects. also userdrawn region-of-interests (ROIs) statistics animal movements, features (c) Create classifiers, perform classifications, analyze has tools correcting inaccuracies when multiple subjects frame, events from optimizing hyperparameters discrimination thresholds. number checkpoints logs included increased Both summary at end analysis. accepts annotations generated elsewhere (such through JWatcher) (d) Visualize several options visualizing movements ROI analyzing durations frequencies classified behaviors. See comprehensive documentation tutorials.

Language: Английский

---

B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors

Nature Communications, Journal Year: 2021, Volume and Issue: 12(1)

Published: Aug. 31, 2021

Abstract Studying naturalistic animal behavior remains a difficult objective. Recent machine learning advances have enabled limb localization; however, extracting behaviors requires ascertaining the spatiotemporal patterns of these positions. To provide link from poses to actions and their kinematics, we developed B-SOiD - an open-source, unsupervised algorithm that identifies without user bias. By training classifier on pose pattern statistics clustered using new methods, our approach achieves greatly improved processing speed ability generalize across subjects or labs. Using frameshift alignment paradigm, overcomes previous temporal resolution barriers. only single, off-the-shelf camera, provides categories sub-action for trained kinematic measures individual trajectories in any model. These behavioral are but critical obtain, particularly study rodent other models pain, OCD, movement disorders.

Language: Английский

---

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Neuron, Journal Year: 2020, Volume and Issue: 108(1), P. 44 - 65

Published: Oct. 1, 2020

Language: Английский

---

Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio

Nature Communications, Journal Year: 2020, Volume and Issue: 11(1)

Published: Sept. 11, 2020

Abstract The rhesus macaque is an important model species in several branches of science, including neuroscience, psychology, ethology, and medicine. utility the would be greatly enhanced by ability to precisely measure behavior freely moving conditions. Existing approaches do not provide sufficient tracking. Here, we describe OpenMonkeyStudio, a deep learning-based markerless motion capture system for estimating 3D pose macaques large unconstrained environments. Our makes use 62 machine vision cameras that encircle open 2.45 m × 2.75 enclosure. resulting multiview image streams allow data augmentation via 3D-reconstruction annotated images train robust view-invariant neural network. This view invariance represents advance over previous 2D tracking approaches, allows fully automatic inference on natural motion. We show OpenMonkeyStudio can used accurately recognize actions track social interactions.

Language: Английский

---

Movement-Related Signals in Sensory Areas: Roles in Natural Behavior

Trends in Neurosciences, Journal Year: 2020, Volume and Issue: 43(8), P. 581 - 595

Published: June 22, 2020

Language: Английский

---