# US 2018/0279921 A1 · Automatically classifying animal behavior

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0279921 A1
- Document ID: US20180279921A1
- Status: Published application
- Kind: A1
- Application number: 15/767,544
- Filed: 2016-10-11
- Published: 2018-10-04
- Earliest claimed: 2015-10-14
- Assignees: President and Fellows of Harvard College
- Applicants: President and Fellows of Harvard College
- Inventors: Sandeep Robert DATTA; Alexander Bame WILTSCHKO; Matthew J. Johnson
- Web page: https://patentgrep.com/patent/US20180279921A1

## CPC

- A61B 5/1128: {using image analysis (A61B5/1127 takes precedence)}
- A61B 5/00: Measuring for diagnostic purposes (radiation diagnosis A61B6/00; diagnosis by ultrasonic, sonic or infrasonic waves A61B8/00); Identification of persons
- A61B 5/1123: {Discriminating type of movement, e.g. walking or running (A61B5/1116, A61B5/112 take precedence)}
- A61B 5/7267: {involving training the classification device}
- G06F 18/2135: based on approximation criteria, e.g. principal component analysis
- G06F 18/295: {Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models}
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/85: {Markov-related models; Markov random fields}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- A61B 2503/40: Animals
- A61B 2503/42: for laboratory research
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19)
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- Y02A 90/10: Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE

## Abstract

Systems and methods are disclosed to objectively identify sub-second behavioral modules in the three-dimensional (3D) video data that represents the motion of a subject. Defining behavioral modules based upon structure in the 3D video data itself—rather than using a priori definitions for what should constitute a measurable unit of action—identifies a previously-unexplored sub-second regularity that defines a timescale upon which behavior is organized, yields important information about the components and structure of behavior, offers insight into the nature of behavioral change in the subject, and enables objective discovery of subtle alterations in patterned action. The systems and methods of the invention can be applied to drug or gene therapy classification, drug or gene therapy screening, disease study including early detection of the onset of a disease, toxicology research, side-effect study, learning and memory process study, anxiety study, and analysis in consumer behavior.

## Claims

### Claim 1

A method for analyzing the motion of a subject to separate it into modules, the method comprising:
processing three dimensional video data frames that represent the motion of the subject using a computational model to partition the frames into at least one set of frames that represent modules and at least one set frames that represent transitions between the modules; and
storing, in a memory, the at least one set of frames that represent modules referenced to a data identifier that represents a type of animal behavior;

### Claim 2 (depends on 1)

The method of claim 1, said processing comprises a step of isolating the subject from the background in the video data.

### Claim 3 (depends on 2)

The method of claim 2, said processing further comprises a step of identifying an orientation of a feature of the subject on a set of frames of the video data with respect to a coordinate system common to each frame.

### Claim 4 (depends on 3)

The method of claim 3, said processing further comprises a step of modifying the orientation of the subject in at least a subset of the set of frames so that the feature is oriented in the same direction with respect to the coordinate system to output a set of aligned frames.

### Claim 5 (depends on 4)

The method of claim 4, said processing further comprises a step of processing the aligned frames using a principal component analysis (PCA) to output pose dynamics data, wherein the pose dynamics data represents a pose of the subject for each aligned frame through principal component space.

### Claim 6 (depends on 5)

The method of claim 5, said processing further comprises a step of processing the aligned frames with a computational model to temporally segment the pose dynamics data into separate sets of modules wherein each of the sub-second module in a set of modules exhibits similar pose dynamics.

### Claim 7 (depends on 6)

The method of claim 6, further comprising a step of displaying a representation of each of the sets of modules that occur with a frequency above a threshold in the three dimensional video data.

### Claim 8 (depends on 1)

The method of claim 1, wherein the computational model comprises modeling the sub-second modules as a vector autoregressive process representing a stereotyped trajectory through PCA space.

### Claim 9 (depends on 1)

The method of claim 1, wherein the computational model comprises modeling transition periods between sub-second modules using a Hidden Markov Model.

### Claim 10 (depends on 1)

The method of claim 1, wherein the three dimensional video data is first processed to output a series of points in a multidimensional vector space, wherein each point represents the three dimensional pose dynamics of the subject.

### Claim 11 (depends on 1)

The method of claim 1, wherein the subject is an animal in an animal study.

### Claim 12

(canceled)

### Claim 13

A method for analyzing the motion of a subject to separate it into modules, the method comprising:
pre-processing three dimensional video data that represents the motion of the subject to isolate the subject from the background;
identifying an orientation of a feature of the subject on a set of frames of the video data with respect to a coordinate system common to each frame;
modifying the orientation of the subject in at least a subset of the set of frames so that the feature is oriented in the same direction with respect to the coordinate system to output a set of aligned frames;
processing the aligned frames using a principal component analysis to output pose dynamics data, wherein the pose dynamics data represents a pose of the subject for each aligned frame through principal component space;
processing the aligned frames to temporally segment the pose dynamics data into separate sets of sub-second modules wherein each of the sub-second module in a set of modules exhibits similar pose dynamics; and
displaying a representation of each of the sets of modules that occur with a frequency above a threshold in the three dimensional video data.

### Claim 14 (depends on 13)

The method of claim 13, wherein the processing the aligned frames step is performed using a model free algorithm.

### Claim 15 (depends on 14)

The method of claim 14, wherein the model free algorithm comprises computing an auto-correlogram.

### Claim 16 (depends on 13)

The method of claim 13, wherein the processing the aligned frames step is performed using a model based algorithm.

### Claim 17 (depends on 16)

The method of claim 16, wherein the model based algorithm is an AR-HMM algorithm.

### Claim 18

(canceled)

### Claim 19

(canceled)

### Claim 20

A method of classifying a test compound, the method comprising:
identifying a test behavioral representation that includes a set of modules in a test subject after the test compound is administered to the test subject;
comparing the test behavioral representation to a plurality of reference behavioral representations, wherein each reference behavioral representation represents each class of drugs; and
determining that the test compound belongs to a class of drugs if the test behavioral representation is identified by a classifier as matching the reference behavioral representation representing said class of drugs.

### Claim 21 (depends on 20)

The method of claim 20, wherein the test behavioral representation is identified by
receiving three dimensional video data representing the motion of the test subject;
processing the three dimensional data using a computational model to partition the data into at least one set of modules and at least one set of transition periods between the modules; and
assigning the at least one set of modules to a category that represents a type of animal behavior.

### Claim 22

(canceled)

### Claim 23

(canceled)

### Claim 24 (depends on 20)

The method of claim 20, wherein the three dimensional video data is first processed to output a series of points in a multidimensional vector space, wherein each point represents the 3D pose dynamics of the test subject.

### Claim 25 (depends on 20)

The method of claim 20, wherein the test compound is selected from the group consisting of a small molecule, an antibody or an antigen-binding fragment thereof, a nucleic acid, a polypeptide, a peptide, a peptidomimetic, a polysaccharide, a monosaccharide, a lipid, a glycosaminoglycan, and a combination thereof.

### Claim at position 26

-39. (canceled)
