# US 2019/0320103 A1 · Fusion of inertial and depth sensors for movement measurements and recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0320103 A1
- Document ID: US20190320103A1
- Status: Published application
- Kind: A1
- Application number: 16/453,999
- Filed: 2019-06-26
- Published: 2019-10-17
- Earliest claimed: 2015-04-06
- Applicants: The Texas A&M University System
- Inventors: Nasser KEHTARNAVAZ; Roozbeh JARARI; Kui LIU; Chen CHEN; Jian WU
- Web page: https://patentgrep.com/patent/US20190320103A1

## CPC

- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06F 18/24155: {Bayesian classification}
- G06F 18/254: {of classification results, e.g. of results related to same input data}
- G06F 18/295: {Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/85: {Markov-related models; Markov random fields}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method of recognizing movement includes measuring, by an inertial sensor, a first unit of inertia of an object. In addition, the method includes measuring a three dimensional shape of the object. Further, the method includes receiving, by a processor, a signal representative of the measured first unit of inertia from the inertial sensor and a signal representative of the measured shape from the depth sensor. Still further, the method includes determining a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

## Claims

### Claim 1

A method of recognizing movement comprising:
measuring, by an inertial sensor, a first unit of inertia of an object;
measuring a three dimensional shape of the object;
receiving, by a processor, a signal representative of the measured first unit of inertia from the inertial sensor and a signal representative of the measured shape from the depth sensor;
determining a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the determining the type of movement comprises:
initializing Hidden Markov model (HMM) parameters, including an HMM probability and a transition matrix, for a plurality of HMMs, each of the plurality of HMMs corresponding to a particular type of movement;
determining an observation sequence of the particular type of movement for the particular HMM being trained;
calculating a probability of the observation sequence;
performing a Baum Welch reestimation of the probability of the observation sequence to update the HMM;
calculating a likelihood of probability for each of the plurality of trained HMMs based on the signal representative of the measured first unit of inertia and the signal representative of the measured shape; and
selecting the type of movement corresponding to the trained HMM having the highest likelihood of probability.

### Claim 3 (depends on 1)

The method of claim 1, wherein the determining the type of movement comprises:
training a first, second, and third plurality of Hidden Markov models (HMMs), each of the first plurality of HMMs corresponding to a particular type of movement for the measured first unit of inertia, each of the second plurality of HMMs corresponding to the particular type of movement for a measured second unit of inertia, and each of the third plurality of HMMs corresponding to the particular type of movement for the measured shape;
calculating a first for each of the first plurality of HMMs based on the signal representative of the measured first unit of inertia, second likelihood of probability for each of the second plurality of HMMs based on the signal representative of the measured second unit of inertia, and third likelihood of probability for each of the third plurality of HMMs based on the signal representative of the measured shape;
pooling together the first, second, and third likelihood of probabilities to generate an overall probability for each of the first, second, and third pluralities of HMMs; and
selecting the type of movement corresponding to the trained HMM having the highest overall probability.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determining the type of movement comprises:
extracting a depth feature set from the signal representative of the measured shape;
extracting a inertial feature set from the signal representative of the measured first unit of inertia; and
fusing the depth feature and the inertial feature at a decision-level.

### Claim 5 (depends on 4)

The method of claim 4, wherein the extracting the depth feature comprises:
extracting a foreground containing the object from the signal representative of the measured shape utilizing a background subtraction algorithm to generate a foreground extracted depth image;
generating three two dimensional projected maps corresponding to a front, view of the foreground extracted depth image; and
accumulating a difference between two consecutive projected maps through an entire depth video sequence to generate a depth motion map (DMM).

### Claim 6 (depends on 4)

The method of claim 4, wherein the fusing the depth feature and the inertial feature comprises:
applying a sparse representation classifier (SRC) or collaborative representation classifier (CRC) to the extracted depth feature set and the extracted inertial feature set to generate a first and second basic probability assignments (BPAs) respectively;
combining the first and second BPAs; and
selecting the type of movement.

### Claim 7

A non-transitory computer-readable medium storing instructions that when executed on a computing system cause the computing system to:
receive a signal representative of a measured first unit of inertia from an inertial sensor coupled to an object and a signal representative of a measured shape of the object from a depth sensor; and
determine a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

### Claim 8 (depends on 7)

The computer-readable medium of claim 7, wherein the instructions further cause the computing system to:
train a plurality of Hidden Markov models (HMMs), each of the plurality of HMMs corresponding to a particular type of movement;
calculate a likelihood of probability for each of the plurality of trained HMMs based on the signal representative of the measured first unit of inertia and the signal representative of the measured shape; and
select the type of movement corresponding to the trained HMM having the highest likelihood of probability.

### Claim 9 (depends on 7)

The computer-readable medium of claim 7, wherein the instructions further cause the computing system to:
train a first, second, and third plurality of Hidden Markov models (HMMs), each of the first plurality of HMMs corresponding to a particular type of movement for the measured first unit of inertia, each of the second plurality of HMMs corresponding to the particular type of movement for a measured second unit of inertia, and each of the third plurality of HMMs corresponding to the particular type of movement for the measured shape;
calculate a first for each of the first plurality of HMMs based on the signal representative of the measured first unit of inertia, second likelihood of probability for each of the second plurality of HMMs based on the signal representative of the measured second unit of inertia, and third likelihood of probability for each of the third plurality of HMMs based on the signal representative of the measured shape;
pool together the first, second, and third likelihood of probabilities to generate an overall probability for each of the first, second, and third pluralities of HMMs; and
select the type of movement corresponding to the trained HMM having the highest overall probability.

### Claim 10 (depends on 7)

The computer-readable medium of claim 7, wherein the instructions further cause the computing system to:
extract a depth feature set from the signal representative of the measured shape;
extract a inertial feature set from the signal representative of the measured first unit of inertia; and
fuse the depth feature and the inertial feature at a decision-level.

### Claim 11 (depends on 7)

The computer-readable medium of claim 7, wherein the measured first unit of inertia comprises acceleration data of the object.