# US 11,113,571 B2 · Target object position prediction and motion tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,571 B2
- Document ID: US11113571B2
- Status: Granted
- Kind: B2
- Application number: 16/599,513
- Filed: 2019-10-11
- Published: 2020-04-16
- Granted: 2021-09-07
- Earliest claimed: 2018-10-11
- Assignees: Kognition LLC
- Applicants: Kognition LLC
- Inventors: James Carroll; Michael Grinshpon
- Web page: https://patentgrep.com/patent/US11113571B2

## CPC

- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24: Classification techniques
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method for target object position prediction includes receiving, via an RGB camera a plurality of images depicting one or more persons positioned on a floor. A plurality of person location labels is assigned to each image indicating where the one or more persons are located relative to the floor. A foot position (FP) classifier is trained to classify the images into the person location labels, wherein the FP classifier is configured according to a multi-layer architecture and the training results in determination of a plurality of weights for connecting layers in the multi-layer architecture. A deployment of the FP classifier is created based on the multi-layer architecture, the plurality of weights, and the plurality of person location labels.

## Claims

### Claim 1

A computer-implemented method for target object position prediction, the method comprising:
receiving, via a RGB camera a plurality of images depicting one or more persons positioned on a floor;
assigning one of a plurality of person location labels to each image indicating where the one or more persons are located relative to the floor;
training a foot position (FP) classifier to classify the images into the person location labels, wherein the FP classifier is configured according to a multi-layer architecture and the training results in determination of a plurality of weights for connecting layers in the multi-layer architecture;
creating a deployment of the FP classifier based on the multi-layer architecture, the plurality of weights, and the plurality of person location labels.

### Claim 2 (depends on 1)

The method of claim 1, wherein the FP classifier is trained using coordinates of the RGB camera.

### Claim 3 (depends on 2)

The method of claim 2, wherein the FP classifier is further trained using a plurality of keypoints corresponding to the one or more persons and lines indicating connections between connected keypoints.

### Claim 4 (depends on 3)

The method of claim 3, wherein the keypoints are generated by a pose estimation (PE) classifier based on the images.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining a subset of the images where at least one of the persons is not occluded by any other objects in the images,
wherein the FP classifier is only trained using the subset of images.

### Claim 6 (depends on 1)

The method of claim 1, wherein the deployment of the FP classifier comprises one or more files describing (i) the multi-layer architecture, (ii) the plurality of weights, (iii) the plurality of person location labels.

### Claim 7 (depends on 1)

The method of claim 1, wherein the FP classifier is a convolutional neural network (CNN).

### Claim 8

A computer-implemented method for target object position prediction, the method comprising:
receiving, via an RGB camera, a plurality of images depicting a person positioned on a floor;
applying a trained pose estimation (PE) model to the images to determine a plurality of keypoints associated with the person;
determining the person's location relative to the floor by applying a trained foot position (FP) classifier to inputs comprising the keypoints and coordinates specifying a location of the RGB camera; and
providing a visualization of each person's location relative to the floor.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
identifying a plurality of connected keypoint pairs,
for each connected keypoint pair, generating a line,
wherein the inputs to the FP classifier comprise the lines generated for the connected keypoint pairs.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
analyzing the plurality of images to confirm that the person is not occluded prior to applying the FP classifier.

### Claim 11 (depends on 8)

The method of claim 8, wherein the FP classifier projects a center of gravity of the person downwards onto a plane of the floor to determine the person's location relative to the floor.

### Claim 12 (depends on 8)

The method of claim 8, wherein the FP classifier is a convolutional neural network (CNN).

### Claim 13 (depends on 12)

The method of claim 12, wherein the PE model is a CNN.

### Claim 14 (depends on 8)

The method of claim 8, wherein the visualization comprises an indication of the person on a building floor plan.

### Claim 15 (depends on 14)

The method of claim 14, wherein the visualization further comprise a depiction of the keypoints overlaid on the person in the images.

### Claim 16

A system for target object position prediction, the system comprising:
a pose estimation (PE) model trained to determine a plurality of keypoints associated with a person using images acquired with an RGB camera;
a trained foot position (FP) classifier trained to determine the person's location relative to a floor in the images based on the keypoints and coordinates specifying a location of the RGB camera; and
a visualization model configured to provide a visualization of the person's location relative to the floor on one or more displays.

### Claim 17 (depends on 16)

The system of claim 16, wherein the trained FP classifier is a convolutional neural network (CNN).

### Claim 18 (depends on 16)

The system of claim 16, further comprising:
an occlusion detection module configured to analyze images to confirm that the person is not occluded prior to applying the FP classifier.

### Claim 19 (depends on 16)

The system of claim 16, wherein the visualization comprises an indication of the person on a building floor plan.

### Claim 20 (depends on 19)

The system of claim 19, wherein the visualization further comprise a depiction of the keypoints overlaid on the person in the images.