# US 2019/0377935 A1 · Method and apparatus for tracking features

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377935 A1
- Document ID: US20190377935A1
- Status: Published application
- Kind: A1
- Application number: 16/488,024
- Filed: 2018-02-16
- Published: 2019-12-12
- Earliest claimed: 2017-02-22
- Assignees: Cubic Motion Limited
- Applicants: Cubic Motion Limited
- Inventors: Gareth Edwards
- Web page: https://patentgrep.com/patent/US20190377935A1

## CPC

- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/7553: {based on shape, e.g. active shape models [ASM]}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments of the present invention provide a systems and methods for tracking features. In particular, some aspects of the present invention relate to a method and system for facial modelling and a method and system for determining facial features. Embodiments of the invention comprise receiving stereo image data comprising a set of corresponding first and second stereo-rectified image frames indicative of a target; annotating the stereo image data to determine a location of an image feature in the first and second stereo-rectified image frames, wherein the determined locations in the first and second corresponding stereo-rectified image frames are positionally constrained according to an epipolar constraint; and training a shape variation model corresponding to the target according to the determined image feature locations.

## Claims

### Claim 1

A computer-implemented method of facial modelling, comprising:
receiving stereo image data comprising a set of corresponding first and second stereo-rectified image frames indicative of a target;
annotating the stereo image data to determine a location of an image feature in the first and second stereo-rectified image frames, wherein the determined locations in the first and second corresponding stereo-rectified image frames are positionally constrained according to an epipolar constraint; and
training a shape variation model corresponding to the target according to the determined image feature locations.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving stereo image test data comprising a set of first and second stereo-rectified image frames indicative of a target; and
processing the stereo image test data, wherein the processing comprises using the shape variation model to determine parameters associated with at least one image feature identified in the stereo image data.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the location of an image feature comprises marking a first point location of the image feature in the first image frame and marking a second corresponding point location of the image feature in the second image frame.

### Claim 4 (depends on 1)

The method of claim 1, wherein the shape variation model is trained to map a fixed vector of point locations, X, to a vector of model parameters, p, wherein the fixed vector of point locations, X, are indicative of the determined locations of the image feature.

### Claim 5

(canceled)

### Claim 6

A computer-implemented method of determining facial features, comprising:
receiving stereo image data comprising a set of corresponding first and second stereo-rectified image frames indicative of a target; and
processing the stereo image data, wherein the processing comprises using a shape variation model to determine parameters associated with at least one image feature, X, identified in the stereo image data.

### Claim 7

(canceled)

### Claim 8 (depends on 6)

The method of claim 6, wherein the processing comprises using the shape variation model to estimate a vector, p, of model parameters according to the identified image feature, X.

### Claim 9 (depends on 6)

The method of claim 6, wherein determining parameters associated with the at least one image feature comprises using the shape variation model to estimate at least one point location, X′, indicative of the image feature, given the vector of model parameters p.

### Claim 10 (depends on 6)

The method of claim 6, wherein the shape variation model is a Linear Point Distribution Model.

### Claim 11 (depends on 6)

The method of claim 6, wherein the image features identified in the stereo image data correspond to image features determined for training the shape variation model.

### Claim 12 (depends on 6)

The method of claim 6, wherein the features identified in the stereo image data are identified using a profile matching algorithm.

### Claim 13 (depends on 12)

The method of claim 12, wherein the profile matching algorithm uses an Active Shape Model.

### Claim 14 (depends on 12)

The method of claim 12, wherein the profile matching algorithm comprises tracking local patches in a regression framework.

### Claim 15

A system for facial modelling, comprising:
input means for receiving stereo image data comprising a set of corresponding first and second stereo-rectified image frames indicative of a target;
annotating means for determining a location of an image feature in the first and second stereo-rectified image frames, wherein the determined locations in the first and second stereo-rectified image frames are positionally constrained according to an epipolar constraint; and
training means for training a shape variation model according to the determined image feature locations.

### Claim 16 (depends on 15)

The system of claim 15, further comprising:
secondary input means for receiving stereo image data comprising a set of corresponding first and second stereo-rectified image frames indicative of a target; and
secondary processor for using the shape variation model to determine parameters associated with at least one image feature identified in the stereo image data; and
output means for outputting the parameters associated with the at least one image feature.

### Claim 17

A system for determining facial features, comprising:
input means for receiving stereo image data comprising a set of corresponding first and second stereo-rectified image frames indicative of a target;
a processor for using a stored shape variation model to determine parameters associated with at least one image feature identified in the stereo image data; and
output means for outputting the parameters associated with the at least one image feature.

### Claim 18 (depends on 15)

The system of claim 15, wherein the input means comprises a stereo camera; optionally the stereo camera attachable to a headset.

### Claim 19

(canceled)

### Claim 20 (depends on 15)

The system of claim 15, wherein the shape variation model is trained according to a training dataset which has been constrained according to an epipolar constraint.

### Claim 21

(canceled)

### Claim 22

(canceled)

### Claim 23 (depends on 15)

The system of claim 15, wherein identifying the at least one image feature in the stereo image data comprises using a profile matching algorithm.

### Claim 24 (depends on 23)

The system of claim 23, wherein the profile matching algorithm uses an Active Shape Model.

### Claim 25

(canceled)

### Claim 26

(canceled)

### Claim 27

(canceled)

### Claim 28 (depends on 1)

A non-transitory computer readable storage medium having instructions stored thereon, which when executed cause the computer to executed the computer-implemented method of claim 1.

### Claim 29 (depends on 6)

A non-transitory computer readable storage medium having instructions stored thereon, which when executed cause the computer to executed the computer-implemented method of claim 6.
