# US 8,279,286 B2 · Apparatus and method of object tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,279,286 B2
- Document ID: US8279286B2
- Status: Granted
- Kind: B2
- Application number: 12/204,358
- Filed: 2008-09-04
- Published: 2009-03-05
- Granted: 2012-10-02
- Earliest claimed: 2007-09-05
- Assignees: Sony Corporation
- Inventors: David Wagg; Michael John Williams
- Web page: https://patentgrep.com/patent/US8279286B2

## CPC

- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06T 7/251: {involving models}
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G11B 27/105: {of operating discs}
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/30221: Sports video; Sports image
- G06T 2207/30241: Trajectory
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A method of extracting image features from objects on a plane within video images; detecting the objects from a relative position on the plane by comparing the extracted image features with sample image features; and generating object identification data identifying the objects on the plane. The method includes generating a 3D model of the plane and logging object identification data and object path data, the object path data including a time history of object position on the 3D model and a path of the objects within the video images. The method also includes detecting an occlusion event indicating whether object features are occluded; associating object identification data with object path data for objects in the occlusion event; identifying one of the objects involved in the occlusion event by comparing the objects image features and the sample image features; and updating the path data after the identification.

## Claims

### Claim 1

A method of tracking objects on a plane within one or more video images of the objects on the plane captured by a video camera, the method comprising:
processing the captured video images so as to extract one or more image features from each object;
detecting each of the objects from a relative position of the objects on the plane as viewed from the captured video images by comparing the one or more extracted image features associated with each object with sample image features from a predetermined set of possible example objects which the captured video images may contain;
generating object identification data for each object, from the comparing, which identifies the respective object on the plane;
generating a three dimensional model of the plane; and
logging, for each detected object, the object identification data together with object path data, the object path data providing a position of the object on the three dimensional model of the plane from the video images with respect to time;
in which the logging includes:
detecting an occlusion event in dependence upon whether a first image feature associated with a first of the objects obscures a whole or part of at least a second image feature associated with at least a second of the objects;
when an occlusion event is detected, associating the object identification data for the first object and the object identification data for the second object with the object path data for both the first object and the second object respectively and logging the associations;
after detection of the occlusion event, identifying at least one of the objects involved in the occlusion event in dependence upon a comparison between the one or more image features associated with that object and the sample image features from the predetermined set of possible example objects; and
updating the logged path data after the identification of at least one of the objects so that the respective path data is associated with the respective identified object.

### Claim 2 (depends on 1)

The method according to claim 1, in which the objects are players of a sports game.

### Claim 3 (depends on 2)

The method according to claim 2, further comprising:
generating game metadata from the logged path data, the game metadata relating to a position of at least one of the players of the sports game on a field of play.

### Claim 4 (depends on 1)

The method according to claim 1, in which the image features are alphanumeric characters.

### Claim 5 (depends on 1)

The method according to claim 1, further comprising:
generating a confidence value in dependence upon the comparison between the one or more image features associated with that object and the image features extracted from possible examples of the objects, in which the confidence value is generated for each of the image features extracted from the possible examples of the objects and relates to the similarity of the one or more image features to each of the image features extracted from the possible examples of the objects; and
detecting to which example of the objects the one or more image features corresponds in dependence upon whether the confidence value for each of the image features extracted from the possible examples of the objects exceeds a predetermined confidence threshold.

### Claim 6 (depends on 1)

The method according to claim 1, further comprising:
maintaining a history of the logged path data that was updated at the updating step.

### Claim 7 (depends on 1, 2, 3, 4, 5, 6)

A non-transitory computer readable medium having instructions stored therein that when executed by a processing circuit causes the processing circuit to carry out the method according to any one of claims 1-6.

### Claim 8 (depends on 1)

A non-transitory computer readable medium having instructions stored therein that when executed by a processing circuit causes the processing circuit to operate as apparatus for tracking objects on a plane within video images of the objects captured by a video camera in accordance with claim 1.

### Claim 9

An image processing apparatus for tracking objects on a plane within one or more video images of the objects on the plane captured by a video camera, the apparatus comprising:
an image processor operable to process the captured video images so as to extract one or more image features from each object;
an object detector operable to detect each of the objects from a relative position of the objects on the plane as viewed from within the captured video images, by comparing the one or more image features with sample image features from a predetermined set of possible example objects, which the captured video images may contain, and to generate object identification data for each object which identifies the respective object from the comparison;
a model generator operable to generate a three dimensional model of the plane; and
a logging device operable to log, for each detected object, the object identification data together with object path data, the object path data providing a position of the object on the three dimensional model of the plane from the video images with respect to time;
in which the logging device comprises:
an occlusion detecting device operable to detect an occlusion event in dependence upon whether a first image feature associated with a first of the objects obscures a whole or part of at least a second image feature associated with at least a second of the objects, and
when an occlusion event is detected, to associate the object identification data for the first object and the object identification data for the second object with the object path data for both the first object and the second object respectively and to log the associations;
an identifying device operable to identify, after the detection of the occlusion event, at least one of the objects involved in the occlusion event in dependence upon a comparison between the one or more image features associated with that object and the predetermined sample image features from the predetermined set of possible example objects; and
an updating device operable to update the logged path data after the identification of at least one of the objects identified by the identifying device so that the respective path data is associated with the respective identified object.

### Claim 10

An apparatus for tracking objects on a plane within one or more video images of the objects on the plane captured by a video camera, the apparatus comprising:
means for processing the captured video images so as to extract one or more image features from each object;
means for detecting each of the objects from a relative position of the objects on the plane as viewed from the captured video images by comparing the one or more extracted image features associated with each object with sample image features from a predetermined set of possible example objects which the captured video images may contain;
means for generating object identification data for each object, from the comparing, which identifies the respective object on the plane;
means for generating a three dimensional model of the plane; and
means for logging, for each detected object, the object identification data together with object path data, the object path data providing a position of the object on the three dimensional model of the plane from the video images with respect to time;
in which the means for logging includes:
means for detecting an occlusion event in dependence upon whether a first image feature associated with a first of the objects obscures a whole or part of at least a second image feature associated with at least a second of the objects; and
when an occlusion event is detected, means for associating the object identification data for the first object and the object identification data for the second object with the object path data for both the first object and the second object respectively and logging the associations;
means for, after the detection of the occlusion event, identifying at least one of the objects involved in the occlusion event in dependence upon a comparison between the one or more image features associated with that object and the sample image features from the predetermined set of possible example objects; and
means for updating the logged path data after the identification of at least one of the objects so that the respective path data is associated with the respective identified object.
