# US 10,375,287 B2 · Object trail-based analysis and control of video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,375,287 B2
- Document ID: US10375287B2
- Status: Granted
- Kind: B2
- Application number: 12/907,607
- Filed: 2010-10-19
- Published: 2011-04-21
- Granted: 2019-08-06
- Earliest claimed: 2009-10-21
- Assignees: Disney Enterprises, Inc.
- Applicants: Smadar Gefen; Gregory House; Yuecheng Zhang
- Inventors: Smadar Gefen; Gregory House; Yuecheng Zhang
- Web page: https://patentgrep.com/patent/US10375287B2

## CPC

- H04N 23/61: based on recognised objects
- G06T 7/248: {involving reference images or patches}
- H04N 23/60: Control of cameras or camera modules
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10024: Color image
- G06T 2207/30221: Sports video; Sports image
- G06T 2207/30224: Ball; Puck
- G06T 2207/30228: Playing field
- G06T 2207/30241: Trajectory
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 20/42: {of sport video content}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for analyzing scenes from cameras imaging an event, such as a sporting event broadcast, are provided. Systems and methods include detecting and tracking patterns and trails. This may be performed with intra-frame processing and without knowledge of camera parameters. A system for analyzing a scene may include an object characterizer, a foreground detector, an object tracker, a trail updater, and a video annotator. Systems and methods may provide information regarding centers and spans of activity based on object locations and trails, which may be used to control camera field of views such as a camera pose and zoom level. A magnification may be determined for images in a video sequence based on the size of an object in the images. Measurements may be determined from object trails in a video sequence based on an effective magnification of images in the video sequence.

## Claims

### Claim 1

A system for analyzing a video sequence from a moveable camera imaging an event, comprising:
a tracking system tracking past positions of a tracked object in the video sequence and tracking positions of two or more stationary reference objects in the video sequence;
a trail updater updating a trail, wherein the trail comprises the past positions of the tracked object, further wherein the updating is based on the positions of the two or more reference objects in the video sequence relative to the past positions of the tracked object in the video sequence; and
a camera controller controlling a field of view of the moveable camera based on the trail of the tracked object, wherein controlling comprises at least one of steering the entire moveable camera or adjusting a position of the entire moveable camera.

### Claim 2 (depends on 1)

The system of claim 1, wherein the trail updater is further configured to use different subsets of tracked objects as the two or more reference objects.

### Claim 3 (depends on 1)

The system of claim 1, wherein the trail updater is further configured to use a stationary pattern as the two or more reference objects.

### Claim 4 (depends on 1)

The system of claim 1, wherein the tracking system comprises an object characterizer configured to construct a model of the tracked object in the video sequence.

### Claim 5 (depends on 4)

The system of claim 4, wherein the tracking system is further configured to extract the model of the tracked object from a database of pre-defined object models and use the model of the tracked object to automatically determine an initial location of the tracked object.

### Claim 6 (depends on 4)

The system of claim 4, wherein the tracking system is further configured to receive an initial location of the tracked object and automatically construct the model of the tracked object based on the initial location of the tracked object.

### Claim 7 (depends on 4)

The system of claim 4, wherein the tracking system further comprises a foreground detector configured to determine one or more foreground descriptors for one or more foregrounds in the video sequence.

### Claim 8 (depends on 7)

The system of claim 7, wherein the foreground detector is further configured to perform intra-frame foreground detection within a local region surrounding the tracked object.

### Claim 9 (depends on 7)

The system of claim 7, wherein the foreground detector is further configured to determine the one or more foreground descriptors based on a plurality of features.

### Claim 10 (depends on 7)

The system of claim 7, wherein the tracking system further comprises an object tracker configured to determine a position of the tracked object in the video sequence based on the model of the tracked object and the one or more foreground descriptors without use of parameters of the camera.

### Claim 11 (depends on 1)

The system of claim 1, further comprising a video annotator configured to annotate the trail of the tracked object in a field of view of the camera.

### Claim 12 (depends on 1)

The system of claim 1, wherein the camera directs the moveable camera to at least one center of activity based on location information of objects in a scene.

### Claim 13 (depends on 1)

The system of claim 1, wherein the tracking system further comprises a size detector configured to determine the size of the tracked object in an image of the video sequence.

### Claim 14 (depends on 1)

The system of claim 1, wherein the tracking system further comprises a magnification detector configured to determine a magnification of one or more images in the video sequence.

### Claim 15 (depends on 1)

The system of claim 1, wherein the tracking system further comprises a measurement detector configured to determine a measurement of the tracked object based in part on the trail of the tracked object.

### Claim 16 (depends on 15)

The system of claim 15, wherein the measurement detector is further configured to determine a measurement of the tracked object based in part on a magnification of an image in the video sequence.

### Claim 17 (depends on 15)

The system of claim 15, wherein the measurement detector is further configured to determine the measurement of the tracked object based in part on the trail of the tracked object in relationship to the respective positions of one or more reference objects in the video sequence.

### Claim 18 (depends on 15)

The system of claim 15, further comprising a video annotator configured to annotate the video sequence based on the measurement of the tracked object.

### Claim 19

A method for analyzing a video sequence from a movable camera imaging an event, comprising:
tracking past positions of a tracked object in the video sequence;
tracking positions of two or more stationary reference objects in the video sequence;
updating a trail, wherein the trail comprises the past positions of the tracked object, further wherein the updating is based on the positions of the two or more reference objects in the video sequence relative to the past positions of the tracked object in the video sequence; and
controlling a field of view of the moveable camera based on the trail of the tracked object, with a camera controller, wherein controlling comprises at least one of steering the entire moveable camera or adjusting a position of the entire moveable camera.

### Claim 20 (depends on 19)

The method of claim 19, wherein updating includes using different subsets of tracked objects as the two or more reference objects.

### Claim 21 (depends on 19)

The method of claim 19, wherein updating includes using a stationary pattern as the two or more reference objects.

### Claim 22 (depends on 19)

The method of claim 19, wherein tracking the trail includes constructing a model of the tracked object in the video sequence.

### Claim 23 (depends on 22)

The method of claim 22, wherein constructing includes extracting the model of the tracked object from a database of pre-defined object models and using the model of the tracked object to automatically determine an initial location of the tracked object.

### Claim 24 (depends on 22)

The method of claim 22, wherein constructing includes receiving an initial location of the tracked object and automatically constructing the model of the tracked object based on the initial location of the tracked object.

### Claim 25 (depends on 19)

The method of claim 19, wherein tracking the trail includes determining one or more foreground descriptors for one or more foregrounds in the video sequence.

### Claim 26 (depends on 25)

The method of claim 25, wherein determining one or more foreground descriptors includes performing intra-frame foreground detection within a local region surrounding the tracked object.

### Claim 27 (depends on 25)

The method of claim 25, wherein determining one or more foreground descriptors includes determining the one or more foreground descriptors based on a plurality of features.

### Claim 28 (depends on 25)

The method of claim 25, wherein tracking the trail includes determining a position of the tracked object in the video sequence based on the model of the tracked object and the one or more foreground descriptors without use of parameters of the camera.

### Claim 29 (depends on 19)

The method of claim 19, further comprising annotating the trail of the tracked object in a field of view of the camera.

### Claim 30 (depends on 19)

The method of claim 19, wherein controlling includes directing the moveable camera to at least one center of activity based on location information of objects in a scene.

### Claim 31 (depends on 19)

The method of claim 19, wherein controlling includes controlling a zoom level of the moveable camera based on the trail of the tracked object.

### Claim 32 (depends on 19)

The method of claim 19, wherein tracking the trail includes determining a size of the tracked object in an image of the video sequence.

### Claim 33 (depends on 19)

The method of claim 19, wherein tracking the trail includes determining a magnification of one or more images in the video sequence.

### Claim 34 (depends on 33)

The method of claim 33, wherein determining a magnification includes determining effective magnifications for the tracked object corresponding to a tracked and a second direction within a plane.

### Claim 35 (depends on 33)

The method of claim 33, wherein determining a measurement includes determining a measurement of the tracked object based in part on a magnification of an image in the video sequence.

### Claim 36 (depends on 19)

The method of claim 19, further comprising determining a measurement of the tracked object based in part on the trail of the tracked object.

### Claim 37 (depends on 36)

The method of claim 36, wherein determining a measurement includes determining the measurement of the tracked object based in part on the trail of the tracked object in relationship to the respective positions of one or more reference objects in the video sequence.

### Claim 38 (depends on 37)

The method of claim 37, further comprising annotating the video sequence based on the measurement of the tracked object.