# US 2008/0187175 A1 · Method and apparatus for tracking object, and method and apparatus for calculating object pose information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0187175 A1
- Document ID: US20080187175A1
- Status: Published application
- Kind: A1
- Application number: 11/987,281
- Filed: 2007-11-28
- Published: 2008-08-07
- Earliest claimed: 2007-02-07
- Assignees: Samsung Electronics Co., Ltd.
- Inventors: Jung-bae Kim; Haitao Wang
- Web page: https://patentgrep.com/patent/US20080187175A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038)
- G06T 3/147: using affine transformations
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/248: {involving reference images or patches}
- G06T 7/38: Registration of image sequences
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/167: {using comparisons between temporally consecutive images}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and apparatus for tracking an object, and a method and apparatus for calculating object pose information are provided. The method of tracking the object obtains object feature point candidates by using a difference between pixel values of neighboring frames. A template matching process is performed in a predetermined region having the object feature point candidates as the center. Accordingly, it is possible to reduce a processing time needed for the template matching process. The method of tracking the object is robust in terms of sudden changes in lighting and partial occlusion. In addition, it is possible to track the object in real time. In addition, since the pose of the object, the pattern of the object, and the occlusion of the object are determined, detailed information on action patterns of the object can be obtained in real time.

## Claims

### Claim 1

A method of tracking an object comprising:
(a) detecting predetermined object feature points in a (N−1)-th (N is a positive integer greater than 1) frame;
(b) estimating object feature point candidates of an N-th frame by using a difference between a pixel value of the (N−1)-th frame and a pixel value of the N-th frame; and
(c) estimating object feature points of the N-th frame by searching a predetermined region surrounding the estimated object feature point candidates of the N-th frame for a template similar to a predetermined template including the detected object feature points.

### Claim 2 (depends on 1)

The method of claim 1, wherein in (b), the object feature point candidates of the N-th frame are estimated by calculating an optical flow depending on the detected object feature points on the basis of the difference between the pixel value of the (N−1)-th frame and the pixel value of the N-th frame and using the calculated optical flow.

### Claim 3 (depends on 1)

The method of claim 1, wherein the template similar to the template including the detected object feature points is searched for in the predetermined region surrounding the estimated object feature point candidates by using a template matching process.

### Claim 4 (depends on 1)

The method of claim 1, wherein a size of the template including the detected object feature points is less than that of the predetermined region surrounding the object feature point candidates.

### Claim 5 (depends on 1)

The method of claim 1, wherein in (b), optical flow is calculated by calculating the difference between the pixel value of the (N−1)-th frame and the pixel value of the N-th frame based on an LKT (Lucas-Kanade-Tomasi) algorithm, and the object feature point candidates are estimated by using the calculated optical flow.

### Claim 6 (depends on 1)

The method of claim 1, wherein the object feature point is detected by using AAMs (Active Appearance Models) or ASMs (Active Shape Models).

### Claim 7 (depends on 1)

A computer-readable recording medium having embodied thereon a computer program for executing the method of claim 1.

### Claim 8

A method of calculating object pose information comprising:
(a) detecting predetermined object feature points in a (N−1)-th (N is a positive integer greater than 1) frame;
(b) estimating object feature point candidates of an N-th frame by using a difference between a pixel value of the (N−1)-th frame and a pixel value of the N-th frame;
(c) estimating object feature points of the N-th frame by searching a predetermined region surrounding the estimated object feature point candidates of the N-th frame for a template similar to a predetermined template including the detected object feature points; and
(d) calculating object pose information by using coordinate values detected in the (N−1)-th frame and coordinate values of the object feature points detected in the N-th frame.

### Claim 9 (depends on 8)

The method of claim 8, wherein in (c), the object feature pints are estimated by estimating 2D (X-Y) locations of the object feature points, and in (d), the object pose information is calculated by further considering a 3D model of a general object and a table in which information (Z) on heights of the general object is stored.

### Claim 10 (depends on 8)

The method of claim 8, wherein the object pose information includes rotation matrix information on directions of the object.

### Claim 11 (depends on 8)

The method of claim 8, wherein the template similar to the template including the detected object feature points is searched for in the predetermined region surrounding the estimated object feature point candidates by using a template matching process.

### Claim 12 (depends on 8)

The method of claim 8, wherein the object pose information further includes occlusion information on whether the object feature points are occluded in the N-th frame.

### Claim 13 (depends on 8)

The method of claim 8, wherein the object feature points includes feature points related to eyes and a mouth selected from a human face, and the object pose information further includes information on blinking of the eyes or information on a shape of the mouth as pattern information.

### Claim 14 (depends on 8)

The method of claim 8, further comprising:
calculating an affine transformation between coordinate values of object feature points detected in the (N−1)-th frame and coordinate values of object feature points detected in the N-th frame;
estimating the object feature points of the N-th frame by using the coordinate values of the feature points of the object detected in the (N−1)-th frame and the calculated affine transformation; and
calculating a similarity between the estimated object feature points of the N-th frame and the calculated object feature points and estimating whether the object feature points are occluded on the basis of the calculated similarity,
wherein the object pose information further includes the occlusion information on whether the object feature points in the N-th frame are occluded.

### Claim 15 (depends on 8)

The method of claim 8,
wherein the object feature points include feature points related to the eyes selected from the human face,
wherein the method of calculating object pose information further comprises calculating an edge of an eye region by applying a Sobel operator to the N-th frame and the (N−1)-th frame, generating a gradient map depending on the calculated edge of the eye region, and estimating whether the eyes blink according to a similarity or correlation between the gradient map of the N-th frame and the gradient map of the (N−1)-th frame, and
wherein the object pose information further includes the pattern information on the blinking of the eyes.

### Claim 16 (depends on 8)

The method of claim 8,
wherein the object feature points include feature points related to the mouth selected from the human face,
wherein the method of calculating object pose information further comprises converting a predetermined image including the feature points related to the mouth into a binary level image and estimating an ellipse function that represents an edge of the mouth by using an ellipse fitting process, and
wherein the object pose information further includes the pattern information on the shape of the mouth according to the estimated ellipse function.

### Claim 17 (depends on 8)

A computer-readable recording medium having embodied thereon a computer program for executing the method of claim 8.

### Claim 18

An apparatus for tracking a video image object, the apparatus comprising:
an initialization unit which initializes a tracking process by detecting object feature points from a video image;
a first tracking unit which estimates object feature point candidates of an N-th frame by using a difference between a pixel value of an (N−1)-th (N is a positive integer greater than 1) frame and a pixel value of the N-th frame; and
a second tracking unit which searches a predetermined region surrounding the determined object feature point candidates for a template similar to a template including the object feature points of the (N−1)-th frame and estimates the object feature points.

### Claim 19

An apparatus for calculating object pose information, the apparatus comprising:
an initialization unit which initializes a tracking process by detecting object feature points from a video image;
a first tracking unit which estimates object feature point candidates of an N-th frame by using a difference between a pixel value of an (N−1)-th (N is a positive integer greater than 1) frame and a pixel value of the N-th frame;
a second tracking unit which searches a predetermined region surrounding the determined object feature point candidates for a template similar to a template including the object feature points of the (N−1)-th frame and estimates the object feature points; and
an object pose information calculation unit which calculates object pose information by using coordinate values of the object feature points detected by initializing the tracking process and coordinate values of the object feature points of the N-th frame.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the object pose calculation unit includes a pose estimation unit generating rotation matrix information on directions of the object, an occlusion estimation unit generating occlusion information on whether the object feature points are occluded, and a pattern estimation unit generating pattern information on blinking of eyes and a shape of a mouth.
