# US 12,002,218 B2 · Method and apparatus with object tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,218 B2
- Document ID: US12002218B2
- Status: Granted
- Kind: B2
- Application number: 17/531,972
- Filed: 2021-11-22
- Published: 2022-05-26
- Granted: 2024-06-04
- Earliest claimed: 2020-11-26
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Hyunjeong Lee; Seungwook Kim; Changbeom Park; Byung In Yoo
- Web page: https://patentgrep.com/patent/US12002218B2

## CPC

- G06T 7/194: involving foreground-background segmentation
- G06V 10/56: relating to colour
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/136: involving thresholding
- G06T 7/248: {involving reference images or patches}
- G06T 7/90: Determination of colour characteristics
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/82: using neural networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method, apparatus, and system with tracking are disclosed. The apparatus is configured to acquire a feature map of a template image, a color histogram of a foreground of the template image that has an object, and a color histogram of a background of the template image other than the foreground, acquire a feature score map, and a bounding box map corresponding to the feature score map, based on the feature map of the template image and a feature map of a search image, acquire a color score map based on the color histogram of the foreground, the color histogram of the background, and a color value of the search image, acquire a final score map, and a bounding box map corresponding to the final score map, based on the feature score map, the color score map, and the bounding box map, and output the corresponding bounding box.

## Claims

### Claim 1

A processor-implemented method, the method comprising:
acquiring a feature map of a template image, a color histogram of a foreground of the template image, and a color histogram of a background of the template image other than the foreground, where the foreground includes an object included in the template image;
acquiring a feature score map, and a bounding box map corresponding to the feature score map, based on the feature map of the template image and a feature map of a search image of a current frame;
acquiring a color score map based on the color histogram of the foreground, the color histogram of the background, and a color value of the search image of the current frame;
acquiring a final score map, and a bounding box map corresponding to the final score map, based on the feature score map, the color score map, and the bounding box map; and
outputting a bounding box corresponding to a pixel having a highest value of the final score map from the bounding box map.

### Claim 2 (depends on 1)

The method of claim 1, wherein the acquiring of the color score map comprises:
acquiring a foreground color similarity map based on the color value of the search image and the color histogram of the foreground;
acquiring a background color similarity map based on the color value of the search image and the color histogram of the background; and
acquiring the color score map based on the foreground color similarity map and the background color similarity map.

### Claim 3 (depends on 2)

The method of claim 2, wherein the acquiring of the color score map based on the foreground color similarity map and the background color similarity map comprises acquiring the color score map by calculating a ratio of a pixel value of the foreground color similarity map to a sum of the pixel value of the foreground color similarity map and a pixel value of the background color similarity map, for each pixel.

### Claim 4 (depends on 2)

The method of claim 2,
wherein the acquiring of the color score map further comprises scaling the foreground color similarity map and the background color similarity map based on an area proportion of the background and an area proportion of the foreground, respectively, and
the acquiring of the color score map based on the foreground color similarity map and the background color similarity map comprises acquiring the color score map based on the scaled foreground color similarity map and the scaled background color similarity map.

### Claim 5 (depends on 4)

The method of claim 4, wherein the acquiring of the color score map based on the scaled foreground color similarity map and the scaled background color similarity map comprises acquiring the color score map by calculating a ratio of a pixel value of the scaled foreground color similarity map to a sum of the pixel value of the scaled foreground color similarity map, a pixel value of the scaled background color similarity map, and an adjustment constant, for each pixel.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
acquiring a foreground color similarity map based on the color value of the search image and the color histogram of the foreground;
acquiring a background color similarity map based on the color value of the search image and the color histogram of the background; and
calculating a color weight representing an extent of non-similarity of the foreground color similarity map and the background color similarity map.

### Claim 7 (depends on 6)

The method of claim 6, wherein the calculating of the color weight comprises:
calculating a feature weight representing a similarity between the foreground color similarity map and the background color similarity map; and
calculating the color weight by subtracting the feature weight from a reference constant.

### Claim 8 (depends on 7)

The method of claim 7, wherein the calculating of the color weight comprises calculating the color weight through an inner product between the foreground color similarity map and the background color similarity map.

### Claim 9 (depends on 7)

The method of claim 7, wherein the acquiring of the final score map, and the bounding box map corresponding to the final score map, comprises acquiring the final score map by calculating a weight average by applying the feature weight to a pixel value of the feature score map and applying the color weight to a pixel value of the color score map, for each pixel.

### Claim 10 (depends on 9)

The method of claim 9, wherein the acquiring of the final score map, and the bounding box map corresponding to the final score map, further comprises adjusting resolutions of the feature score map and the color score map equally to acquire the final score map and the bounding box map corresponding to the final score map.

### Claim 11 (depends on 6)

The method of claim 6, further comprising:
comparing the color weight to a second threshold,
wherein in response to the color weight failing to meet the second threshold, the acquiring of the bounding box map comprises using the feature score map and the bounding box map as the final score map and the bounding box map corresponding to the final score map.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
comparing a maximum value of a pixel of the feature score map to a first threshold,
wherein in response to the maximum value of the pixel of the feature score map failing to meet the first threshold, the acquiring of the color score map and the acquiring of the bounding box map are selected to not be performed, and the outputting of the bounding box comprises outputting a bounding box of a previous frame.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
updating the color histogram of the foreground and the color histogram of the background based on the bounding box corresponding to the pixel having the highest value.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
acquiring a foreground color similarity map based on the color value of the search image and the color histogram of the foreground;
acquiring a background color similarity map based on the color value of the search image and the color histogram of the background;
calculating a color weight representing an extent of non-similarity of the foreground color similarity map and the background color similarity map; and
comparing the color weight to a second threshold,
wherein, in response to the color weight failing to meet the second threshold, the updating is selected to not be performed.

### Claim 15 (depends on 1)

The method of claim 1, wherein the outputting of the bounding box includes tracking the object in the search image using the bounding box corresponding to the pixel.

### Claim 16 (depends on 15)

The method of claim 15, wherein the tracking of the object includes displaying the object on a display.

### Claim 17 (depends on 1)

The method of claim 1, wherein the acquiring of the feature map of the template image includes applying the template image to a neural network feature extractor to obtain the feature map of the template image, and the feature map of the search image is generated by applying the search image to the neural network feature extractor to obtain the feature map of the search image.

### Claim 18 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1.

### Claim 19

A processor-implemented method, the method comprising:
generating a feature map of a search image;
generating a feature score map based on the generated feature map of the search image, and an obtained feature map for a template that includes a target object;
generating a color score map dependent on a distribution of color information in a first portion of the template, a distribution of color information in a second portion of the template distinguished from the first portion, and color information of the search image, where the first portion includes the target object; and
tracking the target object in the search image using a bounding box determined based on the generated feature score map and the generated color score map.

### Claim 20 (depends on 19)

The method of claim 19, wherein the obtained feature map for the template is read from a memory, the feature map for the template having been generated and stored to the memory with respect to a previous tracking of the target object for a previous search image.

### Claim 21 (depends on 19)

The method of claim 19, wherein the second portion of the template is distinguished from the first portion of the template based on a bounding box determined for a previous tracking of the target object for a previous search image.

### Claim 22 (depends on 21)

The method of claim 21,
wherein the distribution of the color information in the first portion of the template is a scaled first histogram of the first portion of the template, and the distribution of the color information in the second portion of the template is a scaled second histogram of the second portion of the template, and
wherein the generating the color score map includes obtaining the scaled first histogram and the scaled second histogram from a memory, the scaled first histogram and the scaled second histogram having been generated and stored to the memory with respect to the previous tracking of the target object for the previous search image.

### Claim 23 (depends on 19)

The method of claim 19, wherein generating of the color score map comprises:
acquiring a first color similarity map dependent on the search image and a color histogram of a foreground of the template, as the distribution of the color information in the first portion of the template;
acquiring a second color similarity map dependent on the search image and a color histogram of a background of the template, as the distribution of the color information in the second portion of the template; and
acquiring the color score map based on the first color similarity map and the second color similarity map.

### Claim 24 (depends on 23)

The method of claim 23, wherein the acquiring of the color score map based on the first color similarity map and the second color similarity map comprises acquiring the color score map by calculating a ratio of a pixel value of a foreground color similarity map, as the first color similarity map, to a sum of the pixel value of the foreground color similarity map and a pixel value of a background color similarity map, as the second color similarity map.

### Claim 25 (depends on 24)

The method of claim 24, wherein the acquiring of the color score map based on the foreground color similarity map and the background color similarity map includes:
scaling the foreground color similarity map and the background color similarity map respectively, based on an area proportion of the background of the template and an area proportion of the foreground of the template; and
acquiring the color score map based on the scaled foreground color similarity map and the scaled background color similarity map.

### Claim 26

An apparatus, the apparatus comprising:
one or more processors configured to:
generate a first feature map of a first search image;
generate a first feature score map dependent on the first feature map of the first search image and an obtained first feature map for a template that includes a target object;
generate a first color score map dependent on color information of the first search image, color information in a first portion of the template, and color information in a second portion of the template distinguished from the first portion, where the first portion includes the target object;
update the template by updating extents of the first portion and the second portion of the template based on a bounding box determined for the target object in the first search image, where the bounding box is determined dependent on the first feature score map and the first color score map; and
track the target object in a second search image, subsequent to the first search image, dependent on the updated template.

### Claim 27 (depends on 26)

The apparatus of claim 26, wherein the tracking of the target object in the second search image includes tracking the target object in the second search image dependent on feature information of the second search image, feature information of the updated template, color information of the second search image, and color information of the updated template.

### Claim 28 (depends on 26)

The apparatus of claim 26, wherein, in the updating of the template, the one or more processors are further configured to generate a color histogram of a foreground of the updated template and a color histogram of a background based on the bounding box, wherein the foreground is defined by the updated extents of the first portion, and the background is defined by the updated extents of the second portion, and
wherein the tracking of the target in the second search image is dependent on the generated color histogram of the foreground, the generated color histogram of the background, and feature and color information of the second search image.

### Claim 29

An apparatus, the apparatus comprising:
one or more processors configured to:
acquire a feature map of a template image, a color histogram of a foreground of the template image, and a color histogram of a background of the template image other than the foreground, where the foreground includes an object included in the template image;
acquire a feature score map, and a bounding box map corresponding to the feature score map, based on the feature map of the template image and a feature map of a search image of a current frame;
acquire a color score map based on the color histogram of the foreground, the color histogram of the background, and a color value of the search image of the current frame;
acquire a final score map, and a bounding box map corresponding to the final score map, based on the feature score map, the color score map, and the bounding box map; and
output a bounding box corresponding to a pixel having a highest value of the final score map from the bounding box map.

### Claim 30

A terminal comprising:
a memory;
a display; and
one or more processors configured to:
acquire a feature map of a template image, a color histogram of a foreground of the template image, and a color histogram of a background of the template image other than the foreground, where the foreground includes an object included in the template image;
acquire a feature score map, and a bounding box map corresponding to the feature score map, based on the feature map of the template image and a feature map of a search image of a current frame;
acquire a color score map based on the color histogram of the foreground, the color histogram of the background, and a color value of the search image of the current frame;
acquire a final score map, and a bounding box map corresponding to the final score map, based on the feature score map, the color score map, and the bounding box map; and
output, through the display, a bounding box corresponding to a pixel having a highest value of the final score map from the bounding box map.