# US 2024/0005528 A1 · Information processing apparatus, learning apparatus, and tracking method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005528 A1
- Document ID: US20240005528A1
- Status: Published application
- Kind: A1
- Application number: 18/335,174
- Filed: 2023-06-15
- Published: 2024-01-04
- Earliest claimed: 2022-07-01
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Yuta HORIKAWA
- Web page: https://patentgrep.com/patent/US20240005528A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G06T 7/74: {involving reference images or patches}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An information processing apparatus obtains a reference image including a target object image to be tracked, determines a plurality of positions in the reference image, and generates a plurality of first features corresponding to the plurality of positions by inputting the reference image to a feature extraction neural network. The apparatus further obtains a search image that is a target of the tracking processing of the target object image, and generates a second feature by inputting the search image to the feature extraction neural network. Then, the apparatus identifies a position of the target object image included in the search image based on a result of a correlation operation between each of the plurality of first features and the second feature.

## Claims

### Claim 1

An information processing apparatus comprising:
a first obtaining unit configured to obtain a reference image including a target object image to be tracked;
a determination unit configured to determine a plurality of positions in the reference image to be used for tracking processing of the target object image;
a first generation unit configured to generate a plurality of first features corresponding to the plurality of positions by inputting the reference image to a feature extraction neural network;
a second obtaining unit configured to obtain a search image that is a target of the tracking processing of the target object image;
a second generation unit configured to generate a second feature by inputting the search image to the feature extraction neural network; and
an identification unit configured to identify a position of the target object image included in the search image based on a result of a correlation operation between each of the plurality of first features and the second feature.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein
the first generation unit generates, as the plurality of first features, a plurality of small regions corresponding to the plurality of positions in a feature image obtained by inputting the reference image to the feature extraction neural network.

### Claim 3 (depends on 1)

The apparatus according to claim 1, wherein
the determination unit generates a position map having a likelihood value representing a position likelihood with a feature suitable for the tracking processing of the target object image by inputting the reference image to a second neural network and determines the plurality of positions based on a peak position in the position map.

### Claim 4 (depends on 1)

The apparatus according to claim 1, wherein
the determination unit generates a feature image by inputting the reference image to the feature extraction neural network and determines the plurality of positions based on the feature image.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein
the determination unit generates a feature image by inputting the reference image to the feature extraction neural network, generates a position map having a likelihood value representing a position likelihood with a feature suitable for the tracking processing of the target object image by inputting the feature image to a second neural network, and determines the plurality of positions based on a peak position in the position map.

### Claim 6 (depends on 3)

The apparatus according to claim 3, wherein
the determination unit generates a first position map having a likelihood value representing a position likelihood with a feature suitable for the tracking processing of a part of the target object image and a second position map having a likelihood value representing a position likelihood with a feature suitable for the tracking processing of one of a color and a pattern of the target object image, and
the plurality of positions include at least one peak position in the first position map and at least one peak position in the second position map.

### Claim 7 (depends on 1)

The apparatus according to claim 1, wherein
the determination unit generates a plurality of likelihood maps each having a likelihood value representing a position likelihood of the target object image included in the search image based on the result of the correlation operation between each of the plurality of first features and the second feature, and identifies the position of the target object image included in the search image based on the plurality of likelihood maps.

### Claim 8 (depends on 7)

The apparatus according to claim 7, wherein
the identification unit identifies the position of the target object included in the search image based on an average likelihood map obtained by averaging the plurality of likelihood maps, or identifies the position of the target object included in the search image based on a likelihood map having a higher peak value in the plurality of likelihood maps.

### Claim 9 (depends on 7)

The apparatus according to claim 7, wherein
the identification unit further generates a size map of the target object image included in the search image based on the result of the correlation operation between each of the plurality of first features and the second feature, and further identifies a size of the target object image based on the size map and the position of the target object image.

### Claim 10 (depends on 9)

The apparatus according to claim 9, further comprising:
an obtaining unit configured to obtain a plurality of time-sequential frame images;
a reference image generation unit configured to generate the reference image that is a partial image including the target object image from a first frame image included in the plurality of frame images; and
a search image generation unit configured to generate the search image that is a partial image of a second frame image that is included in the plurality of frame images and follows the first frame image,
wherein the search image is cut out from the second frame image based on the position and/or the size of the target object image in a frame image preceding the second frame image.

### Claim 11 (depends on 1)

The apparatus according to claim 1, wherein
the feature extraction neural network includes a first Convolutional Neural Network (CNN) for feature extraction.

### Claim 12 (depends on 3)

The apparatus according to claim 3, wherein
the second neural network includes a Backbone network, and a second CNN for feature position inference.

### Claim 13 (depends on 12)

A learning apparatus configured to learn a second CNN of an information processing apparatus defined in claim 12, comprising:
a database configured to store time-series images and supplementary information concerning a position and a size of an object included in the time-series images;
a feature candidate generation unit configured to obtain a feature image generated by a feature extraction neural network for a frame image included in the time-series images and output a plurality of feature position candidates for the frame image to a first generation unit;
a tracking evaluation unit configured to calculate an evaluation result for each of the plurality of feature position candidates based on an identification result by an identification unit for each of the plurality of feature position candidates and the supplementary information included in the database;
a supervisory data generation unit configured to determine, based on the evaluation result, a feature position serving as a supervisor from the plurality of feature position candidates and generate supervisory data; and
an updating unit configured to update a parameter of the second CNN of the second neural network based on the supervisory data.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein
the plurality of feature position candidates for the frame image include a center position of a feature of the frame image and at least one position near the center position.

### Claim 15

A tracking method in an information processing apparatus configured to track an object image included in a moving image, comprising:
obtaining a reference image including a target object image to be tracked;
determining a plurality of positions in the reference image to be used for tracking processing of the target object image;
generating a plurality of first features corresponding to the plurality of positions by inputting the reference image to a feature extraction neural network;
obtaining a search image that is a target of the tracking processing of the target object image;
generating a second feature by inputting the search image to the feature extraction neural network; and
identifying a position of the target object image included in the search image based on a result of a correlation operation between each of the plurality of first features and the second feature.

### Claim 16

A non-transitory computer-readable recording medium storing a program for causing a computer to execute as an information processing apparatus comprising:
a first obtaining unit configured to obtain a reference image including a target object image to be tracked;
a determination unit configured to determine a plurality of positions in the reference image to be used for tracking processing of the target object image;
a first generation unit configured to generate a plurality of first features corresponding to the plurality of positions by inputting the reference image to a feature extraction neural network;
a second obtaining unit configured to obtain a search image that is a target of the tracking processing of the target object image;
a second generation unit configured to generate a second feature by inputting the search image to the feature extraction neural network; and
an identification unit configured to identify a position of the target object image included in the search image based on a result of a correlation operation between each of the plurality of first features and the second feature.
