# US 11,908,144 B2 · Image processing apparatus, method, and medium using degrees of reliability and similarity in motion vectors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,144 B2 - Document ID: US11908144B2 - Status: Granted - Kind: B2 - Application number: 17/519,371 - Filed: 2021-11-04 - Published: 2022-05-12 - Granted: 2024-02-20 - Earliest claimed: 2020-11-09 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Masaaki Kobayashi - Web page: https://patentgrep.com/patent/US11908144B2 ## CPC - G06T 7/215: Motion-based segmentation - G06T 7/223: using block-matching - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - H04N 19/521: {for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors} - H04N 21/2187: Live feed - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23412: {for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects} - H04N 21/25891: {being end-user preferences (retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)} - H04N 23/6811: {based on the image signal} - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A plurality of motion vectors are acquired from consecutive images. From the acquired plurality of motion vectors, a motion vector of interest and neighboring motion vectors neighboring the motion vector of interest are selected and a degree of similarity between two motion vectors is acquired. A value related to the total number of neighboring motion vectors having high degrees of similarity to the motion vector of interest is acquired as a degree of reliability. ## Claims ### Claim 1 An image processing apparatus comprising: one or more processors; and at least one memory, the at least one memory being coupled to the one or more processors and having stored thereon instructions executable by the one or more processors, wherein at least one of the one or more processors or execution of the instructions cause the image processing apparatus to function as: a vector acquisition unit configured to acquire a plurality of motion vectors based on consecutive images; a selection unit configured to select a motion vector of interest and a plurality of neighboring motion vectors neighboring the motion vector of interest from the acquired plurality of motion vectors; a similarity acquisition unit configured to acquire a degree of similarity between two motion vectors; and a reliability acquisition unit configured to acquire a degree of reliability represented by a value related to a total number of neighboring motion vectors having high degrees of similarity to the motion vector of interest, wherein the similarity acquisition unit calculates, as the degree of similarity of motion, a norm of a difference between two motion vectors. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the selection unit selects the neighboring motion vectors by checking each block of blocks obtained by dividing the image to determine whether the block includes at least the motion vector of interest, and, when the block includes the motion vector of, selecting motion vectors included in the block other than the motion vector of interest as the neighboring motion vectors. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the at least one of the one or more processors or the execution of the instructions cause the image processing apparatus to further function as an information acquisition unit configured to acquire image similarity information corresponding to the motion vectors to be acquired, wherein the reliability acquisition unit calculates the degree of reliability based on the number of motion vectors having high degrees of similarity equal to or higher a threshold value in terms of the similarity of motions and the image similarity information. ### Claim 4 (depends on 3) The image processing apparatus according to claim 3, wherein the image similarity information indicates a ratio of an image similarity of a motion vector with a highest degree of similarity of motion to an image similarity of a motion vector with a next highest degree of similarity of motion. ### Claim 5 (depends on 3) The image processing apparatus according to claim 3, wherein the image similarity information indicates a difference between an image similarity of a motion vector with a highest degree of similarity of motion and an image similarity of a motion vector with a next highest degree of similarity of motion. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the reliability acquisition unit acquires the degree of reliability such that in a case where the norms between the neighboring motion vectors and the motion vector of interest are equal to or smaller than a threshold value, the reliability acquisition acquires, as the degree of reliability, a value related to the total number of neighboring motion vectors. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the threshold value for the norm changes depending on a subject whose motion is to be detected. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the at least one of the one or more processors or the execution of the instructions cause the image processing apparatus to further function as a parameter acquisition unit configured to acquire a motion parameter representing the motion of the image based on the motion vectors for which the degree of reliability is acquired. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, wherein the at least one of the one or more processors or the execution of the instructions cause the image processing apparatus to further function as a unit configured to perform at least one of an image shake correction, a free viewpoint generation, or an image composition, based on the motion parameter. ### Claim 10 An image processing method comprising: acquiring a plurality of motion vectors based on consecutive images; selecting a motion vector of interest and a plurality of neighboring motion vectors neighboring the motion vector of interest from the acquired plurality of motion vectors; acquiring a degree of similarity between two motion vectors; and acquiring a degree of reliability represented by a value related to a total number of neighboring motion vectors having high degrees of similarity to the motion vector of interest, wherein the similarity acquisition unit calculates, as the degree of similarity of motion, a norm of a difference between two motion vectors. ### Claim 11 A non-transitory computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method comprising: acquiring a plurality of motion vectors based on consecutive images; selecting a motion vector of interest and a plurality of neighboring motion vectors neighboring the motion vector of interest from the acquired plurality of motion vectors; acquiring a degree of similarity between two motion vectors; and acquiring a degree of reliability represented by a value related to a total number of neighboring motion vectors having high degrees of similarity to the motion vector of interest, wherein the similarity acquisition unit calculates, as the degree of similarity of motion, a norm of a difference between two motion vectors.