# US 10,311,595 B2 · Image processing device and its control method, imaging apparatus, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,595 B2 - Document ID: US10311595B2 - Status: Granted - Kind: B2 - Application number: 14/541,700 - Filed: 2014-11-14 - Published: 2015-05-21 - Granted: 2019-06-04 - Earliest claimed: 2013-11-19 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Yuta Kawamura - Web page: https://patentgrep.com/patent/US10311595B2 ## CPC - G06T 7/73: using feature-based methods - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing device includes an acquisition unit configured to acquire an image data and a depth data correspond to the image data; a calculating unit configured to calculate a position and attitude change for each depth from the image data and the depth data; and a determining unit configured to determine a position and attitude change of a whole image by a position and attitude change data calculated by the calculating unit based on a statistic of the position and attitude change calculated by the calculating unit. ## Claims ### Claim 1 An image processing device comprising: one or more processors; and a memory storing instructions which, when executed by the one or more processors, cause the image processing device to function as: an acquisition unit configured to acquire a plurality of image data and a plurality of depth data corresponding to the plurality of image data, respectively; a calculating unit configured to calculate first position and attitude changes for each depth from the plurality of image data and the plurality of depth data; a grouping unit configured to calculate a frequency distribution of depths corresponding to the first position and attitude changes from each of the plurality of depth data, and group the plurality of depth data in a plurality of groups based on the frequency distribution; a detection unit configured to detect motions in the plurality of image data; a region calculating unit configured to estimate regions, which have a same motion; a processing unit configured to calculate second position and attitude changes of the regions for each motion; and a determining unit configured to determine a position and attitude change of a whole image for the plurality of image data by determining a static region based on data of the first position and attitude changes which respectively correspond to the plurality of groups obtained by the grouping unit and data of the second position and attitude changes. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the calculating unit compares a reference frame obtained by segmenting the plurality of depth data with a non-reference frame obtained by not segmenting the plurality of depth data so as to calculate the first position and attitude changes for each depth. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein, when the determining unit determines a region in which a region occupied by an object image is a greatest region as the static region, the determining unit determines the position and attitude change of the whole image based on the data of the second position and attitude changes. ### Claim 4 (depends on 1) The image processing device according to claim 1, wherein, when the determining unit determines a region in which a proportion occupied by depth in the data of the frequency distribution is a greatest as the static region, the determining unit determines the position and attitude change of the whole image based on the data of the first position and attitude changes. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the instructions stored in the memory cause the image processing device to function as: a selecting unit configured to select judgment criteria of the static region in accordance with an operation, wherein the determining unit determines the static region in accordance with the judgment criteria selected by the selecting unit. ### Claim 6 (depends on 1) The image processing device according to claim 1, wherein the processing unit selects a position and attitude change corresponding to a region of which a proportion to the whole image is greatest from among the regions for each motion. ### Claim 7 (depends on 1) The image processing device according to claim 1, wherein, when a difference between the data of the first position and attitude changes and the data of the second position and attitude changes is less than a threshold value, the determining unit determines the position and attitude change of the whole image by performing a weighting computation. ### Claim 8 (depends on 1) The image processing device according to claim 1, wherein the instructions stored in the memory cause the image processing device to function as: a storage unit configured to store the data of the position and attitude change of the whole image determined by the determining unit, wherein, when a difference between the data of the first position and attitude changes and the data of the second position and attitude changes exceeds a threshold value, the determining unit reads data stored in the storage unit and then determines either the data of the first position and attitude changes or the data of the second position and attitude changes with a smaller difference from a position and attitude change indicated by the read data as the data of the position and attitude change of the whole image. ### Claim 9 (depends on 8) The image processing device according to claim 8, wherein, when the data of the first position and attitude changes is different from the data of the second position and attitude changes and no data of the position and attitude change is stored in the storage unit, the determining unit does not use both the data of the first position and attitude changes and the data of the second position and attitude changes as the data of the position and attitude change of the whole image. ### Claim 10 (depends on 1) The image processing device according to claim 1, wherein the calculating unit calculates depth data of the same position and attitude change including at least the rearmost depth from the plurality of depth data, and the determining unit estimates a first background region candidate from the plurality of depth data of the same position and attitude change including the rearmost depth. ### Claim 11 (depends on 10) The image processing device according to claim 10, wherein the processing unit comprises a second estimating unit configured to estimate a second background region candidate for the regions for each motion. ### Claim 12 (depends on 11) The image processing device according to claim 11, wherein the second estimating unit comprises: a first extracting unit configured to extract a local feature amount from the plurality of image data; a second extracting unit configured to extract a local feature amount from image data of a previously-learned image; a database creating unit configured to create database from the local feature amount extracted by the second extracting unit; and a collating unit configured to collate the local feature amount extracted by the first extracting unit with data in the database, wherein the collating unit estimates the second background region candidate. ### Claim 13 (depends on 11) The image processing device according to claim 11, wherein the determining unit determines a background region based on the first background region candidate, the data of the first position and attitude changes, the second background region candidate, and the data of the second position and attitude changes and then determines the static region based on the background region. ### Claim 14 (depends on 13) The image processing device according to claim 13, wherein the determining unit determines the position and attitude change of the whole image based on the static region determined from the background region by the determining unit, the data of the first position and attitude changes, and the data of the second position and attitude changes. ### Claim 15 (depends on 1) The image processing device according to claim 1, wherein a motion of the image data is a trajectory of motion vectors or corresponding points or points-of-interest. ### Claim 16 (depends on 1) The image processing device according to claim 1, wherein the determining unit selects a group having a greatest proportion occupied by depth from a plurality of groups obtained by the grouping unit and then determines the data of the position and attitude change corresponding to the group as data of the position and attitude change of the whole image. ### Claim 17 (depends on 16) The image processing device according to claim 16, wherein the instructions stored in the memory cause the image processing device to function as: a storage unit configured to store the data of the position and attitude change of the whole image determined by the determining unit, and wherein, when a group having a greatest proportion is present in plural, the determining unit reads data stored in the storage unit and then determines data of the position and attitude change with a smallest difference from the position and attitude change indicated by the read data as the data of the position and attitude change of the whole image. ### Claim 18 (depends on 16) The image processing device according to claim 16, wherein the instructions stored in the memory cause the image processing device to function as a selecting unit configured to select any one of the plurality of groups obtained by the grouping unit, and wherein, when a group having the greatest proportion is present in plural, the determining unit determines data of the position and attitude change corresponding to the group selected from the plurality of groups by the selecting unit as the data of the position and attitude change of the whole image. ### Claim 19 (depends on 16) The image processing device according to claim 16, wherein, when a group having the greatest proportion is present in plural, the determining unit does not determine the position and attitude change of the whole image for a reference frame obtained by segmenting the plurality of depth data. ### Claim 20 An imaging apparatus comprising: one or more first processors; a first memory storing instructions which, when the instructions are executed by the one or more first processors, cause the imaging apparatus to function as: an image acquiring unit configured to acquire a plurality of image data imaged by an imaging optical system and an imaging element; a depth image acquiring unit configured to acquire a plurality of depth data which are paired with the plurality of image data respectively; and an image processing device, wherein the image processing device comprises: one or more second processors; and a second memory storing instructions which, when the instructions are executed by the one or more second processors, cause the image processing device to function as: an acquisition unit configured to acquire the plurality of image data and the plurality of depth data corresponds to the plurality of image data respectively; a calculating unit configured to calculate first position and attitude changes for each depth from the plurality of image data and the plurality of depth data; a grouping unit configured to calculate a frequency distribution of depths corresponding to the first position and attitude changes from each of the plurality of depth data, and group the plurality of depth data in a plurality of groups based on the frequency distribution; a detection unit configured to detect motions in the plurality of image data; a region calculating unit configured to estimate regions, which have a same motion; a processing unit configured to calculate second position and attitude changes of the regions for each motion; and a determining unit configured to determine a position and attitude change of a whole image for the plurality of image data by determining a static region based on data of the first position and attitude changes which respectively correspond to the plurality of groups obtained by the grouping unit and data of the second position and attitude changes. ### Claim 21 (depends on 20) The imaging apparatus according to claim 20, wherein the instructions stored in the first memory cause the imaging apparatus to function as: a correcting unit configured to perform image shake correction for the plurality of image data; and a control unit configured to control the correcting unit by acquiring data of the position and attitude change of the whole image, which has been determined by the determining unit, and calculating an image shake correction amount. ### Claim 22 A control method to be executed by an image processing device, the method comprising: acquiring a plurality of image data and a plurality of depth data corresponding to the plurality of image data respectively; calculating first position and attitude changes for each depth from the plurality of image data and the plurality of depth data; calculating a frequency distribution of depths corresponding to the first position and attitude changes from each of the plurality of depth data; grouping the plurality of depth data in a plurality of groups based on the frequency distribution; detecting motions in the plurality of image data; estimating regions, which have a same motion; calculating second position and attitude changes of the regions for each motion; and determining a position and attitude change of a whole image for the plurality of image data by determining a static region based on data of the first position and attitude changes which respectively correspond to the plurality of groups obtained in the grouping and data of the second position and attitude changes. ### Claim 23 (depends on 22) The control method according to claim 22, further comprising: performing image shake correction using the data of the determined position and attitude change of the whole image. ### Claim 24 A non-transitory storage medium on which is stored a program for causing a computer to execute a control method to be executed by an image processing device, the method comprising: acquiring a plurality of image data and a plurality of depth data corresponds to the plurality of image data respectively; calculating first position and attitude changes for each depth from the plurality of image data and the plurality of depth data; calculating a frequency distribution of depths corresponding to the first position and attitude changes from each of the plurality of depth data; grouping the plurality of depth data in a plurality of groups based on the frequency distribution; detecting motions in the plurality of image data; estimating regions, which have a same motion; calculating second position and attitude changes of the regions for each motion; and determining a position and attitude change of a whole image for the plurality of image data by determining a static region based on data of the first position and attitude changes which respectively correspond to the plurality of groups obtained in the grouping and data of the second position and attitude changes. ### Claim 25 An imaging apparatus comprising: one or more first processors; a first memory storing instructions which, when the instructions are executed by the one or more first processors, cause the imaging apparatus to function as: a first input unit configured to input a plurality of image data of images imaged by an imaging optical system and an imaging element; a second input unit configured to input a plurality of depth data which are paired with the plurality of image data respectively; and an image processing device, wherein the image processing device comprises: one or more second processors; and a second memory storing instructions which, when the instructions are executed by the one or more second processors, cause the image processing device to function as: an acquisition unit configured to acquire the plurality of image data and the plurality of depth data corresponds to the plurality of image data respectively; a calculating unit configured to calculate first position and attitude changes for each depth from the plurality of image data and the plurality of depth data; a grouping unit configured to calculate a frequency distribution of depths corresponding to the first position and attitude changes from each of the plurality of depth data, and group the plurality of depth data in a plurality of groups based on the frequency distribution; a detection unit configured to detect motions in the plurality of image data; a region calculating unit configured to estimate regions, which have a same motion; a processing unit configured to calculate second position and attitude changes of the regions for each motion; and a determining unit configured to determine a position and attitude change of a whole image for the plurality of image data by determining a static region based on data of the first position and attitude changes which respectively correspond to the plurality of groups obtained by the grouping unit and data of the second position and attitude changes. ### Claim 26 (depends on 25) The imaging apparatus according to claim 25, wherein the instructions stored in the first memory cause the imaging apparatus to function as: a correcting unit configured to perform image shake correction for the plurality of image data; and a control unit configured to control the correcting unit by acquiring data of the position and attitude change of the whole image, which has been determined by the determining unit, and calculating an image shake correction amount.