# US 12,008,778 B2 · Information processing apparatus, control method for same, non-transitory computer-readable storage medium, and vehicle driving support system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,778 B2 - Document ID: US12008778B2 - Status: Granted - Kind: B2 - Application number: 16/921,977 - Filed: 2020-07-07 - Published: 2020-10-22 - Granted: 2024-06-11 - Earliest claimed: 2018-01-15 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Nozomu Kasuya; Kazuhiko Kobayashi; Masahiro Suzuki; Daisuke Kotake; Makoto Tomioka - Web page: https://patentgrep.com/patent/US12008778B2 ## CPC - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - G06T 7/593: from stereo images - G01B 11/00: Measuring arrangements characterised by the use of optical techniques - G01C 3/06: Use of electric means to obtain final indication - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 7/4814: {of transmitters alone} - G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48) - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)} - G05D 1/2435: {Extracting 3D information} - G06T 7/11: Region-based segmentation - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 7/579: from motion - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G08G 1/16: Anti-collision systems (road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08) - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - G06T 2207/10012: Stereo images - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract This invention provides an information processing apparatus comprising a first acquiring unit configured to acquire first distance information from a distance sensor, a second acquiring unit configured to acquire an image from an image capturing device, a holding unit configured to hold a learning model for estimating distance information from images, an estimating unit configured to estimate second distance information corresponding to the image acquired by the second acquiring unit using the learning model, and a generating unit configured to generate third distance information based on the first distance information and the second distance information. ## Claims ### Claim 1 An information processing apparatus comprising: (A) one or more circuits, or (B) at least one processor and at least one memory, or (C) at least one processor, at least one memory, and one or more circuits, wherein the (A) one or more circuits, (B) at least one processor and at least one memory, or (C) at least one processor, at least one memory, and one more circuits is configured to function as a plurality of units comprising: (1) a first acquiring unit configured to acquire first distance image information comprising distance as a pixel value using a distance sensor; (2) a second acquiring unit configured to acquire an image captured by an image capturing device; (3) an estimating unit configured to estimate second distance image information comprising distance as a pixel value from the image acquired by the second acquiring unit using a learning model which has been generated so as to output distance image information corresponding to an input image; and (4) a calculation unit configured to calculate third distance image information comprising distance as a pixel value in accordance with (a) the first distance image information and (b) the second distance image information. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the plurality of units further comprises: a map holding unit configured to hold, as a three-dimensional map, a position and orientation of the image capturing device and distance image information corresponding to the position and orientation; a position-and-orientation-calculation unit configured to calculate a position and orientation of the image capturing device based on the three-dimensional map and the image acquired by the second acquiring unit; an updating unit configured to update the three-dimensional map based on the position and orientation calculated by the position-and-orientation-calculation unit and the second distance image information estimated by the estimating unit; and a degree-of-reliability-determining unit configured to determine a degree of reliability of the first distance image information and the distance image information in the three-dimensional map, and wherein the calculation unit calculates the third distance image information based on (1) the first distance image information, (2) the distance image information in the three-dimensional map updated by the updating unit, and (3) the degree of reliability. ### Claim 3 (depends on 2) The information processing apparatus according to claim 2, wherein the degree-of-reliability-determining unit determines the degree of reliability based on whether or not the first distance image information is present. ### Claim 4 (depends on 2) The information processing apparatus according to claim 2, wherein the distance sensor is a LIDAR-based sensor, and wherein the degree-of-reliability-determining unit determines a degree of reliability of the first distance image information in accordance with the intensity of laser light reflected from a target object. ### Claim 5 (depends on 2) The information processing apparatus according to claim 2, wherein the degree-of-reliability-determining unit determines a degree of reliability of the first distance image information based on an operating distance of a sensor. ### Claim 6 (depends on 2) The information processing apparatus according to claim 2, wherein the degree-of-reliability-determining unit determines the degree of reliability based on continuity of distance information. ### Claim 7 (depends on 2) The information processing apparatus according to claim 2, wherein the distance sensor is a stereo-type distance sensor, and wherein the degree-of-reliability-determining unit performs the determination based on a degree of similarity in stereo matching performed when the distance sensor acquires distance image information. ### Claim 8 (depends on 2) The information processing apparatus according to claim 2, wherein the plurality of units further comprises a semantic region segmenting unit configured to segment the image acquired by the second acquiring unit into semantic regions, and wherein the calculation unit combines the first distance image information and the distance image information in the three-dimensional map updated by the updating unit based on the semantic regions. ### Claim 9 (depends on 1) The information processing apparatus according to claim 1, wherein a position and an orientation of the information processing apparatus are estimated using the third distance image information. ### Claim 10 (depends on 1) The information processing apparatus according to claim 1, wherein the plurality of units further comprises: an environment recognizing unit configured to recognize a surrounding environment based on the third distance image information and the image acquired by the second acquiring unit; and a position-and-orientation-acquiring unit configured to acquire a position and an orientation of the apparatus relative to the environment. ### Claim 11 (depends on 2) A vehicle driving support system comprising: the distance sensor; the image capturing device; and the information processing apparatus according to claim 2. ### Claim 12 (depends on 11) The vehicle driving support system according to claim 11, further comprising an actuator for controlling the movement of a vehicle, wherein the plurality of units further comprises a driving-control unit configured to control the actuator based on (1) the surrounding environment, (2) the position, and (3) the orientation. ### Claim 13 A control method for an information processing apparatus comprising a holding unit which holds a learning model for estimating distance information from images, the control method comprising: acquiring first distance image information comprising distance as a pixel value using a distance sensor; acquiring an image captured by an image capturing device; estimating second distance image information comprising distance as a pixel value from the acquired image using a learning model which has been generated so as to output distance image information corresponding to an input image; and calculating third distance image information comprising distance as a pixel value in accordance with (a) the first distance image information and (b) the second distance image information. ### Claim 14 A non-transitory computer-readable storage medium storing a program which causes, when read and executed by a computer, the computer to execute a control method for an information processing apparatus comprising a holding unit which holds a learning model for estimating distance information from images, the control method comprising: acquiring first distance image information comprising distance as a pixel value using a distance sensor; acquiring an image captured by an image capturing device; estimating second distance image information comprising distance as a pixel value from the acquired image using a learning model which has been generated so as to output distance image information corresponding to an input image; and calculating third distance image information comprising distance as a pixel value in accordance with (a) the first distance image information and (b) the second distance image information. ### Claim 15 (depends on 1) The information processing apparatus according to claim 1, wherein the plurality of units further comprises a degree-of-reliability-determining unit configured to determine a first degree of reliability of the first distance image information and a second degree of reliability of the second distance image information, and wherein the calculation unit calculates the third distance image information by performing weighted averaging of the first distance image information and the second distance image information using the first degree of reliability and the second degree of reliability. ### Claim 16 (depends on 15) The information processing apparatus according to claim 15, wherein the distance sensor is a LIDAR based sensor, and wherein the degree-of-reliability-determining unit determines the first degree of reliability in accordance with the intensity of laser light reflected from a target object, and determines the second degree of reliability in accordance with likelihood of the estimation of the second distance image information using the learning model. ### Claim 17 (depends on 15) The information processing apparatus according to claim 15, wherein the degree-of-reliability-determining unit determines the first degree of reliability and the second degree of reliability based on continuity of distance image information. ### Claim 18 (depends on 15) The information processing apparatus according to claim 15, wherein the distance sensor is a stereo-type distance sensor, and wherein the degree-of-reliability-determining unit performs the determination based on a degree of similarity in stereo matching performed when the distance sensor acquires distance image information.