# US 12,112,502 B2 · Three-dimensional map estimation apparatus and obstacle detection apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,502 B2 - Document ID: US12112502B2 - Status: Granted - Kind: B2 - Application number: 17/464,455 - Filed: 2021-09-01 - Published: 2022-10-27 - Granted: 2024-10-08 - Earliest claimed: 2021-04-21 - Assignees: Kabushiki Kaisha Toshiba; Toshiba Electronic Devices & Storage Corporation - Applicants: Kabushiki Kaisha Toshiba; Toshiba Electronic Devices & Storage Corporation - Inventors: Manabu Nishiyama - Web page: https://patentgrep.com/patent/US12112502B2 ## CPC - G06T 7/74: {involving reference images or patches} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G01S 13/89: for mapping or imaging - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/579: from motion - G06T 17/05: Geographic models - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G08G 1/165: {for passive traffic, e.g. including static obstacles, trees} - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2554/4029: Pedestrians - G06T 2207/30244: Camera pose - G06T 2207/30248: Vehicle exterior or interior - G06T 2207/30261: Obstacle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract According to one embodiment, a three-dimensional map estimation apparatus includes a processor that selects an imaging apparatus from a plurality of imaging apparatuses and then estimates a position and orientation for a moving object on which the selected imaging apparatus is mounted based on images captured by the selected imaging apparatus. The processor outputs a first position and orientation estimation result for the moving object based on images from selected imaging apparatuses. The processor calculates a second position and orientation estimation result indicating an estimated position and orientation for the moving object using the first position and orientation estimation result. The processor estimates a three-dimensional map for the surroundings of the moving object based on the second position and orientation estimation result. ## Claims ### Claim 1 A three-dimensional map estimation apparatus, comprising: a processor configured to: select an imaging apparatus from a plurality of imaging apparatuses; estimate a position and orientation for a moving object on which the selected imaging apparatus is mounted based on images captured by the selected imaging apparatus then output the estimated position and orientation for the moving object as a first position and orientation estimation result; calculate a second position and orientation estimation result indicating an estimated position and orientation for the moving object, the second position and orientation estimation result being calculated using the first position and orientation estimation result; and estimate a three-dimensional map for the surroundings of the moving object based on the second position and orientation estimation result. ### Claim 2 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the processor is further configured to: convert a change in position and orientation estimated for one imaging apparatus of the plurality of imaging apparatuses into a change in position and orientation of another one of the imaging apparatuses in the plurality of imaging apparatuses based on the second position and orientation estimation result. ### Claim 3 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the processor is further configured to: calculate a similarity value for each of a plurality of first position and orientation estimation results that are respectively based on images from different imaging apparatuses in the plurality of imaging apparatus and then select one or more of the first position and orientation estimation results, based on the calculated similarity values, to be used in calculating the second position and orientation estimation result. ### Claim 4 (depends on 3) The three-dimensional map estimation apparatus according to claim 3, wherein the processor is configured to estimate the three-dimensional map for the surroundings of the moving object based on the second position and orientation estimation result and at least one image from an imaging apparatus selected for estimating a first position and orientation estimation result but not selected for calculating the second position and orientation estimation result. ### Claim 5 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the processor is further configured to: estimate a position and orientation for the moving object based on sensor information from a sensor on the moving object. ### Claim 6 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the processor selects the imaging apparatus based on a movement direction of the moving object or a movement speed of the moving object. ### Claim 7 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the processor is further configured to: a detect an object in the surroundings of the moving object based on the three-dimensional map. ### Claim 8 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the second position and orientation estimation result is a weighted average of a plurality of first position and orientation estimation results. ### Claim 9 (depends on 8) The three-dimensional map estimation apparatus according to claim 8, wherein a weighting coefficient of each of the plurality of first position and orientation estimation results is based on a reliability value for the corresponding imaging apparatus or a reliability value for the first position and orientation estimation result. ### Claim 10 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the three-dimensional map is estimated using feature points in images acquired at two different times. ### Claim 11 (depends on 1) The three-dimensional map estimation apparatus according to claim 1, wherein the processor estimates the three-dimensional map based on the second position and orientation estimation result and at least one image from an imaging apparatus other than the selected imaging apparatus. ### Claim 12 (depends on 1) An obstacle detection apparatus, comprising: a three-dimensional map estimation apparatus according to claim 1; and an obstacle detection unit configured to detect an obstacle by using a three-dimensional map estimated by the three-dimensional map estimation apparatus. ### Claim 13 A three-dimensional map estimation apparatus, comprising: an interface connectable to a plurality of cameras; a processor configured to: select at least one camera from the plurality of cameras from which to receive, via the interface, images of the surroundings of a moving object; estimate a position and orientation for the moving object using images from each selected camera in the plurality of cameras and output first position and orientation estimation results based on images from each selected imaging apparatus; calculate a second position and orientation estimation result indicating an estimated position and orientation for the moving object, the second position and orientation estimation result being calculated using the first position and orientation estimation results; and estimate three-dimensional maps for the surroundings of the moving object based on the images from each camera in the plurality of cameras and the second position and orientation estimation result. ### Claim 14 (depends on 13) The three-dimensional map estimation apparatus according to claim 13, wherein the second position and orientation estimation result is a weighted average of a plurality of first position and orientation estimation results. ### Claim 15 (depends on 13) The three-dimensional map estimation apparatus according to claim 13, wherein the processor is further configured to: a detect an object in the surroundings of the moving object based on at least one of the estimated three-dimensional maps. ### Claim 16 (depends on 13) The three-dimensional map estimation apparatus according to claim 13, wherein the processor selects two or more cameras, and the second position and orientation estimation result is calculated using less than all of the first position and orientation estimation results. ### Claim 17 (depends on 13) The three-dimensional map estimation apparatus according to claim 13, wherein the interface is connectable to a sensor, and the processor is further configured to: estimate a position and orientation for the moving object using information from the sensor and output a first position and orientation estimation result based on the information from the sensor. ### Claim 18 A moving object, comprising: a plurality of cameras; a processor connected to the plurality of cameras and configured to: select at least one camera from the plurality of cameras from which to receive images; estimate a position and orientation for the moving object using images from each selected camera in the plurality of cameras and output first position and orientation estimation results based on images from each selected imaging apparatus; calculate a second position and orientation estimation result indicating an estimated position and orientation for the moving object, the second position and orientation estimation result being calculated using the first position and orientation estimation results; and estimate three-dimensional maps for the surroundings of the moving object based on the images from each camera in the plurality of cameras and the second position and orientation estimation result. ### Claim 19 (depends on 18) The moving object according to claim 18, further comprising: a sensor connected to the processor, wherein the processor is further configured to estimate a position and orientation for the moving object using information from the sensor and output a first position and orientation estimation result based on the information from the sensor. ### Claim 20 (depends on 18) The moving object according to claim 18, wherein the processor is further configured to: a detect an object in the surroundings of the moving object based on at least one of the three-dimensional maps.