# US 11,906,970 B2 · Information processing device and information processing method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,906,970 B2 - Document ID: US11906970B2 - Status: Granted - Kind: B2 - Application number: 17/601,497 - Filed: 2020-03-04 - Published: 2022-06-23 - Granted: 2024-02-20 - Earliest claimed: 2019-04-15 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Kento Suzuki; Atsushi Okamori - Web page: https://patentgrep.com/patent/US11906970B2 ## CPC - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - G05D 1/648: Performing a task within a working area or space, e.g. cleaning - G01C 21/1652: {with ranging devices, e.g. LIDAR or RADAR} - G01C 21/206: {specially adapted for indoor navigation} - G01C 21/30: Map- or contour-matching - G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons - G05D 1/617: Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80) - G08G 1/00: Traffic control systems for road vehicles (arrangement of road signs or traffic signals E01F9/00 {; automatic vehicle control B62D}) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract An information processing device according to the present disclosure includes an acquisition unit that acquires an arrangement of a landmark and surrounding terrain data, a first estimation unit that estimates a range in which a self-position is located on a basis of a landmark map that is a map indicating the arrangement of the landmark, and a second estimation unit that estimates the self-position from the range in which the self-position is located on a basis of a terrain data map that is a map indicating the surrounding terrain data. ## Claims ### Claim 1 An information processing device, comprising: an image sensor configured to capture an image of a landmark; a depth sensor configured to detect depth data associated with a terrain surrounding the information processing device; and circuitry configured to: execute an image recognition process on the captured image; acquire an arrangement of the landmark based on the image recognition process on the captured image; acquire a landmark movement range of the landmark; generate, based on the detected depth data, a first terrain data map indicating surrounding terrain data of the terrain; estimate, based on the arrangement of the landmark acquired, the landmark movement range and a landmark map indicating the arrangement of the landmark, a range in which a self-position of the information processing device is located; position, on the first terrain data map, the estimated range in which the self-position is located; generate a second terrain data map based on the position of the estimated range on the first terrain data map; estimate, based on the second terrain data map and the landmark movement range, the self-position from the range in which the self-position is located; and control self-driving of the information processing device based on the estimated self-position. ### Claim 2 (depends on 1) The information processing device according to claim 1, wherein the circuitry is further configured to: acquire movement ranges of a plurality of landmarks together with an arrangement of the plurality of landmarks, wherein the plurality of landmarks includes the landmark; estimate the range in which the self-position is located, based on the arrangement of the plurality of landmarks and the movement ranges of the plurality of landmarks; and estimate the self-position from the range in which the self-position is located, based on the second terrain data map and the movement ranges of the plurality of landmarks. ### Claim 3 (depends on 2) The information processing device according to claim 2, wherein the circuitry is further configured to: estimate a plurality of candidates for the self-position based on a movement range of each landmark of the plurality of landmarks; and estimate the self-position from the range in which the self-position is located based on the plurality of candidates for the self-position. ### Claim 4 (depends on 1) The information processing device according to claim 1, wherein the landmark movement range is a movement range that is set in advance for each category of the landmark. ### Claim 5 (depends on 1) The information processing device according to claim 1, wherein the circuitry is further configured to: acquire control data that controls an operation of the information processing device; and estimate the self-position from the range in which the self-position is located, based on the second terrain data map, the landmark movement range and the control data. ### Claim 6 (depends on 1) The information processing device according to claim 1, wherein the landmark includes personal belongings including at least one of a stationery or a tableware that can be carried by a person. ### Claim 7 (depends on 1) The information processing device according to claim 1, wherein the image sensor is a red green blue (RGB) camera, and the depth sensor is configured to detect the depth data using a light detection and ranging, or laser imaging detection and ranging (LiDAR) method. ### Claim 8 An information processing method, comprising: capturing, by an image sensor of an information processing device, an image of a landmark; detecting, by a depth sensor of the information processing device, depth data associated with a terrain surrounding the information processing device; executing, by circuitry of the information processing device, an image recognition process on the captured image; acquiring, by the circuitry, an arrangement of the landmark based on the image recognition process on the captured image; acquiring, by the circuitry, a landmark movement range of the landmark; generating, by the circuitry, based on the detected depth data, a first terrain data map indicating surrounding terrain data of the terrain; estimating, by the circuitry, based on the arrangement of the landmark acquired, the landmark movement range and a landmark map indicating the arrangement of the landmark, a range in which a self-position of the information processing device is located; positioning, by the circuitry, on the first terrain data map, the estimated range in which the self-position is located; generating, by the circuitry, a second terrain data map based on the position of the estimated range on the first terrain data map; estimating, by the circuitry, based on the second terrain data map and the landmark movement range, the self-position in the range in which the self-position is located; and controlling, by the circuitry, self-driving of the information processing device based on the estimated self-position. ### Claim 9 A non-transitory computer-readable medium having stored thereon, computer-executable instructions which, when executed by an information processing device, cause the information processing device to execute operations, the operations comprising: capturing, by an image sensor of the information processing device, an image of a landmark; detecting, by a depth sensor of the information processing device, depth data associated with a terrain surrounding the information processing device; executing, by circuitry of the information processing device, an image recognition process on the captured image; acquiring, by the circuitry, an arrangement of the landmark based on the image recognition process on the captured image; acquiring, by the circuitry, a landmark movement range of the landmark; generating, by the circuitry, based on the detected depth data, a first terrain data map indicating surrounding terrain data of the terrain; estimating, by the circuitry, based on the arrangement of the landmark acquired, the landmark movement range and a landmark map indicating the arrangement of the landmark, a range in which a self-position of the information processing device is located; positioning, by the circuitry, on the first terrain data map, the estimated range in which the self-position is located; generating, by the circuitry, a second terrain data map based on the position of the estimated range on the first terrain data map; estimating, by the circuitry, based on the second terrain data map and the landmark movement range, the self-position from the range in which the self-position is located; and controlling, by the circuitry, self-driving of the information processing device based on the estimated self-position.