# US 11,443,540 B2 · Information processing apparatus and information processing method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,540 B2 - Document ID: US11443540B2 - Status: Granted - Kind: B2 - Application number: 16/964,589 - Filed: 2019-01-17 - Published: 2021-02-11 - Granted: 2022-09-13 - Earliest claimed: 2018-02-07 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Takaaki Kato; Shunichi Homma; Yuya Yamaguchi; Masaki Fukuchi - Web page: https://patentgrep.com/patent/US11443540B2 ## CPC - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G01C 21/26: specially adapted for navigation in a road network - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 17/05: Geographic models - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract There is provided an information processing apparatus and an information processing method capable of performing position estimation of a controller corresponding to a wearable device having an inside-out type. The information processing apparatus includes a user-position estimating unit that recognizes, based on a first captured image that is captured by a first image capturing unit provided to a wearable device mounted on a user, peripheral environment information of the user to execute position estimation of the user in a real space and a controller-position estimating unit that executes, based on a second captured image captured by a second image capturing unit that is provided to a controller being operated by the user, position estimation of the controller with reference to the recognized environment information. ## Claims ### Claim 1 An information processing apparatus, comprising: a user-position estimating unit configured to: recognize, based on a first captured image that is captured by a first image capturing unit, peripheral environment information of a user, wherein the first image capturing unit is in a wearable device mountable on a head of the user, and the wearable device includes a head mounted display; estimate a position of the user in a real space based on the recognized peripheral environment information; and estimate a posture of the user; a content display controlling unit configured to generate a content image based on the estimated position and the estimated posture of the user; a controller-position estimating unit configured to estimate a position of a controller with reference to the recognized peripheral environment information of the user, wherein the estimation of the position of the controller is based on: a position of a feature point in a second captured image captured by a second image capturing unit, and an orientation of the wearable device, and the second image capturing unit is in the controller operated by the user; and a display controlling unit configured to control the head mounted display to display the content image. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the controller-position estimating unit is further configured to estimate a posture of the controller. ### Claim 3 (depends on 2) The information processing apparatus according to claim 2, further comprising a mapping unit configured to generate, based on the first captured image, the peripheral environment information of the user as an environment map, wherein the environment map is stored in a storage unit. ### Claim 4 (depends on 3) The information processing apparatus according to claim 3, wherein the mapping unit is further configured to generate the environment map based on the second captured image. ### Claim 5 (depends on 3) The information processing apparatus according to claim 3, wherein the user-position estimating unit is further configured to execute, based on the first captured image, a simultaneous localization and mapping (SLAM) to estimate the position of the user. ### Claim 6 (depends on 3) The information processing apparatus according to claim 3, wherein the content display controlling unit is further configured to generate the content image based on the estimated position and the estimated posture of the controller. ### Claim 7 (depends on 6) The information processing apparatus according to claim 6, wherein the information processing apparatus is realized by the wearable device. ### Claim 8 (depends on 1) The information processing apparatus according to claim 1, wherein the controller-position estimating unit is further configured to estimate the position and a posture of the controller with reference to first posture information detected by a posture sensor in the controller, and the first posture information is detected at a first time instant. ### Claim 9 (depends on 8) The information processing apparatus according to claim 8, wherein the posture sensor includes a gyro acceleration sensor. ### Claim 10 (depends on 8) The information processing apparatus according to claim 8, further comprising a position correcting unit configured to correct, based on second posture information acquired from the controller, the position and the posture of the controller, wherein the second posture information is detected at a second time instant different from the first time instant. ### Claim 11 (depends on 1) The information processing apparatus according to claim 1, wherein the controller-position estimating unit is further configured to estimate, based on gyro acceleration information transmitted from the controller and an Inertial Navigation System (INS), the position and a posture of the controller. ### Claim 12 (depends on 1) The information processing apparatus according to claim 1, further comprising a communication unit, configured to transmit, to the controller, information associated with the estimated position of the controller. ### Claim 13 (depends on 1) The information processing apparatus according to claim 1, wherein the controller-position estimating unit is further configured to: receive posture data from the controller at a first time period for the estimation of the position of the controller; and receive both the posture data and the second captured image at a second time period for the estimation of the position of the controller. ### Claim 14 (depends on 1) The information processing apparatus according to claim 1, further comprising a communication unit configured to transmit, to the controller, a collision determination result with a virtual object, wherein the collision determination result causes the controller to vibrate. ### Claim 15 An information processing method, comprising: recognizing, based on a first captured image that is captured by a first image capturing unit, peripheral environment information of a user, wherein the first image capturing unit is in a wearable device mountable on a head of the user, and the wearable device includes a head mounted display; estimating a position of the user in a real space based on the recognized peripheral environment information; estimating a posture of the user; generating a content image based on the estimated position and the estimated posture of the user; estimating a position of a controller with reference to the recognized peripheral environment information of the user, wherein the estimation of the position of the controller is based on: a position of a feature point in a second captured image captured by a second image capturing unit, and an orientation of the wearable device, and the second image capturing unit is in the controller operated by the user; and controlling the head mounted display to display the content image. ### Claim 16 A non-transitory computer-readable medium having stored thereon, computer-executable instructions, which when executed by a computer, cause the computer to execute operations, the operations comprising: recognizing, based on a first captured image that is captured by a first image capturing unit, peripheral environment information of a user, wherein the first image capturing unit is in a wearable device mountable on a head of the user, and the wearable device includes a head mounted display; estimating a position of the user in a real space based on the recognized peripheral environment information; estimating a posture of the user; generating a content image based on the estimated position and the estimated posture of the user; estimating a position of a controller with reference to the recognized peripheral environment information of the user, wherein the estimation of the position of the controller is based on: a position of a feature point in a second captured image captured by a second image capturing unit, and an orientation of the wearable device, and the second image capturing unit is in the controller operated by the user; and controlling the head mounted display to display the content image.