# US 10,310,595 B2 · Information processing apparatus, information processing method, computer program, and image processing system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,310,595 B2 - Document ID: US10310595B2 - Status: Granted - Kind: B2 - Application number: 15/311,341 - Filed: 2015-04-21 - Published: 2017-03-23 - Granted: 2019-06-04 - Earliest claimed: 2014-06-30 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Hirokazu Tatsuta; Kenta Kawamoto; Yuichiro Koyama; Takayasu Kon; Yuichi Hasegawa; Masaru Shimura; Yuki Yamamoto; Eiichi Tanaka - Web page: https://patentgrep.com/patent/US10310595B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G02B 27/017: {Head mounted} - G06F 3/012: {Head tracking input arrangements} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/048: Interaction techniques based on graphical user interfaces [GUI] - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/147: using display panels - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 15/20: Perspective computation - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/161: {Detection; Localisation; Normalisation} - H04N 7/185: {from a mobile camera, e.g. for remote control} - H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD] - H04N 13/366: using viewer tracking - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G09G 3/003: {to produce spatial visual effects} - G09G 2320/068: for control of viewing angle adjustment - G09G 2320/0693: Calibration of display systems - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract Provided is an image display device configured to display a free viewpoint image that tracks a movement of observer's head and trunk. The line-of-sight direction and trunk orientation of the user are individually oriented in a free viewpoint space displayed on a display device 500 based on posture information of the user's head and trunk, which are respectively obtained from a head motion tracking device 200 and a trunk motion tracking device 300. A behavior that is natural in real space can be represented by moving a line of sight to a front direction of the trunk on a space in a state where the line of sight does not match the front direction of the trunk. An unnatural UI that goes straight to the line-of-sight direction is prevented regardless of orientation of the trunk. ## Claims ### Claim 1 An information processing apparatus comprising: a receiver configured to receive first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; and an image rendering processing unit configured to generate a display image corresponding to a posture of the observer based on the first information and the second information, wherein the receiver receives at least a posture of a trunk of the observer as the second information, and the image rendering processing unit is configured to generate a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the first information and by orienting an orientation (viewpoint position) of the body of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the second information, and when receiving a control signal used to instruct the observer to move in the free viewpoint space as an input, determine a movement amount in a front direction of the trunk of the observer in the free viewpoint space based on the posture of the trunk of the observer indicated in the second information and by recognizing the orientation of the body of the observer as a position of a front of the observer. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the image rendering processing unit generates the free viewpoint image that tracks the posture of the head of the observer by arranging in a fixed position oriented based on the posture obtained from the second information. ### Claim 3 (depends on 1) The information processing apparatus according to claim 1, wherein the image rendering processing unit generates the free viewpoint image that tracks the posture of the head of the observer by arranging a predetermined image widget in a fixed position oriented based on the posture of the trunk of the observer. ### Claim 4 (depends on 1) The information processing apparatus according to claim 1, further comprising: a calibration processing unit configured to acquire a calibration parameter, wherein the image rendering processing unit performs image generation using posture information corrected by the calibration parameter. ### Claim 5 An information processing apparatus comprising: a receiver configured to receive first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; an image rendering processing unit configured to generate a display image corresponding to a posture of the observer based on the first information and the second information; and a calibration processing unit configured to acquire a calibration parameter for mutually converting a coordinate system in which the posture of the head of the observer is indicated in the first information and a coordinate system in which the posture of the body other than the head of the observer is indicated in the second information, wherein the image rendering processing unit performs image generation using posture information corrected by the calibration parameter, and wherein the calibration processing unit, based on head posture information obtained from the first information and second posture information obtained from the second information, calculates a calibration parameter used to correct the second posture information and corrects the second posture information using the calibration parameter. ### Claim 6 (depends on 5) The information processing apparatus according to claim 5, wherein the calibration processing unit calculates a posture conversion parameter used to match the second posture information with the head posture information as the calibration parameter, by using the head posture information and the second posture information. ### Claim 7 (depends on 5) The information processing apparatus according to claim 5, wherein the posture information is represented by a quaternion, the calibration processing unit calculates a calibration quaternion used to correct the second posture information by multiplying the head posture information by an inverse of a quaternion of the second posture information from a right side, and the correction is performed by multiplying the quaternion of the second posture information by the calibration quaternion from a left side. ### Claim 8 An information processing apparatus comprising: a receiver configured to receive first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; an image rendering processing unit configured to generate a display image corresponding to a posture of the observer based on the first information and the second information; and a calibration processing unit configured to acquire a calibration parameter, wherein the image rendering processing unit performs image generation using posture information corrected by the calibration parameter, and wherein the calibration processing unit estimates a calibration parameter used to correct the posture information based on rotational displacement for a certain period of a coordinate system of a head posture obtained from the first information and a coordinate system of a posture obtained from the second information and a property of a human body. ### Claim 9 An information processing apparatus comprising: a receiver configured to receive first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; an image rendering processing unit configured to generate a display image corresponding to a posture of the observer based on the first information and the second information; and a calibration processing unit configured to acquire a calibration parameter, wherein the image rendering processing unit performs image generation using posture information corrected by the calibration parameter, and wherein the calibration processing unit sets a time-series average of calibration parameters calculated for a certain period based on head posture information obtained from the first information and second posture information obtained from the second information to be a final calibration parameter. ### Claim 10 (depends on 9) The information processing apparatus according to claim 9, wherein the posture information is represented by a quaternion, and the calibration processing unit updates a calibration quaternion by performing spherical linear interpolation between a calibration quaternion newly calculated based on a quaternion of posture information obtained from each of the first information and the second information that have been newly received and a last calibration quaternion determined by the time-series average. ### Claim 11 (depends on 9) The information processing apparatus according to claim 9, wherein a posture of the trunk of the observer is calculated based on the second information, and the calibration processing unit determines an average value of an angular difference of a rotation direction, when a gravity direction is set to a rotation axis, of posture information of a head and a trunk obtained respectively from the first information and the second information that have been newly received, and calculates a calibration parameter on a gravity direction and the average value. ### Claim 12 (depends on 1) An information processing apparatus according to claim 1, wherein the image rendering processing unit processes a display image on a display unit based on a difference between a head posture obtained from the first information and a trunk posture obtained from the second information. ### Claim 13 An information processing apparatus comprising: a receiver configured to receive first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; an image rendering processing unit configured to generate a display image corresponding to a posture of the observer based on the first information and the second information, wherein the receiver receives at least a posture of a trunk of the observer as the second information, wherein the image rendering processing unit generates a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the first information and by orienting an orientation (viewpoint position) of the body of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the second information, and wherein the image rendering processing unit generates a free viewpoint image by correcting a world coordinate system in a manner that a difference between a head posture and a trunk posture in the free viewpoint space decreases under a biosafety support mode. ### Claim 14 An information processing apparatus comprising: a receiver configured to receive first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; and an image rendering processing unit configured to generate a display image corresponding to a posture of the observer based on the first information and the second information, wherein the receiver receives at least a posture of a trunk of the observer as the second information, wherein the image rendering processing unit generates a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the first information and by orienting an orientation (viewpoint position) of the body of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the second information, and wherein the image rendering processing unit generates a free viewpoint image by fixing a head posture in a free viewpoint space at a point of time when a biosafety support mode is set and by changing a trunk posture in a free viewpoint space depending on a head posture detected by ahead posture detection unit under the biosafety support mode. ### Claim 15 (depends on 13) The information processing apparatus according to claim 13, wherein the biosafety support mode is set when a state where a difference between a head posture obtained from the first information and a trunk posture obtained from the second information exceeds a first threshold continues for a certain time, and the biosafety support mode is released when the difference is equal to or less than a second threshold that is less than the first threshold. ### Claim 16 An information processing method comprising: a receiving step of receiving first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; and an image rendering processing step of generating a display image corresponding to a posture of the observer based on the first information and the second information, wherein at least a posture of a trunk of the observer as the second information is received in the receiving step, and wherein, in the image rendering processing step, a free viewpoint image is generated that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the first information and by orienting an orientation (viewpoint position) of the body of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the second information, and when a control signal used is received to instruct the observer to move in the free viewpoint space as an input, a movement amount in a front direction of the trunk of the observer in the free viewpoint space is determined based on the posture of the trunk of the observer indicated in the second information and by recognizing the orientation of the body of the observer as a position of a front of the observer. ### Claim 17 A non-transitory storage medium configured to store a computer program written in a computer readable format to cause a computer to function as: a head posture operation unit configured to calculate posture information of a head of an observer based on a result obtained by detection in a head posture detection unit configured to detect a posture of the head of the observer; a second posture operation unit configured to calculate posture information of a second part of a body other than the head of the observer based on a result obtained by detection in a second posture detection unit configured to detect a posture of one or more of the second part; and an image rendering processing unit configured to process an image to be displayed on a display unit based on a posture of the head of the observer and a posture of the second part, the display unit being fixed to the head or a face of the observer, wherein the detection in the second posture detection unit detects at least a posture of a trunk of the observer as the one or more of the second part, and wherein the image rendering processing unit is configured to generate a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the posture information of the head of the observer calculated in the head posture operation unit and by orienting an orientation (viewpoint position) of the body other than the head of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the posture information of the second part of the body other than the head calculated in the second posture operation unit, and when receiving a control signal used to instruct the observer to move in the free viewpoint space as an input, determine a movement amount in a front direction of the trunk of the observer in the free viewpoint space based on the posture of the trunk of the observer indicated in the second information and by recognizing the orientation of the body of the observer as a position of a front of the observer. ### Claim 18 An image processing system comprising: a display unit fixed to a head or a face of an observer; a head posture detection unit configured to detect a posture of the head of the observer; a second posture detection unit configured to detect a posture of one or more second parts of a body other than the head of the observer; and an image rendering processing unit configured to process a display image on the display unit based on a posture of the head of the observer and a posture of each of the second parts, wherein detection in the second posture detection unit detects at least a posture of a trunk of the observer as the one or more second parts, and wherein the image rendering processing unit is configured to generate a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on detection of the posture of the head of the observer by the head posture detection unit and by orienting an orientation (viewpoint position) of the one or more second parts of the body other than the head of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the detection in the second posture detection unit, and when receiving a control signal used to instruct the observer to move in the free viewpoint space as an input, determine a movement amount in a front direction of the trunk of the observer in the free viewpoint space based on the posture of the trunk of the observer indicated in the second information and by recognizing the orientation of the body of the observer as a position of a front of the observer.