# US 12,118,733 B2 · Apparatus and method for virtual reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,733 B2 - Document ID: US12118733B2 - Status: Granted - Kind: B2 - Application number: 18/145,540 - Filed: 2022-12-22 - Published: 2023-07-06 - Granted: 2024-10-15 - Earliest claimed: 2021-12-31 - Assignees: Sony Interactive Entertainment Europe Limited - Applicants: Sony Interactive Entertainment Europe Limited - Inventors: Sharwin Winesh Raghoebardajal; Jose Rubio; Steven Tattersall - Web page: https://patentgrep.com/patent/US12118733B2 ## CPC - G06T 7/248: {involving reference images or patches} - G06T 7/254: involving subtraction of images - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0172: {characterised by optical features} - G02B 2027/0134: {of stereoscopic type} - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract A data processing apparatus comprises receiving circuitry to receive tracking data indicative of at least one of a tracked position and orientation of a head-mountable display (HMD), image processing circuitry to generate a sequence of images frames for display in dependence upon the tracking data, detection circuitry to detect an image feature in a first image frame and to detect a corresponding image feature in a second image frame, and correlation circuitry to: calculate a difference between the image feature in the first image frame and the corresponding image feature in the second image frame; generate difference data indicative of a difference between a viewpoint for the first image frame and a viewpoint for the second image frame in dependence upon the difference between the image feature in the first image frame and the corresponding image feature in the second image frame; and generate output data in dependence upon a difference between the difference data and the tracking data associated with the first and second image frames. ## Claims ### Claim 1 A data processing apparatus comprising: receiving circuitry to receive tracking data indicative of at least one of a tracked position and a tracked orientation of a head-mountable display (HMD); image processing circuitry to generate a sequence of image frames for display in dependence upon the tracking data for the HMD, in which the image processing circuitry is configured to generate the sequence of image frames with viewpoints that change according to changes in at least one of the tracked position and the tracked orientation of the HMD indicated by the tracking data for the HMD; detection circuitry to detect image features in the sequence of image frames generated by the image processing circuitry, in which the detection circuitry is configured to detect at least an image feature included in a first image frame generated for display at a first time with a first viewpoint and a corresponding image feature included in a second image frame generated for display at a second time with a second viewpoint different from the first viewpoint; and correlation circuitry to: calculate a difference between the image feature in the first image frame and the corresponding image feature in the second image frame; generate difference data indicative of a difference between the first viewpoint for the first image frame and the second viewpoint for the second image frame in dependence upon the difference between the image feature in the first image frame and the corresponding image feature in the second image frame; and generate output data in dependence upon a difference between the difference data and the tracking data used to generate the first and second image frames, the output data being indicative of discrepancy for the difference between the first viewpoint and the second viewpoint and a difference between at least one of the tracked position and the tracked orientation of the HMD indicated by the tracking data used to generate the first and second image frames. ### Claim 2 (depends on 1) The data processing apparatus according to claim 1, wherein the detection circuitry is configured to detect a position and orientation of the image feature in the first image frame and to detect a position and orientation of the corresponding image feature in the second image frame. ### Claim 3 (depends on 1) The data processing apparatus according to claim 1, wherein the correlation circuitry is configured to calculate a transform between the first viewpoint for the first image frame and the second viewpoint for the second image frame in dependence upon the difference between the image feature and the corresponding image feature. ### Claim 4 (depends on 1) The data processing apparatus according to claim 1, wherein the sequence of images frames comprises a sequence of stereoscopic images each including a left image and a right image. ### Claim 5 (depends on 4) The data processing apparatus according to claim 4, wherein the correlation circuitry is configured to calculate a difference between the image feature in either the left image or the right image of the first image frame and the corresponding image feature in either the left image or the right image of the second image frame, respectively. ### Claim 6 (depends on 4) The data processing apparatus according to claim 4, wherein the detection circuitry is configured to detect the image feature in the first image frame by detecting the image feature in both the left image and the right image of the first image frame. ### Claim 7 (depends on 6) The data processing apparatus according to claim 6, wherein the correlation circuitry is configured to calculate a first vergence distance for the image feature in dependence upon the image feature in both the left image and the right image of the first image frame. ### Claim 8 (depends on 7) The data processing apparatus according to claim 7, wherein the detection circuitry is configured to detect the image feature in the second image frame by detecting the image feature in both the left image and the right image of the second image frame, and wherein the correlation circuitry is configured to: calculate a second vergence distance for the image feature in dependence upon the image feature in both the left image and the right image of the second image frame; generate vergence distance difference data indicative of a difference between the first vergence distance and the second vergence distance; and generate the output data in dependence upon a difference between the vergence distance difference data and the tracking data associated with the first and second image frames. ### Claim 9 (depends on 1) The data processing apparatus according to claim 1, comprising testing circuitry configured to generate test tracking data, wherein the image processing circuitry is configured to generate at least one image frame for display in dependence upon the test tracking data. ### Claim 10 (depends on 9) The data processing apparatus according to claim 9, wherein the testing circuitry is configured to predict a pose for the HMD in dependence upon the tracking data and to generate the test tracking data comprising at least one predicted pose for the HMD. ### Claim 11 (depends on 1) The data processing apparatus according to claim 1, wherein the correlation circuitry is configured to calculate a latency for the image frame generation processing by the image processing circuitry. ### Claim 12 (depends on 1) The data processing apparatus according to claim 1, wherein the image processing circuitry is configured to generate each image frame for display by applying an image distortion in dependence upon at least one optical element of the HMD. ### Claim 13 (depends on 1) The data processing apparatus according to claim 1, wherein the tracking data is generated in dependence upon captured images and/or one or more inertial sensors of the HMD. ### Claim 14 A data processing method comprising: receiving tracking data indicative of at least one of a tracked position and a tracked orientation for a head-mountable display (HMD); generating, by image processing circuitry, a sequence of image frames for display in dependence upon the tracking data for the HMD, in which the step of generating comprises generating the sequence of image frames with viewpoints that change according to changes in at least one of the tracked position and the tracked orientation of the HMD indicated by the tracking data for the HMD; detecting image features in the sequence of image frames generated by the image processing circuitry by detecting at least an image feature included in a first image frame generated for display at a first time with a first viewpoint and detecting a corresponding image feature included in a second image frame generated for display at a second time with a second viewpoint different from the first viewpoint; calculating a difference between the image feature in the first image frame and the corresponding image feature in the second image frame; generating difference data indicative of a difference between the first viewpoint for the first image frame and the second viewpoint for the second image frame in dependence upon the difference between the image feature in the first image frame and the corresponding image feature in the second image frame; and generating output data in dependence upon a difference between the difference data and the tracking data used to generate the first and second image frames, the output data being indicative of discrepancy for the difference between the first viewpoint and the second viewpoint and a difference between at least one of the tracked position and the tracked orientation of the HMD indicated by the tracking data used to generate the first and second image frames. ### Claim 15 Computer software which, when executed by a computer, causes the computer to carry out a method comprising: receiving tracking data indicative of at least one of a tracked position and a tracked orientation for a head-mountable display (HMD); generating, by image processing circuitry, a sequence of imageimages frames for display in dependence upon the tracking data for the HMD, in which the step of generating comprises generating the sequence of image frames with viewpoints that change according to changes in at least one of the tracked position and the tracked orientation of the HMD indicated by the tracking data for the HMD; detecting image features in the sequence of image frames generated by the image processing circuitry by detecting at least an image feature included in a first image frame generated for display at a first time with a first viewpoint and detecting a corresponding image feature included in a second image frame generated for display at a second time with a second viewpoint different from the first viewpoint; calculating a difference between the image feature in the first image frame and the corresponding image feature in the second image frame; generating difference data indicative of a difference between the first viewpoint for the first image frame and the second viewpoint for the second image frame in dependence upon the difference between the image feature in the first image frame and the corresponding image feature in the second image frame; and generating output data in dependence upon a difference between the difference data and the tracking data used to generate the first and second image frames, the output data being indicative of discrepancy for the difference between the first viewpoint and the second viewpoint and a difference between at least one of the tracked position and the tracked orientation of the HMD indicated by the tracking data used to generate the first and second image frames.