# US 10,504,291 B2 · Optimizing head mounted displays for augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,291 B2 - Document ID: US10504291B2 - Status: Granted - Kind: B2 - Application number: 16/051,099 - Filed: 2018-07-31 - Published: 2019-04-25 - Granted: 2019-12-10 - Earliest claimed: 2016-01-15 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Anna Petrovskaya; Peter Varvak - Web page: https://patentgrep.com/patent/US10504291B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 3/005: {Input arrangements through a video camera} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0308: {comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 15/04: Texture mapping - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G02B 2027/014: {comprising information/image processing systems} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract While many augmented reality systems provide “see-through” transparent or translucent displays upon which to project virtual objects, many virtual reality systems instead employ opaque, enclosed screens. Indeed, eliminating the user's perception of the real-world may be integral to some successful virtual reality experiences. Thus, head mounted displays designed exclusively for virtual reality experiences may not be easily repurposed to capture significant portions of the augmented reality market. Various of the disclosed embodiments facilitate the repurposing of a virtual reality device for augmented reality use. Particularly, by anticipating user head motion, embodiments may facilitate scene renderings better aligned with user expectations than naïve renderings generated within the enclosed field of view. In some embodiments, the system may use procedural mapping methods to generate a virtual model of the environment. The system may then use this model to supplement the anticipatory rendering. ## Claims ### Claim 1 A system comprising: a movable display; memory including instructions; at least one processor to execute the instructions to, at least: retrieve first image data during a first time, the first image data corresponding to a first camera position; transform a frame of the first image data into a transformed frame; impose a border constraint on a first portion of the transformed frame, the border constraint outside a field of view of the movable display, the border constraint to cause a second portion of the transformed frame to be within the field of view of the movable display during the first time; retrieve inertial measurement unit (IMU) data during a second time, the IMU data corresponding to a second camera position to retrieve second image data; and cause a subset of the first portion of the transformed frame to be within the field of view of the movable display during the second time to reduce processing latency effects associated with the second image data. ### Claim 2 (depends on 1) The system as defined in claim 1, wherein the at least one processor is to retrieve the IMU data at a rate faster than a camera capture rate of the second image data. ### Claim 3 (depends on 2) The system as defined in claim 2, wherein the at least one processor is to determine a predicted pose during the second time, the predicted pose indicative of user head orientation. ### Claim 4 (depends on 2) The system as defined in claim 2, wherein the at least one processor is to apply at least one of positional or rotational velocity data to determine a predicted pose during the second time. ### Claim 5 (depends on 4) The system as defined in claim 4, wherein the at least one processor is to apply a Kalman filter to determine the predicted pose. ### Claim 6 (depends on 1) The system as defined in claim 1, wherein the at least one processor is to reduce user nausea by rendering the subset of the first portion of the transformed frame from the border constraint to the field of view of the movable display. ### Claim 7 (depends on 1) The system as defined in claim 1, wherein the at least one processor is to render the second image data during a third time as a mesh in response to detecting the IMU data during the second time. ### Claim 8 A computer readable storage device or storage disk comprising instructions that, when executed, cause the at least one processor to at least: retrieve first image data during a first time, the first image data corresponding to a first camera position; transform a frame of the first image data into a transformed frame; impose a border constraint on a first portion of the transformed frame, the border constraint outside a field of view of the movable display, the border constraint to cause a second portion of the transformed frame to be within the field of view of the movable display during the first time; retrieve inertial measurement unit (IMU) data during a second time, the IMU data corresponding to a second camera position to retrieve second image data; and cause a subset of the first portion of the transformed frame to be within the field of view of the movable display during the second time to reduce processing latency effects associated with the second image data. ### Claim 9 (depends on 8) The storage device or storage disk as defined in claim 8, wherein the instructions, when executed, cause the at least one processor to retrieve the IMU data at a rate faster than a camera capture rate of the second image data. ### Claim 10 (depends on 9) The storage device or storage disk as defined in claim 9, wherein the instructions, when executed, cause the at least one processor to determine a predicted pose during the second time, the predicted pose indicative of user head orientation. ### Claim 11 (depends on 9) The storage device or storage disk as defined in claim 9, wherein the instructions, when executed, cause the at least one processor to apply at least one of positional or rotational velocity data to determine a predicted pose during the second time. ### Claim 12 (depends on 11) The storage device or storage disk as defined in claim 11, wherein the instructions, when executed, cause the at least one processor to apply a Kalman filter to determine the predicted pose. ### Claim 13 (depends on 8) The storage device or storage disk as defined in claim 8, wherein the instructions, when executed, cause the at least one processor to reduce user nausea by rendering the subset of the first portion of the transformed frame from the border constraint to the field of view of the movable display. ### Claim 14 (depends on 8) The storage device or storage disk as defined in claim 8, wherein the instructions, when executed, cause the at least one processor to render the second image data during a third time as a mesh in response to detecting the IMU data during the second time. ### Claim 15 A computer implemented method, comprising: retrieving first image data during a first time, the first image data corresponding to a first camera position; transforming a frame of the first image data into a transformed frame; imposing a border constraint on a first portion of the transformed frame, the border constraint outside a field of view of the movable display, the border constraint to cause a second portion of the transformed frame to be within the field of view of the movable display during the first time; retrieving inertial measurement unit (IMU) data during a second time, the IMU data corresponding to a second camera position to retrieve second image data; and causing a subset of the first portion of the transformed frame to be within the field of view of the movable display during the second time to reduce processing latency effects associated with the second image data. ### Claim 16 (depends on 15) The method as defined in claim 15, further including retrieving the IMU data at a rate faster than a camera capture rate of the second image data. ### Claim 17 (depends on 16) The method as defined in claim 16, further including determining a predicted pose during the second time, the predicted pose indicative of user head orientation. ### Claim 18 (depends on 16) The method as defined in claim 16, further including applying at least one of positional or rotational velocity data to determine a predicted pose during the second time. ### Claim 19 (depends on 18) The method as defined in claim 18, further including applying a Kalman filter to determine the predicted pose. ### Claim 20 (depends on 15) The method as defined in claim 15, further including reducing user nausea by rendering the subset of the first portion of the transformed frame from the border constraint to the field of view of the movable display. ### Claim 21 (depends on 15) The method as defined in claim 15, further including rendering the second image data during a third time as a mesh in response to detecting the IMU data during the second time.