# US 10,482,575 B2 · Super-resolution apparatus and method for virtual and mixed reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,575 B2 - Document ID: US10482575B2 - Status: Granted - Kind: B2 - Application number: 15/719,458 - Filed: 2017-09-28 - Published: 2019-03-28 - Granted: 2019-11-19 - Earliest claimed: 2017-09-28 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Zhengmin Li; Atsuo Kuwahara; Deepak Vembar - Web page: https://patentgrep.com/patent/US10482575B2 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/063: using electronic means - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06N 20/00: Machine learning - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An apparatus and method for efficiently improving virtual/real interactions in augmented reality. For example, one embodiment of a method comprises: capturing a raw image including depth data; identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects; generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof; detecting interactions between the virtual objects and the real objects using the super-resolution map; and performing one or more graphics processing or general purpose processing operations based on the detected interactions. ## Claims ### Claim 1 A method comprising: capturing a raw image including depth data; identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances; generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof; detecting interactions between the virtual objects and the real objects using the super-resolution map; and performing one or more graphics processing or general-purpose processing operations based on the detected interactions. ### Claim 2 (depends on 1) The method of claim 1 wherein the machine-learning techniques comprise: performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and evaluating results of the comparisons. ### Claim 3 (depends on 2) The method of claim 2 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics. ### Claim 4 (depends on 3) The method of claim 3 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map. ### Claim 5 (depends on 4) The method of claim 4 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model. ### Claim 6 (depends on 1) The method of claim 1 wherein identifying the one or more regions of interest further comprises: if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest. ### Claim 7 (depends on 1) The method of claim 1 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount. ### Claim 8 (depends on 1) The method of claim 1 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions. ### Claim 9 An apparatus comprising: a camera to capture a raw image including depth data; region detection circuitry to identify one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances; depth super-resolution circuitry to generate a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof; interaction detection circuitry to detect interactions between the virtual objects and the real objects using the super-resolution map; and wherein one or more graphics processing or general-purpose processing operations are to be performed based on the detected interactions. ### Claim 10 (depends on 9) The apparatus of claim 9 wherein the machine-learning techniques comprise: performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and evaluating results of the comparisons. ### Claim 11 (depends on 10) The apparatus of claim 10 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics. ### Claim 12 (depends on 11) The apparatus of claim 11 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map. ### Claim 13 (depends on 12) The apparatus of claim 12 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model. ### Claim 14 (depends on 9) The apparatus of claim 9 wherein identifying the one or more regions of interest further comprises: if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest. ### Claim 15 (depends on 9) The apparatus of claim 9 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount. ### Claim 16 (depends on 9) The apparatus of claim 9 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions. ### Claim 17 A non-transitory machine-readable medium having program code stored therein which, when executed by a machine, causes the machine to perform the operations of: capturing a raw image including depth data; identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances; generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof; detecting interactions between the virtual objects and the real objects using the super-resolution map; and performing one or more graphics processing or general-purpose processing operations based on the detected interactions. ### Claim 18 (depends on 1) The non-transitory machine-readable medium of claim 1 wherein the machine-learning techniques comprise: performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and evaluating results of the comparisons. ### Claim 19 (depends on 18) The non-transitory machine-readable medium of claim 18 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics. ### Claim 20 (depends on 19) The non-transitory machine-readable medium of claim 19 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map. ### Claim 21 (depends on 20) The non-transitory machine-readable medium of claim 20 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model. ### Claim 22 (depends on 17) The non-transitory machine-readable medium of claim 17 wherein identifying the one or more regions of interest further comprises: if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest. ### Claim 23 (depends on 17) The non-transitory machine-readable medium of claim 17 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount. ### Claim 24 (depends on 17) The non-transitory machine-readable medium of claim 17 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions.