# US 10,769,752 B2 · Mixed reality system with virtual content warping and method of generating virtual content using same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,752 B2 - Document ID: US10769752B2 - Status: Granted - Kind: B2 - Application number: 15/924,011 - Filed: 2018-03-16 - Published: 2018-09-20 - Granted: 2020-09-08 - Earliest claimed: 2017-03-17 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Reza Nourai; Robert Blake Taylor - Web page: https://patentgrep.com/patent/US10769752B2 ## CPC - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 15/205: {Image-based rendering} - G06T 15/40: Hidden part removal - G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 2210/44: Morphing - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A computer implemented method for warping virtual content includes generating warped virtual content by transforming source virtual content. The method also includes determining whether a memory location corresponding to an X, Y location of the warped virtual content in an output frame of reference is occupied by pre-existing virtual content. The method further includes storing the warped virtual content in the memory location if the memory location is not occupied. Moreover, the method includes comparing respective Z locations of the warped virtual content and the pre-existing virtual content to identify virtual content with a Z location closer to a viewing location if the memory location is occupied. The method also includes storing the warped virtual content in the memory location corresponding to the X, Y location if a Z location of warped virtual content is closer to the viewing location than a pre-existing Z location of pre-existing virtual content. ## Claims ### Claim 1 A computer implemented method for warping virtual content: generating warped virtual content by transforming source virtual content; determining whether a memory location corresponding to an X, Y location of the warped virtual content in an output frame of reference is occupied by pre-existing virtual content; storing the warped virtual content in the memory location if the memory location is not occupied by the pre-existing virtual content; comparing respective Z locations of the warped virtual content and the pre-existing virtual content to identify virtual content with a Z location closer to a viewing location in the output frame of reference if the memory location is occupied by the pre-existing virtual content; and storing the warped virtual content in the memory location corresponding to the X, Y location if a Z location of the warped virtual content is closer to the viewing location than a pre-existing Z location of the pre-existing virtual content in the output frame of reference, wherein generating the warped virtual content, determining whether the memory location is occupied, comparing the respective Z locations of the warped virtual content and the pre-existing virtual content if the memory location is occupied, and storing the warped virtual content in the memory location all occur in a single pass. ### Claim 2 (depends on 1) The method of claim 1, further comprising discarding the warped virtual content if the pre-existing Z location of the pre-existing virtual content is closer to the viewing location than the Z location of the warped virtual content in the output frame of reference. ### Claim 3 (depends on 1) The method of claim 1, wherein transforming the source virtual content comprises: generate a pixel map by mapping pixels of an image; dividing the pixel map into a plurality of primitives, and performing a transformation on one of the plurality of primitives into the output frame of reference, wherein the source virtual content is the one of the plurality of primitives. ### Claim 4 (depends on 3) The method of claim 3, wherein each of the plurality of primitives is a quadrilateral. ### Claim 5 (depends on 3) The method of claim 3, wherein each of the plurality of primitives is a triangle. ### Claim 6 (depends on 3) The method of claim 3, wherein each of the plurality of primitives is a pixel. ### Claim 7 (depends on 3) The method of claim 3, wherein the transformation is a Fourier transformation. ### Claim 8 (depends on 1) The method of claim 1, further comprising generating the source virtual content using a source frame of reference different from the output frame of reference. ### Claim 9 (depends on 8) The method of claim 8, wherein the source virtual content is transformed from the source frame of reference to the output frame of reference. ### Claim 10 (depends on 9) The method of claim 9, wherein the respective Z locations of the warped virtual content and the pre-existing virtual content in the output frame of reference are different from corresponding Z locations of the warped virtual content and the pre-existing virtual content in the source frame of reference. ### Claim 11 (depends on 1) The method of claim 1, wherein the source virtual content comprises image information and a source X, Y location in a source frame of reference. ### Claim 12 (depends on 11) The method of claim 11, wherein the image information comprises a brightness. ### Claim 13 (depends on 11) The method of claim 11, wherein the image information comprises a color. ### Claim 14 (depends on 11) The method of claim 11, wherein the image information comprises a Z location in the source frame of reference. ### Claim 15 (depends on 11) The system of claim 11, wherein the warped virtual content comprises an output X, Y location in the output frame of reference. ### Claim 16 A system for warping virtual content, comprising: an input memory to store source virtual content; a command processor to divide the source virtual content into a plurality of scheduling units; a scheduler to determine a processing sequence; a graphics processing unit (GPU) core, the GPU core having a plurality of shader cores to process one of the plurality of scheduling units to generate warped virtual content; a first in, first out (FIFO) memory to store output from the GPU core; a raster operations processor (ROP) to rasterize the warped virtual content for display; and a buffer memory to temporarily store the warped virtual content, wherein when pre-existing virtual content temporarily stored in the buffer memory conflicts with the warped virtual content, the ROP compares respective Z locations of the pre-existing virtual content and the warped virtual content and writes the warped virtual content in the buffer memory when a Z location of the warped virtual content is closer to a viewing location than a pre-existing Z location of the pre-existing virtual content, and wherein the following occur in a single pass: the GPU generates the warped virtual content, the ROP determines whether the pre-existing virtual content conflicts with the warped virtual content, the ROP compares the respective Z locations of the pre-existing virtual content and the warped virtual content if the pre-existing virtual content conflicts with the warped virtual content, and the ROP writes the warped virtual content in the buffer memory when the Z location of the warped virtual content is closer to the viewing location than the pre-existing Z location of the pre-existing virtual content. ### Claim 17 (depends on 16) The system of claim 16, wherein the source virtual content comprises a source X, Y value in a source frame of reference. ### Claim 18 (depends on 17) The system of claim 17, wherein the warped virtual content comprises an output X, Y value in an output frame of reference. ### Claim 19 A computer program product embodied in a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for warping virtual content, the method comprising: generating warped virtual content by transforming source virtual content; determining whether a memory location corresponding to an X, Y location of the warped virtual content in an output frame of reference is occupied by pre-existing virtual content; storing the warped virtual content in the memory location if the memory location is not occupied by the pre-existing virtual content; comparing respective Z locations of the warped virtual content and the pre-existing virtual content to identify virtual content with a Z location closer to a viewing location in the output frame of reference if the memory location is occupied by the pre-existing virtual content; and storing the warped virtual content in the memory location corresponding to the X, Y location if a Z location of the warped virtual content is closer to the viewing location than a pre-existing Z location of the pre-existing virtual content in the output frame of reference, wherein generating the warped virtual content, determining whether the memory location is occupied, comparing the respective Z locations of the warped virtual content and the pre-existing virtual content if the memory location is occupied, and storing the warped virtual content in the memory location all occur in a single pass.