# US 11,079,597 B2 · Low latency distortion unit for head mounted displays > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,079,597 B2 - Document ID: US11079597B2 - Status: Granted - Kind: B2 - Application number: 15/998,400 - Filed: 2018-08-15 - Published: 2019-02-21 - Granted: 2021-08-03 - Earliest claimed: 2017-08-15 - Assignees: Imagination Technologies Limited - Applicants: Imagination Technologies Limited - Inventors: Simon Fenney - Web page: https://patentgrep.com/patent/US11079597B2 ## CPC - G02B 27/0172: {characterised by optical features} - G02B 27/0025: {for optical correction, e.g. distorsion, aberration} - G02B 27/017: {Head mounted} - G06T 5/70: Denoising; Smoothing - G06T 5/80: Geometric correction - G06T 15/005: {General purpose rendering architectures} - G06T 15/04: Texture mapping - H04N 9/3173: {wherein the projection device is specially adapted for enhanced portability} - H04N 9/3182: {Colour adjustment, e.g. white balance, shading or gamut (white balance per se H04N9/73; control of amplitude of colour signals H04N9/68; colour control circuits for displays in general G09G5/02, G09G3/2003, G09G3/3607)} - H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)} - H04N 9/3194: {including sensor feedback} - G02B 2027/011: {comprising device for correcting geometrical aberrations, distortion} - G02B 2027/0116: {comprising devices for correcting chromatic aberration} - G02B 2027/014: {comprising information/image processing systems} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A graphics processing system for a head mounted display (or other non-standard projection display) comprises a low latency distortion unit which is separate from a graphics processing unit in the graphics processing system. The low latency distortion unit receives pixel data generated by the graphics processing system using a standard projection and performs a mapping operation to introduce distortion which is dependent upon the optical properties of the optical arrangement within the head mounted display. The distorted pixel data which is generated by the low latency distortion unit is then output to the display in the head mounted display. ## Claims ### Claim 1 A graphics processing system for a non-standard projection display comprising: a graphics processing unit (GPU) configured to generate pixel data rendered with one or more standard projections; and a hardware distortion unit (HDU) separate from the GPU, said HDU implemented in fixed function logic circuitry and configured to transform the pixel data generated by the GPU into distorted pixel data and output the distorted pixel data for display on a non-standard projection display; wherein the HDU comprises: a hardware logic mapping unit arranged to perform a mapping of the pixel data to introduce distortion, and a transformation unit configured to adjust the mapping performed by the mapping unit in response to input data, wherein the input data comprises updated head position and/or orientation data. ### Claim 2 (depends on 1) A graphics processing system according to claim 1, wherein the non-standard projection display is a head mounted display. ### Claim 3 (depends on 1) The graphics processing system according to claim 1, wherein the hardware distortion unit comprises: an input configured to receive pixel data generated by the graphics processing unit; wherein the mapping unit is arranged to sample the pixel data; and an output configured to output distorted pixel data generated, at least in part, by the mapping unit. ### Claim 4 (depends on 3) The graphics processing system according to claim 3, wherein the output is configured to transmit the distorted pixel data in raster order to the non-standard projection display. ### Claim 5 (depends on 1) The graphics processing system according to claim 1, wherein the hardware distortion unit further comprises one or more of: a filter unit comprising hardware logic arranged to perform filtering of pixel data output by the mapping unit and provide the distorted pixel data to the output; and one or more look-up tables arranged to store mapping data and wherein the hardware logic in the mapping unit is arranged to access the mapping data and to perform the mapping using the mapping data. ### Claim 6 (depends on 1) The graphics processing system according to claim 1, wherein the input data comprises a transformational matrix. ### Claim 7 (depends on 1) The graphics processing system according to claim 1, wherein the distortion introduced is dependent upon an optical arrangement in the non-standard projection display. ### Claim 8 (depends on 7) The graphics processing system according to claim 7, wherein the mapping of the pixel data introduces geometric distortion dependent upon the optical arrangement and also corrects for chromatic aberration of the optical arrangement. ### Claim 9 (depends on 1) The graphics processing system according to claim 1, wherein the mapping performed by the mapping unit is a texture mapping. ### Claim 10 (depends on 1) The graphics processing system according to claim 1, wherein the distortion of the distorted pixel data counteracts distortion introduced by the non-standard projection display when viewing images on the non-standard projection display. ### Claim 11 (depends on 1) The graphics processing system according to claim 1, wherein the graphics processing unit is separate from the hardware distortion unit. ### Claim 12 (depends on 1) The graphics processing system according to claim 1, further comprising a frame buffer memory wherein the graphics processing unit is arranged to output the pixel data to the frame buffer memory and the hardware distortion unit is arranged to read the pixel data from the frame buffer memory. ### Claim 13 (depends on 12) The graphics processing system according to claim 12, further comprising a display pipeline comprising a display feed, wherein the display pipeline is arranged to receive the distorted pixel data from the hardware distortion unit and output it to a display. ### Claim 14 (depends on 1) The graphics processing system according to claim 1, further comprising a frame buffer memory and an enhanced display feed comprising the hardware distortion unit, wherein the graphics processing unit is arranged to output the pixel data to the frame buffer memory and the enhanced display feed is arranged to read the pixel data from the frame buffer memory and output it to a display. ### Claim 15 (depends on 1) The graphics processing system according to claim 1, wherein the graphics processing unit is configured to generate the pixel data in a single render pass. ### Claim 16 A method comprising: generating, in a graphics processing unit (GPU) in a non-standard projection display, pixel data rendered with one or more standard projections; transforming, in a hardware distortion unit (HDU) separate from the GPU in the non-standard projection display and implemented in fixed function logic circuitry, the pixel data generated by the GPU into distorted pixel data; and outputting the distorted pixel data for display on the non-standard projection display; wherein transforming the pixel data generated by the GPU into distorted pixel data comprises: performing a mapping of the pixel data to introduce distortion and generate the distorted pixel data, wherein the mapping is adjusted in response to input data, wherein the input data comprises updated head position and/or orientation data. ### Claim 17 (depends on 16) The method according to claim 16, wherein transforming the pixel data generated by the GPU into distorted pixel data comprises one or more of: receiving pixel data generated by the graphics processing unit; sampling the pixel data; and filtering the distorted pixel data. ### Claim 18 (depends on 16) The method according to claim 16, further comprising: transmitting the distorted pixel data in raster order to the non-standard projection display. ### Claim 19 A non-transitory computer readable storage medium having stored thereon a computer readable description of a graphics processing system that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the graphics processing system, wherein the graphics processing system comprises: a graphics processing unit (GPU) configured to generate pixel data rendered with one or more standard projections; and a hardware distortion unit (HDU) separate from the GPU, said HDU implemented in fixed function logic circuitry and configured to transform the pixel data generated by the GPU into distorted pixel data and output the distorted pixel data for display on a non-standard projection display; wherein the HDU comprises: a hardware logic mapping unit arranged to perform a mapping of the pixel data to introduce distortion, and a transformation unit configured to adjust the mapping performed by the mapping unit in response to input data, wherein the input data comprises updated head position and/or orientation data.