# US 9,519,144 B2 · System, method, and computer program product to produce images for a near-eye light field display having a defect > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,144 B2 - Document ID: US9519144B2 - Status: Granted - Kind: B2 - Application number: 14/323,965 - Filed: 2014-07-03 - Published: 2014-11-20 - Granted: 2016-12-13 - Earliest claimed: 2013-05-17 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Douglas Robert Lanman; David Patrick Luebke - Web page: https://patentgrep.com/patent/US9519144B2 ## CPC - G02B 27/017: {Head mounted} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 15/04: Texture mapping - G09G 3/00: Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes - H04N 13/344: with head-mounted left-right displays - H04N 13/398: Synchronisation thereof; Control thereof - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system, method, and computer program product are provided for producing images for a near-eye light field display. Defect information for a first pixel of a microdisplay of a near-eye light field display device is received and a second pixel of the microdisplay is identified, where the first pixel and the second pixel contribute to a portion of the retinal image. Based on the defect information, a value of the second pixel within an array of elemental images is modified to produce a corrected array of elemental images for display by the microdisplay. An optical apparatus of the near-eye light field display device may, for example, be a microlens of a microlens array positioned between a viewer and an emissive microdisplay or a pinlight of a pinlight array positioned behind a transmissive microdisplay relative to the viewer. ## Claims ### Claim 1 A method, comprising: receiving defect information for a first pixel of a microdisplay of a near-eye light field display device, the first pixel contributing to a portion of a retinal image; receiving a projection matrix that encodes a contribution of each pixel of the microdisplay to the portion of the retinal image; identifying, based on the projection matrix, a second pixel of the microdisplay that also contributes to the portion of the retinal image; and modifying, based on the defect information and the projection matrix, a value of the second pixel within an array of elemental images to produce a corrected array of elemental images for display by the microdisplay. ### Claim 2 (depends on 1) The method of claim 1, wherein the defect information indicates that the first pixel is a hot pixel comprising at least one continuously illuminated subpixel. ### Claim 3 (depends on 1) The method of claim 1, wherein the defect information indicates that the first pixel is a dead pixel comprising at least one continuously dark subpixel. ### Claim 4 (depends on 1) The method of claim 1, wherein the retinal image appears to a user of the near-eye light field display device at a two-dimensional virtual display plane beyond a near focus plane. ### Claim 5 (depends on 4) The method of claim 4, wherein a depth of the two-dimensional virtual display plane relative to the user's eye is based on an eyeglass prescription. ### Claim 6 (depends on 1) The method of claim 1, further comprising storing the modified value of the second pixel in a data structure for the corrected array of elemental images in a memory. ### Claim 7 (depends on 1) The method of claim 1, wherein the corrected array of elemental images corresponds to a left-eye image or a right-eye image of a stereoscopic image pair. ### Claim 8 (depends on 1) The method of claim 1, wherein multiple processing threads are configured to execute a fragment shader program to compute modified values for the second pixel and additional pixels of the corrected array of elemental images in parallel. ### Claim 9 (depends on 1) The method of claim 1, wherein modifying the value of the second pixel comprises minimizing an error between an ideal retinal image corresponding to the array of elemental images and the retinal image. ### Claim 10 (depends on 1) The method of claim 1, wherein each elemental image in the array of elemental images is a view of the retinal image corresponding to one optical apparatus within an array of optical apparatuses included in the near-eye light field display device. ### Claim 11 (depends on 10) The method of claim 10, wherein the array of optical apparatus is an array of pinlights. ### Claim 12 (depends on 10) The method of claim 10, wherein the array of optical apparatus is an array of microlenses. ### Claim 13 (depends on 1) The method of claim 1, wherein the projection matrix is based on an eyebox position. ### Claim 14 (depends on 1) The method of claim 1, wherein the projection matrix is based on eye-tracking information. ### Claim 15 (depends on 1) The method of claim 1, wherein the projection matrix accounts for a subpixel structure of the microdisplay. ### Claim 16 (depends on 1) The method of claim 1, wherein subpixels within the first pixel are interleaved with subpixels of the second pixel within a portion of the retinal image. ### Claim 17 A system comprising: a microdisplay of a near-eye light field display device, the microdisplay positioned between a near focus plane and a retina; and a processor configured to: receive defect information for a first pixel of the microdisplay, the first pixel contributing to a portion of a retinal image; receive a projection matrix that encodes a contribution of each pixel of the microdisplay to the portion of the retinal image; identify, based on the projection matrix, a second pixel of the microdisplay that also contributes to the portion of the retinal image; and adapt, based on the defect information and the projection matrix, a value of the second pixel within an array of elemental images to produce a corrected array of elemental images for display by the microdisplay. ### Claim 18 (depends on 17) The system of claim 17, further comprising an optical apparatus positioned between the microdisplay and the retina. ### Claim 19 (depends on 17) The system of claim 17, wherein the processor is further configured to process multiple processing threads that each execute a fragment shader program to compute values for the second pixel and additional pixels of the corrected array of elemental images in parallel. ### Claim 20 A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, causes the processor to perform steps comprising: receiving defect information for a first pixel of a microdisplay of a near-eye light field display device, the first pixel contributing to a portion of a retinal image; receiving a projection matrix that encodes a contribution of each pixel of the microdisplay to the portion of the retinal image; identifying, based on the projection matrix, a second pixel of the microdisplay that also contributes to the portion of the retinal image; and modifying, based on the defect information and the projection matrix, a value of the second pixel within an array of elemental images to produce a corrected array of elemental images for display by the microdisplay.