# US 11,445,170 B2 · Eyeball camera system and methods for display system calibration > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,170 B2 - Document ID: US11445170B2 - Status: Granted - Kind: B2 - Application number: 16/926,570 - Filed: 2020-07-10 - Published: 2021-01-14 - Granted: 2022-09-13 - Earliest claimed: 2019-07-12 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Zhiheng Jia; Jeffrey Todd Daiker; Timothy Alan Ellis; Xiao Li; Jeremy A. Grata - Web page: https://patentgrep.com/patent/US11445170B2 ## CPC - G06T 7/85: {Stereo camera calibration} - H04N 17/004: {for digital television systems} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/01: Head-up displays - G02B 27/017: {Head mounted} - G06T 7/60: Analysis of geometric attributes - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 7/90: Determination of colour characteristics - H04N 17/002: {for television cameras} - G06T 2207/10012: Stereo images - G06T 2207/30204: Marker - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract Embodiments provide a camera design (e.g., an eyeball camera) that mimics a human eye in geometry, optical performance and/or motion. The eyeball camera adopts the same cornea and pupil geometry from the human eye, and has the iris and pupil configured with multiple texture, color or diameter options. The resolution of the eyeball camera is designed to match the acuity of typical 20/20 human vision, and focus is adjusted from 0 to 4 diopters. A pair of eyeball cameras are mounted independently on two hexapods to simulate the human eye gaze and vergence. The perceived virtual and real world are calibrated and evaluated based on eye conditions like pupil location and gaze using the eyeball cameras. The eyeball camera serves as a bridge to combine the data from spatial computing like eye tracking, 3D geometry of the digital world, display color accuracy/uniformity, and display optical quality (sharpness, contrast, etc.). ## Claims ### Claim 1 A system for applying a correction to a virtual image display system, the system comprising: a display device configured to display a virtual image; an image capture device positioned in front of the display device, wherein the image capture device has one or more properties similar to a human eye; and a processor coupled to the display device and the image capture device to receive image data from the image capture device, wherein the processor is programmed to: receive a first image of a scene; determine a first set of visual properties associated with the first image; receive a second image of the scene captured with the image capture device; determine a second set of visual properties associated with the second image, wherein the first set of visual properties and the second set of visual properties include at least one of a spatial positioning, a color or a geometry attribute; identify a discrepancy between at least one of the first set of visual properties of the first image and at least one of the second set of visual properties of the second image corresponding to he at least one of the first set of visual properties of the first image; determine a correction to apply to the virtual image display system to at least partially correct for the discrepancy using the one or more properties of the image capture device, wherein the correction matches the spatial positioning, the color or the geometry attribute associated with the first image to that of the second image; and apply the correction to the virtual image display system based on the one or more properties of the image capture device. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more properties of the image capture device include one or more of a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, or a gaze orientation. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more properties of the image capture device are configurable. ### Claim 4 (depends on 1) The system of claim 1, wherein the display device comprises a first display associated with a first eye of a user and a second display associated with a second eye of the user, and wherein the image capture device includes a first image capture element positioned in front of the first display and a second image capture element positioned in front of the second display. ### Claim 5 (depends on 4) The system of claim 4, wherein the first image capture element and the second image capture element are configured to move independently from one another. ### Claim 6 (depends on 4) The system of claim 4, wherein one or more properties of the first image capture element are configurable independently from one or more properties of the second image capture element. ### Claim 7 (depends on 1) The system of claim 1, further comprising: a real-world image capture device coupled to the display device and configured to capture the first image, wherein the correction is applied to the virtual image display system to calibrate the real-world image capture device. ### Claim 8 (depends on 1) The system of claim 1, wherein the first set of visual properties and the second set of visual properties include the spatial positioning, wherein the processor is further programmed to: determine a spatial positioning of a point in the first image; identify the discrepancy between the spatial positioning of the point in the first image and a spatial positioning of a corresponding point in the second image; determine the correction to apply to the virtual image display system to align the point in the first image with the corresponding point in the second image; and apply the correction to the virtual image display system. ### Claim 9 (depends on 1) The system of claim 1, wherein the first set of visual properties and the second set of visual properties include the color, wherein the processor is further programmed to: determine a color of an area in the first image; identify the discrepancy between the color of the area in the first image and a color of a corresponding area in the second image; determine the correction to apply to the virtual image display system to match the color of the area in the first image with the color of the corresponding area in the second image; and apply the correction to the virtual image display system. ### Claim 10 (depends on 9) The system of claim 9, wherein the correction includes one or more of a correction in hue, saturation, or contrast. ### Claim 11 (depends on 1) The system of claim 1, wherein the first set of visual properties and the second set of visual properties include the geometry attribute, wherein the processor is further programmed to: determine the geometry attribute of an object in the first image; identify the discrepancy between the geometry attribute of the object in the first image and a geometry attribute of a corresponding object in the second image; determine the correction to apply to the virtual image display system to match the geometry attribute of the object in the first image with the geometry attribute of the corresponding object in the second image; and apply the correction to the virtual image display system. ### Claim 12 (depends on 11) The system of claim 11, wherein the correction includes one or more of a correction in collinearity, curvature, length, width, breadth, added marks, missing marks, numerosity, shape, size, spatial grouping, or spatial orientation of the object in the second image. ### Claim 13 A method for applying a correction to a virtual image display system, the method comprising: receiving, by a processor, a first image of a scene from a real-world image capture device; determining, by the processor, a first set of visual properties associated with the first image; receiving, by the processor from an image capture device, a second image of the scene captured with the image capture device having one or more properties similar to a human eye, wherein the image capture device is positioned in front of a display device; determining, by the processor, a second set of visual properties associated with the second image, wherein the first set of visual properties and the second set of visual properties include at least one of a spatial positioning, a color or a geometry attribute; identifying, by the processor, a discrepancy between at least one of the first set of visual properties of the first image and at let one of the second set of visual properties of the second image corresponding to the at least one of first set of visual properties of the first image; determining, by the processor, a correction to apply to the virtual image display system to at least partially correct for the discrepancy using the one or more properties of the image capture device, wherein the correction matches the spatial positioning, the color or the geometry attribute associated with the first image to that of the second image; and applying, by the processor, the correction to the virtual image display system based on the one or more properties of the image capture device. ### Claim 14 (depends on 13) The method of claim 13, wherein the one or more properties of the image capture device include one or more of a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, or a gaze orientation. ### Claim 15 (depends on 13) The method of claim 13, wherein the one or more properties of the image capture device are configurable. ### Claim 16 (depends on 13) The method of claim 13, wherein the display device comprises a first display associated with a first eye of a user and a second display associated with a second eye of the user, and wherein the image capture device includes a first image capture element positioned in front of the first display and a second image capture element positioned in front of the second display. ### Claim 17 (depends on 16) The method of claim 16, wherein the first image capture element and the second image capture element are configured to move independently from one another. ### Claim 18 (depends on 16) The method of claim 16, wherein one or more properties of the first image capture element are configurable independently from one or more properties of the second image capture element. ### Claim 19 (depends on 13) The method of claim 13, wherein the first image is received from a real-world image capture device coupled to the display device, wherein applying the correction further comprises: calibrating the real-world image capture device. ### Claim 20 (depends on 13) The method of claim 13, wherein the first set of visual properties and the second set of visual properties include the spatial positioning, and the method further comprises: determining, by the processor, a spatial positioning of a point in the first image; identifying, by the processor, the discrepancy between the spatial positioning of the point in the first image and a spatial positioning of a corresponding point in the second image; determining, by the processor, the correction to apply to the virtual image display system to align the point in the first image with the corresponding point in the second image; and applying, by the processor, the correction to the virtual image display system. ### Claim 21 (depends on 13) The method of claim 13, wherein the first set of visual properties and the second set of visual properties include the color, and the method further comprises: determining, by the processor, a color of an area in the first image; identifying, by the processor, the discrepancy between the color of the area in the first image and a color of a corresponding area in the second image; determining, by the processor, the correction to apply to the virtual image display system to match the color of the area in the first image with the color of the corresponding area in the second image; and applying, by the processor, the correction to the virtual image display system. ### Claim 22 (depends on 21) The method of claim 21, wherein the correction includes one or more of a correction in hue, saturation, or contrast. ### Claim 23 (depends on 13) The method of claim 13, wherein the first set of visual properties and the second set of visual properties include the geometry attribute, and the method further comprises: determining, by the processor, the geometry attribute of an object in the first image; identifying, by the processor, the discrepancy between the geometry attribute of the object in the first image and a geometry attribute of a corresponding object in the second image; determining, by the processor, the correction to apply to the virtual image display system to match the geometry attribute of the object in the first image with the geometry attribute of the corresponding object in the second image; and applying, by the processor, the correction to the virtual image display system. ### Claim 24 (depends on 23) The method of claim 23, wherein the correction includes one or more of a correction in collinearity, curvature, length, width, breadth, added marks, missing marks, numerosity, shape, size, spatial grouping, or spatial orientation of the object in the second image.