# US 2021/0279972 A1 · Adjusted overlaid images in scenes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279972 A1 - Document ID: US20210279972A1 - Status: Published application - Kind: A1 - Application number: 17/328,998 - Filed: 2021-05-24 - Published: 2021-09-09 - Earliest claimed: 2019-06-27 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Barry David Silverstein; Romain Bachy; Edward Buckley - Web page: https://patentgrep.com/patent/US20210279972A1 ## CPC - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G09G 5/026: {Control of mixing and/or overlay of colours in general (G09G5/022 and G09G5/024 take precedence)} - G02B 27/017: {Head mounted} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 2027/0112: {comprising device for genereting colour display} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G06T 2207/10024: Color image - G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)} - G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence) - G09G 2320/0242: Compensation of deficiencies in the appearance of colours - G09G 2320/0693: Calibration of display systems - G09G 2340/06: Colour space transformation - G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels - G09G 2360/144: the light being ambient light - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract Adjusted overlaid images are generated in response to at least one color reference object or color reference point. A display is driven to present the adjusted overlaid image. ## Claims ### Claim 1 A device comprising: a display configured to generate overlaid images for overlaying with real-world scenes; a light sensor configured to capture a light measurement of scene light from an external environment of the device; and processing logic configured to: determine a color space adjustment value in response to the light measurement; generate adjusted overlaid images by adjusting an image color of at least a portion of the overlaid images in response to the color space adjustment value; and drive the display to present the adjusted overlaid images overlaid with the real-world scenes. ### Claim 2 (depends on 1) The device of claim 1 further comprising: a color reference object configured to direct the scene light to the light sensor, wherein the light sensor is configured to measure a spectral reflectance of the color reference object. ### Claim 3 (depends on 2) The device of claim 2, wherein determining the color space adjustment value in response to the light measurement includes comparing a calibrated color value of the color reference object to the light measurement of the color reference object when the color reference object of the device is illuminated with the scene light. ### Claim 4 (depends on 2) The device of claim 2, wherein the color reference object is encapsulated in a transparent protection barrier so that the color reference object is protected from the outside environment while still receiving the scene light. ### Claim 5 (depends on 1) The device of claim 1, wherein the light sensor includes an image sensor oriented to capture a scene image as the light measurement. ### Claim 6 (depends on 5) The device of claim 5, wherein the determining the color space adjustment value in response to the light measurement includes: identifying a color reference point in the scene image; and determining a difference vector between pixel values of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value, wherein the image color of the at least a portion of the overlaid image is adjusted based on the difference vector. ### Claim 7 (depends on 6) The device of claim 6, wherein identifying the color reference point includes identifying a whitest object in the scene image, and wherein the calibrated color value is a white color reference point. ### Claim 8 (depends on 1) The device of claim 1, wherein the processing logic is configured to receive the light measurement from the light sensor. ### Claim 9 (depends on 1) The device of claim 1, wherein the device is a head mounted display (HMD). ### Claim 10 A computer-implemented method comprising: capturing a light measurement with a light sensor of a head mounted display (HMD) configured to receive scene light from an external environment of the HMD; determining a color space adjustment value in response to the light measurement; generating an adjusted overlaid image by adjusting an image color of at least a portion of an overlaid image in response to the color space adjustment value; and driving a display of the HMD to present the adjusted overlaid image overlaid with real-world scenes viewable to a user of the HMD. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein the HMD includes a color reference object configured to direct the scene light to the light sensor, and wherein determining the color space adjustment value in response to the light measurement includes comparing a calibrated color value of the color reference object to the light measurement of the color reference object when the color reference object of the HMD is illuminated with the scene light. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein the light sensor includes an image sensor and the image sensor is oriented to capture a scene image that includes a color reference object included in the HMD, the light measurement including the scene image, and wherein determining the color space adjustment value in response to the light measurement includes: identifying a color reference point in the scene image; and determining a difference vector between pixel values of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value, wherein the image color of the at least a portion of the overlaid image is adjusted based on the difference vector. ### Claim 13 (depends on 10) The computer-implemented method of claim 10, wherein adjusting the image color includes adjusting sub-pixel values of the overlaid image based on the color space adjustment value. ### Claim 14 (depends on 10) The computer-implemented method of claim 10, wherein the display includes a first, second, and third light source, and wherein adjusting the image color includes adjusting an intensity of at least one of the first, second, or third light source based on the color space adjustment value. ### Claim 15 A computer-implemented method comprising: receiving a scene image of a real-world scene; identifying at least one color reference point in the scene image of the real-world scene; generating a color space adjustment map for an overlaid image utilizing the at least one color reference point; applying the color space adjustment map to the overlaid image to generate an adjusted overlaid image by adjusting an image color of at least a portion of the overlaid image; and driving a display to present the adjusted overlaid image to be overlaid with real-world scenes viewable by a user. ### Claim 16 (depends on 15) The computer-implemented method of claim 15 further comprising: identifying an overlay zone of the real-world scene that the overlaid image will occupy; and selecting a closest color reference point from the at least one color reference point that is closest to the overlay zone, wherein the closest color reference point is utilized in generating the color space adjustment map. ### Claim 17 (depends on 15) The computer-implemented method of claim 15, wherein identifying the at least one color reference point includes identifying a first color reference point and a second color reference point, and wherein generating the color space adjustment map for the overlaid image includes: determining a first difference vector between first pixel values of the first color reference point and a first calibrated color value of the first color reference point; and identifying a second difference vector between second pixel values of the second color reference point and a second calibrated color value of the second color reference point, wherein generating the color space adjustment map includes generating a blending vector for the color space adjustment map between the first color reference point and the second color reference point based on the first difference vector and the second difference vector, the blending vector gradually transitioning the image color of a transition portion of the adjusted overlaid image to be overlaid between the first color reference point and the second color reference point. ### Claim 18 (depends on 15) The computer-implemented method of claim 15, wherein identifying the at least one color reference point includes matching objects in the real-world scene to a known color reference object included in an array of prior photos that are accessible to a user. ### Claim 19 (depends on 15) The computer-implemented method of claim 15, wherein the color reference point includes a sclera. ### Claim 20 (depends on 15) The computer-implemented method of claim 15, wherein the scene image is captured by a front-facing camera of a head mounted display.