# US 11,908,234 B2 · Method and apparatus to estimate scene illuminant based on skin reflectance database > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,234 B2 - Document ID: US11908234B2 - Status: Granted - Kind: B2 - Application number: 17/298,357 - Filed: 2019-11-22 - Published: 2022-04-28 - Granted: 2024-02-20 - Earliest claimed: 2018-11-30 - Assignees: InterDigital VC Holdings, Inc. - Applicants: InterDigital VC Holdings, Inc. - Inventors: David Wyble - Web page: https://patentgrep.com/patent/US11908234B2 ## CPC - G06V 40/162: {using pixel segmentation or colour matching} - G06T 15/506: {Illumination models} - G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model - G06V 20/20: in augmented reality scenes - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Some embodiments of a method may include obtaining an image a real-world environment, determining an estimated illuminant spectral power distribution of an illuminant of the real-world environment, and detecting a region of the image representing human skin. The method may further include determining a representative skin color value of the region and based on the estimated illuminant spectral power distribution and the representative skin color value most closely matching the representative color value, selecting a candidate skin reflectance spectrum. The method may further include updating the estimated illuminant spectral power distribution base on the representative skin color value and the selected candidate skin reflectance spectrum. ## Claims ### Claim 1 A method comprising: obtaining an image of a real-world scene; determining a first estimated illuminant spectral power distribution for an illuminant of the real-world scene; identifying, within the image, at least a first region that represents human skin; from the image, determining a representative skin color value for the first region; based on the first estimated illuminant spectral power distribution and the representative skin color value, selecting a first candidate skin reflectance spectrum; selecting a first updated estimated illuminant spectral power distribution based on the representative skin color value and the selected first candidate skin reflectance spectrum; and rendering a virtual object by an augmented reality device using the first updated estimated illuminant spectral power distribution as an illuminant of the virtual object. ### Claim 2 (depends on 1) The method of claim 1, wherein obtaining the image of the real-world scene comprises capturing the image with a camera. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the representative skin color value comprises converting representative RGB coordinates to device-independent color coordinates. ### Claim 4 (depends on 1) The method of claim 1, wherein the selecting of the first candidate skin reflectance spectrum comprises: for each of a plurality of stored skin reflectance spectra, determining a respective expected color value by applying the first estimated illuminant spectral power distribution to the respective stored skin reflectance spectrum; and selecting the first candidate skin reflectance spectrum from among the plurality of stored skin reflectance spectra based on comparison of the respective expected color values with the representative skin color value. ### Claim 5 (depends on 4) The method of claim 4, wherein the first candidate skin reflectance spectrum is selected to substantially minimize a color difference between the representative skin color value and the expected color value associated with the first candidate skin reflectance spectrum. ### Claim 6 (depends on 1) The method of claim 1, wherein the selecting of the first candidate skin reflectance spectrum comprises: selecting at least one parameter value of a parameterized skin reflectance model to substantially minimize a color difference between: (i) an expected color value obtained by applying the first estimated illuminant spectral power distribution to a skin reflectance spectrum parameterized by the selected parameter value and (ii) the representative skin color value; wherein the first candidate skin reflectance spectrum is the skin reflectance spectrum parameterized by the selected parameter value. ### Claim 7 (depends on 1) The method of claim 1, wherein the selecting of the first updated estimated illuminant comprises: for each of a plurality of stored illuminant spectral power distributions, determining a respective expected color value by applying the stored illuminant spectral power distribution to the first candidate skin reflectance spectrum; and selecting the first updated estimated illuminant from among the plurality of stored illuminant spectral power distributions based on comparison of the respective expected color values with the representative color value. ### Claim 8 (depends on 7) The method of claim 7, wherein determining the respective expected color value comprises: applying a spectral camera model to generate camera RGB coordinates; and converting the camera RGB coordinates to device-independent color coordinates representing the respective expected color value. ### Claim 9 (depends on 7) The method of claim 7, wherein the plurality of stored illuminant spectral power distributions comprise a plurality of International Commission on Illumination, CIE, standard illuminants. ### Claim 10 (depends on 1) The method of claim 1, further comprising performing iterative refinement by a method comprising: based on the first updated estimated illuminant spectral power distribution and the representative skin color value, selecting a second candidate skin reflectance spectrum; and selecting a second updated estimated illuminant spectral power distribution based on the representative skin color value and the selected second candidate skin reflectance spectrum. ### Claim 11 A system comprising a processor and a non-transitory computer-readable medium storing instructions operative to perform a method comprising: obtaining an image of a real-world scene; determining a first estimated illuminant spectral power distribution for an illuminant of the real-world scene; identifying, within the image, at least a first region that represents human skin; from the image, determining a representative skin color value for the first region; based on the first estimated illuminant spectral power distribution and the representative skin color value, selecting a first candidate skin reflectance spectrum; selecting a first updated estimated illuminant spectral power distribution based on the representative skin color value and the selected first candidate skin reflectance spectrum; and rendering a virtual object by an augmented reality device using the first updated estimated illuminant spectral power distribution as an illuminant of the virtual object. ### Claim 12 (depends on 11) The system of claim 11, wherein determining the representative skin color value comprises converting representative RGB coordinates to device-independent color coordinates. ### Claim 13 (depends on 11) The system of claim 11, wherein the selecting of the first candidate skin reflectance spectrum comprises: for each of a plurality of stored skin reflectance spectra, determining a respective expected color value by applying the first estimated illuminant spectral power distribution to the respective stored skin reflectance spectrum; and selecting the first candidate skin reflectance spectrum from among the plurality of stored skin reflectance spectra based on comparison of the respective expected color values with the representative skin color value. ### Claim 14 (depends on 13) The system of claim 13, wherein the first candidate skin reflectance spectrum is selected to substantially minimize a color difference between the representative skin color value and the expected color value associated with the first candidate skin reflectance spectrum. ### Claim 15 (depends on 11) The system of claim 11, wherein the selecting of the first candidate skin reflectance spectrum comprises: selecting at least one parameter value of a parameterized skin reflectance model to substantially minimize a color difference between: (i) an expected color value obtained by applying the first estimated illuminant spectral power distribution to a skin reflectance spectrum parameterized by the selected parameter value and (ii) the representative skin color value; wherein the first candidate skin reflectance spectrum is the skin reflectance spectrum parameterized by the selected parameter value. ### Claim 16 (depends on 11) The system of claim 11, wherein the selecting of the first updated estimated illuminant comprises: for each of a plurality of stored illuminant spectral power distributions, determining a respective expected color value by applying the stored illuminant spectral power distribution to the first candidate skin reflectance spectrum; and selecting the first updated estimated illuminant from among the plurality of stored illuminant spectral power distributions based on comparison of the respective expected color values with the representative color value. ### Claim 17 A method comprising: obtaining an image of a real-world scene; identifying, within the image, at least a first region that represents human skin; from the image, determining a representative skin color value for the first region; based on the representative color value, selecting at least one candidate skin color value from among a plurality of candidate skin color values, where each of the candidate skin color values is associated with a respective illuminant; selecting, as an estimated illuminant for the real-world scene, the respective illuminant associated with the selected candidate skin color value; and rendering a virtual object by an augmented reality device using the estimated illuminant as an illuminant of the virtual object. ### Claim 18 (depends on 17) The method of claim 17, wherein the candidate skin color values are stored skin color values associated with respective predetermined illuminants.