# US 10,375,264 B2 · Multispectral lighting reproduction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,375,264 B2 - Document ID: US10375264B2 - Status: Granted - Kind: B2 - Application number: 15/582,522 - Filed: 2017-04-28 - Published: 2017-11-02 - Granted: 2019-08-06 - Earliest claimed: 2016-04-28 - Assignees: University of Southern California - Applicants: University of Southern California - Inventors: Paul E. Debevec; Chloe Legendre; Sumanta Pattanaik - Web page: https://patentgrep.com/patent/US10375264B2 ## CPC - H04N 1/6086: {by scene illuminant, i.e. conditions at the time of picture capture, e.g. flash, optical filter used, evening, cloud, daylight, artificial lighting, white point measurement, colour temperature} - G01J 3/505: {measuring the colour produced by lighting fixtures other than screens, monitors, displays or CRTs} - G06T 15/50: Lighting effects - G06V 10/141: Control of illumination - G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching} - G08B 13/19634: {Electrical details of the system, e.g. component blocks for carrying out specific functions} - H04N 1/0284: {in combination with a light integrating, concentrating or diffusing cavity} - H04N 1/6055: {using test pattern analysis} - H04N 1/64: Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor {(H04N19/00 takes precedence)} - G01J 2003/1282: {Spectrum tailoring} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS ## Abstract Systems and techniques for multispectral lighting reproduction, in one aspect, include: one or more light sources having different lighting spectra; and one or more computers comprising at least one processor and at least one memory device, the one or more computers programmed to drive the one or more light sources directly using intensity coefficients that have been determined by comparing first data for a multi-color reference object photographed by a camera in a scene with second data for the multi-color reference object photographed when lit by respective ones of the different lighting spectra of the one or more light sources. ## Claims ### Claim 1 A system comprising: one or more light sources having different lighting spectra; and one or more computers comprising at least one processor and at least one memory device, the one or more computers programmed to drive the one or more light sources directly using intensity coefficients that have been determined by comparing first data for a multi-color reference object photographed by a camera in a scene with second data for the multi-color reference object photographed when lit by respective ones of the different lighting spectra of the one or more light sources. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more light sources comprise Light Emitting Diode (LED) light sources that generate the different lighting spectra comprising three or more LED spectra. ### Claim 3 (depends on 2) The system of claim 2, wherein the three or more LED spectra comprise red, green, and blue. ### Claim 4 (depends on 3) The system of claim 3, wherein the three or more LED spectra further comprise at least one of amber and white. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more light sources are multiple light sources arranged in a geometric shape to illuminate a subject from multiple directions. ### Claim 6 (depends on 1) The system of claim 1, wherein the one or more light sources are multiple light sources arranged in an omnidirectional lighting sphere to illuminate a subject from various directions. ### Claim 7 (depends on 1) The system of claim 1, wherein the one or more computers are programmed to determine the intensity coefficients by solving for the intensity coefficients using nonnegative least squares (NNLS). ### Claim 8 (depends on 1) The system of claim 1, wherein the scene has an illumination environment, and the one or more computers are programmed to use the data from photographs of the multi-color reference object taken using different angular sections of the illumination environment as an illuminant for the multi-color reference object to determine the intensity coefficients. ### Claim 9 (depends on 8) The system of claim 8, wherein the one or more computers are programmed to, for a light source, (i) select at least three of a nearest multi-color reference object lighting data among the photographs taken at different angular sections of the illumination environment that are closest to a direction of the light source, (ii) interpolate the selected multi-color reference object lighting data to create an interpolated set of multi-color reference object lighting data, and (iii) determine the intensity coefficients for the light source using the interpolated set of multi-color reference object lighting data. ### Claim 10 (depends on 1) The system of claim 1, wherein the scene included one or more copies of the multi-color reference object oriented in various directions and the illumination environment is recorded using the camera via panoramic photography techniques, and the one or more computers are programmed to, for a light source, (i) identify sets of reference object lighting data corresponding to a subset of multiple copies of the multi-color reference objects that are closest to a position of the light source, (ii) interpolate the sets of reference object lighting data, wherein the one or more copies of the multi-color reference object are oriented in different directions, (iii) scale color channels of the interpolated set of reference object lighting data, based on an average pixel color of a lighting environment area corresponding to the light source, to generate a scaled set of reference object lighting data, and (iv) determine the intensity coefficients for the light source using the scaled set of reference object lighting data. ### Claim 11 (depends on 10) The system of claim 10, wherein the scene included at least a mirrored sphere and a black sphere for recording scene illumination using the camera via panoramic photography techniques, and the one or more computers are programmed to (i) correct first reflection data derived from light reflected from the mirrored sphere and second reflection data derived from light reflected from the black sphere, and (ii) combine the first reflection data and the second reflection data into a high dynamic range (HDR) map. ### Claim 12 (depends on 11) The system of claim 11, wherein the one or more computers are programmed to use the HDR map to determine the intensity coefficients for the light source using a scaled set of multi-color reference object lighting data, the intensity coefficients usable to drive the light sources with a captured multispectral illumination environment. ### Claim 13 (depends on 12) The system of claim 12, wherein the one or more computers are programmed to adjust the intensity coefficients as a function of time based on different frames of a video camera capturing an angular variation of illumination spectra. ### Claim 14 A method comprising: determining light intensity coefficients by minimizing a difference between (i) photograph of a multi-color reference object taken by a camera in a scene and (ii) a recorded set of multiple bases for an appearance of the multi-color reference object from photographs taken of the multi-color reference object when lit by respective ones of multiple light spectra; driving light sources in a lighting stage directly using the determined light intensities coefficients; and capturing imagery of a subject in the lighting stage illuminated by the light sources as they are driven using the determined light intensity coefficients. ### Claim 15 (depends on 14) The method of claim 14, wherein the photographs of the multi-color reference object are taken as illuminated by the light sources in the lighting stage, and the light sources comprise Light Emitting Diode (LED) light sources that generate different lighting spectra comprising three or more LED spectra. ### Claim 16 (depends on 14) The method of claim 14, wherein the three or more LED spectra comprise red, green, and blue. ### Claim 17 (depends on 16) The method of claim 16, wherein the three or more LED spectra further comprise at least one of amber and white. ### Claim 18 (depends on 14) The method of claim 14, wherein the light sources are multiple light sources arranged in a geometric shape to illuminate a subject from multiple directions. ### Claim 19 (depends on 14) The method of claim 14, wherein the light sources are multiple light sources arranged in an omnidirectional lighting sphere to illuminate a subject from different directions. ### Claim 20 (depends on 14) The method of claim 14, wherein the determining comprises solving for the light intensity coefficients using nonnegative least squares (NNLS). ### Claim 21 (depends on 14) The method of claim 14, wherein the scene has an illumination environment, and the method comprises directing the camera to take photographs of the multi-color reference object using different angular sections of the illumination environment as an illuminant for the multi-color reference object, and the determining comprises using the photographs taken using the different angular sections of the illumination environment to determine the light intensity coefficients. ### Claim 22 (depends on 21) The method of claim 21, comprising for a light source in the lighting stage: selecting at least three of the nearest multi-color reference object lighting data among the photographs taken at different angular sections of the illumination environment that are closest to a direction of the light source; interpolating the selected multi-color reference object lighting data to create an interpolated set of multi-color reference object lighting data; and determining the light intensity coefficients for the light source using the interpolated set of multi-color reference object lighting data. ### Claim 23 (depends on 14) The method of claim 14, wherein the scene included one or more copies of the multi-color reference object oriented in various directions and the illumination environment is recorded using the camera via panoramic photography techniques, and the method comprises, for a light source, identifying sets of reference object lighting data corresponding to a subset of multiple copies of the multi-color reference objects that are closest to a position of the light source; interpolating the sets of reference object lighting data, wherein the one or more copies of the multi-color reference object are oriented in different directions; scaling color channels of the interpolated set of reference object lighting data, based on an average pixel color of a lighting environment area corresponding to the light source, to generate a scaled set of reference object lighting data; determining the intensity coefficients for the light source using the scaled set of reference object lighting data. ### Claim 24 (depends on 23) The method of claim 23, wherein the one or more copies of the multi-color reference object comprise a first multi-color reference object facing the camera, a second oriented vertically upward with respect to the camera by 45 degrees, a third oriented vertically downward with respect to the camera by 45 degrees, a fourth oriented horizontally leftward with respect to the camera by 45 degrees, and a fifth oriented horizontally rightward with respect to the camera by 45 degrees. ### Claim 25 (depends on 23) The method of claim 23, wherein the scene included at least a mirrored sphere and a black sphere for recording using the camera via panoramic photography techniques, and the method comprises: correcting first reflection data derived from light reflected from the mirrored sphere and second reflection data derived from light reflected from the black sphere; and combining the first reflection data and the second reflection data into a high dynamic range (HDR) map. ### Claim 26 (depends on 25) The method of claim 25, comprising: using the HDR map to determine the intensity coefficients for the light source by scaling a set of multi-color reference object lighting data, the intensity coefficients usable to drive the light sources with a captured multispectral illumination environment. ### Claim 27 (depends on 26) The method of claim 26, comprising: adjusting the light intensity coefficients as a function of time based on different frames of a video camera capturing an angular variation of illumination spectra. ### Claim 28 (depends on 21) The method of claim 21, wherein directing the camera to take photographs comprises using an apparatus comprising a pan/tilt unit, the camera, the multi-color reference object, and an enclosure configured to allow only a cone of light to fall on the multi-color reference object, and wherein the cone has width chosen to match angular resolution of the lighting stage. ### Claim 29 A non-transitory computer-readable medium encoding a computer program operable to cause data processing apparatus to perform operations comprising: (a) determining light intensity coefficients by minimizing a difference between (i) photograph of a multi-color reference object taken by a camera in a scene and (ii) a recorded set of multiple bases for an appearance of the multi-color reference object from photographs taken of the multi-color reference object when lit by respective ones of multiple light spectra; (b) driving light sources in a lighting stage directly using the determined light intensities coefficients.