# US 11,308,652 B2 · Rendering objects to match camera noise > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,652 B2 - Document ID: US11308652B2 - Status: Granted - Kind: B2 - Application number: 16/740,972 - Filed: 2020-01-13 - Published: 2020-08-27 - Granted: 2022-04-19 - Earliest claimed: 2019-02-25 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Daniel Kurz; Tobias Holl - Web page: https://patentgrep.com/patent/US11308652B2 ## CPC - G06T 11/00: Two-dimensional [2D] image generation - G06T 5/70: Denoising; Smoothing - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2200/32: involving image mosaicing - G06T 2215/16: Using real world measurements to influence rendering - G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Various implementations disclosed herein render virtual content with noise that is similar to or that otherwise better matches the noise found in the images with which the virtual content is combined. Some implementations involve identifying noise data for an image, creating a parameterized noise model based on the noise data, generating a noise pattern approximating noise of the image or another image using the parameterized noise model, and rendering content that includes the image and virtual content with noise added based on the noise pattern. ## Claims ### Claim 1 A method, comprising: at an electronic device having a processor: identifying noise data for a first image captured by an image capture device; creating a parameterized noise model based on the noise data; generating a noise pattern approximating noise in a second image using the parameterized noise model; and rendering computer-generated reality (CGR) content comprising: a first portion comprising the virtual content, the virtual content comprising noise added to the virtual content based on the noise pattern; and a second portion comprising content of the second image and comprising the noise in the second image. ### Claim 2 (depends on 1) The method of claim 1, wherein identifying the noise data comprises measuring camera noise for a plurality of different signal-to-noise ratio (SNR) levels using images captured by the image capture device in a plurality of different lighting conditions or imaging conditions. ### Claim 3 (depends on 1) The method of claim 1, wherein creating the parameterized noise model comprises identifying parameters of the parameterized noise model to fit the parameterized noise model to the noise data. ### Claim 4 (depends on 3) The method of claim 3, wherein identifying the parameters comprises: determining an intensity deviation and an associated spatial correlation based on the noise data; and identifying parameters for the parameterized noise model to produce output corresponding to the spatial correlation and intensity deviation distribution. ### Claim 5 (depends on 4) The method of claim 4, wherein the intensity deviation is determined by creating a difference image, the difference image comprising a difference between two images of measured noise, wherein the spatial correlation is determined based on shifting and correlating the difference image with itself. ### Claim 6 (depends on 4) The method of claim 4, wherein the parameterized noise model describes the spatial correlation via smoothing using a kernel described by a Gaussian mixture model. ### Claim 7 (depends on 1) The method of claim 1, wherein the second image is the first image, wherein parameters for the parameterized noise model are identified using images captured by the image capture device during the rendering of the CGR content. ### Claim 8 (depends on 1) The method of claim 1, wherein the second image is different from the first image, wherein parameters for the parameterized noise model for a plurality of different signal-to-noise ratio (SNR) levels are identified using images captured by the image capture device prior to the rendering of the CGR content. ### Claim 9 (depends on 8) The method of claim 8, wherein generating the noise pattern comprises: identifying a signal-to-noise ratio (SNR) level of the second image; identifying parameters based on the SNR level; and generating the noise pattern using the parameterized-noise model and the parameters. ### Claim 10 (depends on 9) The method of claim 9, wherein identifying the SNR level comprises receiving the SNR level from an image signal processor (ISP). ### Claim 11 (depends on 9) The method of claim 9, wherein identifying the SNR level comprises determining the SNR level using a machine learning model that assesses the second image. ### Claim 12 (depends on 1) The method of claim 1, wherein the noise pattern is precomputed for a signal-to-noise (SNR) range and the noise pattern is looked up for the second image at runtime based on identifying a SNR level of the second image. ### Claim 13 (depends on 1) The method of claim 1, wherein rendering the CGR content comprises tiling the noise pattern over the virtual content. ### Claim 14 (depends on 1) The method of claim 1, wherein rendering the CGR content comprises the second image, the CGR content is composited with the second image, or the CGR content is composited with a third image. ### Claim 15 A system comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: identifying noise data for a first image captured by an image capture device; creating a parameterized noise model based on the noise data; generating a noise pattern approximating noise in a second image using the parameterized noise model; and rendering computer-generated reality (CGR) content comprising: a first portion comprising the virtual content, the virtual content comprising noise added to the virtual content based on the noise pattern; and a second portion comprising content of the second image and comprising the noise in the second image. ### Claim 16 (depends on 15) The system of claim 15, wherein creating the parameterized noise model comprises identifying parameters of the parameterized noise model to fit the parameterized noise model to the noise data. ### Claim 17 (depends on 16) The system of claim 16, wherein identifying the parameters comprises: determining an intensity deviation and an associated spatial correlation based on the noise data; and identifying parameters for the parameterized noise model to produce output similar to the spatial correlation and intensity deviation distribution. ### Claim 18 (depends on 17) The system of claim 17, wherein determining the intensity deviation comprises creating a difference image, the difference image comprising a difference between two images of measured noise, wherein the spatial correlation is determined based on shifting and correlating the difference image with itself. ### Claim 19 A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising: identifying noise data for a first image captured by an image capture device; creating a parameterized noise model based on the noise data; generating a noise pattern approximating noise in a second image using the parameterized noise model; and rendering computer-generated reality (CGR) content comprising: a first portion comprising the virtual content, the virtual content comprising noise added to the virtual content based on the noise pattern; and a second portion comprising content of the second image and comprising the noise in the second image. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein creating the parameterized noise model comprises identifying parameters of the parameterized noise model to fit the parameterized noise model to the noise data.